LE 11 Nightly freezing when playing Live TV

    I have experienced several times a strange issue on current LE 11 Nightlies, installed on RPi 4B / 2GB at external HDD, connected through USB/SATA adapter. TVheadend server 4.3 + TVheadend client used.

    It's a more or less random freeze of LE (did not even respond to PING, SSH console was frozen as well, no response to remote control and during that time there was an intensive HDD activity) and it looks it's related to TimeShift function which activated oom-killer.

    After some time (up to several minutes) the LE usually starts responding again, even the SSH console starts where it stopped responding (without disconnection).

    http://ix.io/3zkn

    http://ix.io/3zkq

    There are also some random issues with audio - sometimes audio does not start when I switch the channel - it's necessary to switch to another channel and back. Sometimes (usually when the live stream has a temporary corruption due to bad signal) it leads to freeze playing (a percentage circle appears on display and stops somewhere between 0-100%). Using TimeShift to move back or forward is usually possible (even though the TimeShift pointer gets confused and does not work correctly until stopped / started or switch to another channel). When I got back, the stream was playng until the problematic point where it freezed again. Unfortunately I don't have a stream recorded as it's not possible to save the TimeShift data (configured to use RAM only).

    This is a part of kernel log, showing the several attempts to shift back before the freeze point:

    External Content pastebin.com
    Content embedded from external sources will not be displayed without your consent.
    Through the activation of external content, you agree that personal data may be transferred to third party platforms. We have provided more information on this in our privacy policy.

    Repeating issue on LE Nightly 20210930.

    External Content pastebin.com
    Content embedded from external sources will not be displayed without your consent.
    Through the activation of external content, you agree that personal data may be transferred to third party platforms. We have provided more information on this in our privacy policy.

    Also encountered on another RPi 4B / 4G, running LE Nightly 20210928.

    External Content pastebin.com
    Content embedded from external sources will not be displayed without your consent.
    Through the activation of external content, you agree that personal data may be transferred to third party platforms. We have provided more information on this in our privacy policy.

    It's a very annoying behaviour as it takes several minutes until LE returns to operation and there are intensive (perhaps write) operations with storage. It looks it appears more likely when the TV channel with high bitrate is played.

    Edited once, last by ghtester (October 2, 2021 at 8:53 PM).

    dmesg shows tvheadend is using a large amount of RAM which probably is the cause for the OOM killer kicking in and killing it.

    I'd suggest you look into that first, eg disable timeshift-to-RAM if you are using that.

    so long,

    Hias

    Thanks for the reply / advice but the timeshift to RAM is a basic functionality which I can't 'live' without. ;)

    It looks that latest freeze on RPi 4B / 4G happened when there're still a plenty of free RAM and there was no obvious reason for OOM killer to be activated (I know such cases were reported on earlier - 32 bit - kernel and should be already fixed now).

    But I wanted to point out that now it's a worse case (than earlier) due to several minutes complete freeze after OOM event.

    Another freeze encountered on RPi 4B / 4G, running LE 11 Nightly 20211002.

    As the issue happens too often and it's very annoying, I am afraid I'll have to revert to some earlier LE version as the stability is one of the most important features to me...

    If someone have an idea how to prevent LE from freezing even though OOM killer is activated (like before), please let me know. Perhaps I should uncomment the Storage=auto option in /etc/systemd/journald.conf or set it to volatile?

    http://ix.io/3AKh

    External Content pastebin.com
    Content embedded from external sources will not be displayed without your consent.
    Through the activation of external content, you agree that personal data may be transferred to third party platforms. We have provided more information on this in our privacy policy.

    Edited 6 times, last by ghtester (October 3, 2021 at 9:13 PM).

    Thank you for feedback / advice.

    Yes I can try to decrease a Timeshift Maximum RAM size parameter a bit (currently 3090 MB on RPi 4B/4GB, running Tvheadend 4.2 and 1100 MB on RPi 4B/2GB).

    But why the hell the LE is now freezing and generates a huge disk / SD card activity for several minutes when OOM killer is activated?!?

    It did not happen in older LE releases (if OOM killer terminated Tvheadend, it was restarted immediately, the Timeshift data were lost but LE was not completely hanged for long time).

    So it looks this behaviour was started on in some recent kernel version and it's a nightmare (and also not very good for SD card if there are writes).

    The nightmare is that you do accept OOM situations.

    For research reasons a core dump is generated by default if possible. This can be disabled by

    Code
    echo 0 >/proc/sys/vm/oom_dump_tasks
    Quote from mglae

    The nightmare is that you do accept OOM situations.

    I know, you are right but it's only temporary... ;)

    Quote from mglae

    For research reasons a core dump is generated by default if possible. This can be disabled by



    Code echo 0 >/proc/sys/vm/oom_dump_tasks

    Great, thanks a lot! That's exactly what I needed to know.

    Quote from CvH

    that is basically too much


    try 2500/800, that should work more reliable

    Thank you, I have already reduced those values a bit and started monitoring free RAM online by script. It looks that latest LE releases are a slightly bit more hungry regarding to RAM as I did not have OOM issues with the mentioned settings couple weeks ago.

    I believe the thread can be marked as Resolved.

    Several months later I have to say this issue is still here, still experiencing it on latest LE 11 nightlies (20220130). :(

    Also the recommended command echo 0 >/proc/sys/vm/oom_dump_tasks does not work (to disable long huge disk activity after OOM killer stops Tvheadend server).

    The issue usually hapens when I get back with Timeshift and keep watching the same TV program for a long time. Especially when the Video stream has a higher bitrate. The memory gets exhausted, soon or later, but more or less randomly, despite the Timeshift Maximum RAM size reduction. For some (usually even quite long) time the free RAM is quite high, oscilating about bottom limit (based on Timeshift Maximum RAM settings) but suddenly it gets exhausted quickly to zero.

    I suppose it's a Tvheadend / Timeshift issue (currently using addon Tvheadend Server 4.3 (Alpha) 10.80.4.103 but the same with 4.2).

    The worst and most annoying issue is that the LibreELEC is completely frozen for unpredictably long time after OOM killer's action (can take hours - tested). It depends on RAM size so I suppose the dumps are flushed to boot media. Is there any way to stop it please?

    Quote from ghtester

    I suppose it's a Tvheadend / Timeshift issue (currently using addon Tvheadend Server 4.3 (Alpha) 10.80.4.103 but the same with 4.2)

    Can you run tvheadend server on a different device and connect tvh client to it over network.

    That should narrow down if the issue is in the client or server end.

    So I have configured one RPi 4B/4GB as Tvheadend Server (LE 11 Nightly 20220130). There's Tvheadend 4.2 (10.80.4.126) running and Tvheadend HTSP Client disabled.

    Another RPi 4b/2GB (LE 11 Nightly 20220206) acts as client - Tvheadend HTSP Client reconfigured to connect the remote Tvheadend Server on first RPi 4B/4GB instead of local Tvheadend Server service.

    During last few days I did not encounter any OOM killer action, but randomly the Timeshift gets 'disabled' at client and kernel log on server RPi display these messages:

    [14213.650491] si2168 22-0064: downloading firmware from file 'dvb-demod-si2168-d60-01.fw'

    [14214.786939] si2168 22-0064: firmware version: D 6.0.13

    [14214.803072] si2157 23-0060: found a 'Silicon Labs Si2141-A10'

    [14214.803208] si2157 23-0060: downloading firmware from file 'dvb-tuner-si2141-a10-01.fw'

    [14215.371252] si2157 23-0060: firmware version: 1.1.11

    So it looks like the USB DVB-T2 adapter is reinitialized. It's a question if this is due to kernel issue or if it is invoked by Tvheadend.

    Thanks for the reply. I don't think it's a power issue in this case as the PSU is really strong (30W) and the T230C DVB-T2 adapter is not so power hungry but I'll check. There's a short USB extension cable, maybe it could play some role - to be checked.

    I am not sure if the USB2 or USB3 port is used, to be checked in the evening and I can try to swap. Yesterday I have also updated to LE Nightly 20220207 so I'll see...

    Update - the DVB-T2 adapter was connected to USB3 port and during the night it was reinitialized again so I reconnected it to USB2 port instead.

    It looks the adapter reinitialization depends on data stream - happens more often when the video stream is 1920x1080p than with lower one.

    Update2 - the same or similar issue with DVB-T2 adapter connected to USB2 port. But also observed the situation that Timeshift got disabled on client without DVB-T2 reinitialization on server LE.

    Also encountered the same issue on client (Timeshift got disabled ) while only partial DVB-T2 adapter reinit - just the si2168 firmware was downloaded again:

    [89578.426312] si2168 22-0064: downloading firmware from file 'dvb-demod-si2168-d60-01.fw'

    [89579.552758] si2168 22-0064: firmware version: D 6.0.13

    Has anyone an idea why this happen and what could be a reason of si2168 firmware repeated download to adapter?

    Update3 - The Tvheand42's service.log is started at the same time as si2168 22-0064 firmware download is performed... So it looks like the Tvheadend service was restarted?

    Edited 10 times, last by ghtester (February 11, 2022 at 5:16 AM).