LE 11 Nightly freezing when playing Live TV

  • I have experienced several times a strange issue on current LE 11 Nightlies, installed on RPi 4B / 2GB at external HDD, connected through USB/SATA adapter. TVheadend server 4.3 + TVheadend client used.

    It's a more or less random freeze of LE (did not even respond to PING, SSH console was frozen as well, no response to remote control and during that time there was an intensive HDD activity) and it looks it's related to TimeShift function which activated oom-killer.

    After some time (up to several minutes) the LE usually starts responding again, even the SSH console starts where it stopped responding (without disconnection).


    http://ix.io/3zkn


    http://ix.io/3zkq


    There are also some random issues with audio - sometimes audio does not start when I switch the channel - it's necessary to switch to another channel and back. Sometimes (usually when the live stream has a temporary corruption due to bad signal) it leads to freeze playing (a percentage circle appears on display and stops somewhere between 0-100%). Using TimeShift to move back or forward is usually possible (even though the TimeShift pointer gets confused and does not work correctly until stopped / started or switch to another channel). When I got back, the stream was playng until the problematic point where it freezed again. Unfortunately I don't have a stream recorded as it's not possible to save the TimeShift data (configured to use RAM only).

    This is a part of kernel log, showing the several attempts to shift back before the freeze point:


    External Content pastebin.com
    Content embedded from external sources will not be displayed without your consent.
    Through the activation of external content, you agree that personal data may be transferred to third party platforms. We have provided more information on this in our privacy policy.

  • Repeating issue on LE Nightly 20210930.

    External Content pastebin.com
    Content embedded from external sources will not be displayed without your consent.
    Through the activation of external content, you agree that personal data may be transferred to third party platforms. We have provided more information on this in our privacy policy.

    Also encountered on another RPi 4B / 4G, running LE Nightly 20210928.

    External Content pastebin.com
    Content embedded from external sources will not be displayed without your consent.
    Through the activation of external content, you agree that personal data may be transferred to third party platforms. We have provided more information on this in our privacy policy.

    It's a very annoying behaviour as it takes several minutes until LE returns to operation and there are intensive (perhaps write) operations with storage. It looks it appears more likely when the TV channel with high bitrate is played.

    Edited once, last by ghtester ().

  • dmesg shows tvheadend is using a large amount of RAM which probably is the cause for the OOM killer kicking in and killing it.


    I'd suggest you look into that first, eg disable timeshift-to-RAM if you are using that.


    so long,


    Hias

  • Thanks for the reply / advice but the timeshift to RAM is a basic functionality which I can't 'live' without. ;)

    It looks that latest freeze on RPi 4B / 4G happened when there're still a plenty of free RAM and there was no obvious reason for OOM killer to be activated (I know such cases were reported on earlier - 32 bit - kernel and should be already fixed now).

    But I wanted to point out that now it's a worse case (than earlier) due to several minutes complete freeze after OOM event.

  • Another freeze encountered on RPi 4B / 4G, running LE 11 Nightly 20211002.

    As the issue happens too often and it's very annoying, I am afraid I'll have to revert to some earlier LE version as the stability is one of the most important features to me...

    If someone have an idea how to prevent LE from freezing even though OOM killer is activated (like before), please let me know. Perhaps I should uncomment the Storage=auto option in /etc/systemd/journald.conf or set it to volatile?


    http://ix.io/3AKh

    External Content pastebin.com
    Content embedded from external sources will not be displayed without your consent.
    Through the activation of external content, you agree that personal data may be transferred to third party platforms. We have provided more information on this in our privacy policy.

    Edited 6 times, last by ghtester ().

  • Thank you for feedback / advice.

    Yes I can try to decrease a Timeshift Maximum RAM size parameter a bit (currently 3090 MB on RPi 4B/4GB, running Tvheadend 4.2 and 1100 MB on RPi 4B/2GB).

    But why the hell the LE is now freezing and generates a huge disk / SD card activity for several minutes when OOM killer is activated?!?

    It did not happen in older LE releases (if OOM killer terminated Tvheadend, it was restarted immediately, the Timeshift data were lost but LE was not completely hanged for long time).

    So it looks this behaviour was started on in some recent kernel version and it's a nightmare (and also not very good for SD card if there are writes).

  • The nightmare is that you do accept OOM situations.


    For research reasons a core dump is generated by default if possible. This can be disabled by


    Code
    echo 0 >/proc/sys/vm/oom_dump_tasks
  • The nightmare is that you do accept OOM situations.

    I know, you are right but it's only temporary... ;)

    For research reasons a core dump is generated by default if possible. This can be disabled by



    Code echo 0 >/proc/sys/vm/oom_dump_tasks

    Great, thanks a lot! That's exactly what I needed to know.

    that is basically too much


    try 2500/800, that should work more reliable

    Thank you, I have already reduced those values a bit and started monitoring free RAM online by script. It looks that latest LE releases are a slightly bit more hungry regarding to RAM as I did not have OOM issues with the mentioned settings couple weeks ago.

    I believe the thread can be marked as Resolved.