LE 11 Nightly freezing when playing Live TV

Just encountered a quick memory exhausting on client's RPi 4B/2G while watching LiveTV (fullHD stream, H.265), running LE 11 Nightly 20220209, Tvheadend HTSP client. Tvheadend server is running on another RPi 4B/4GB with LE 11 Nightly 20220207, LAN connection.

I am monitoring free (available) memory with a script, displaying alert on screen with free memory value when avail Mem is less than 300 MB. So if I am currently watching, I have some time to react.

The issue happened some time after Timeshift move back (by left arrow IR Remote key push one times). I have encountered the Timeshift bar jumped somewhere but not a few seconds back as it should. Looked like the Timeshift bar was 'confused'. I let it be and kept watching.

After several minutes the free memory fell under 300 MB and went down quickly. When it was less than 100 MB, I started pushing left and right arrows to move the Timeshift point and prevent an OOM crash. This helped freeing memory again but not to usual value:

Now:

top - 04:03:32 up 1 day, 6:14, 1 user, load average: 1.65, 1.66, 1.62

Tasks: 155 total, 1 running, 154 sleeping, 0 stopped, 0 zombie

%Cpu(s): 3.0 us, 2.3 sy, 0.3 ni, 94.3 id, 0.0 wa, 0.0 hi, 0.2 si, 0.0 st

MiB Mem : 1847.1 total, 806.1 free, 894.8 used, 146.2 buff/cache

MiB Swap: 0.0 total, 0.0 free, 0.0 used. 860.2 avail Mem

Usually:

top - 18:05:47 up 1 day, 17:14, 1 user, load average: 1.47, 1.46, 1.40

Tasks: 156 total, 2 running, 154 sleeping, 0 stopped, 0 zombie

%Cpu(s): 9.6 us, 2.8 sy, 0.2 ni, 85.7 id, 1.7 wa, 0.0 hi, 0.0 si, 0.0 st

MiB Mem : 1847.1 total, 906.9 free, 487.9 used, 452.3 buff/cache

MiB Swap: 0.0 total, 0.0 free, 0.0 used. 1236.9 avail Mem

After several minutes the memory started falling down again. Moving Timeshift point again helped, did not have time to look which app was eating the memory. But kodi.bin currently consumes 40,4% of memory which looks a quite high:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

1665 root 20 0 1364944 764848 79392 D 19.5 40.4 437:56.89 kodi.bin

I was waiting but the issue did not repeat again.

BTW. Is there any way to disable the rpivid feb10000.codec: PPS changed message, spamming the kernel log quickly for ages?

btw. this is a top output after upgrade to LE 11 Nightly 20220212 after fresh boot:

top - 18:18:26 up 2 min, 1 user, load average: 1.81, 0.97, 0.39

Tasks: 162 total, 2 running, 160 sleeping, 0 stopped, 0 zombie

%Cpu(s): 4.7 us, 2.7 sy, 0.4 ni, 92.1 id, 0.0 wa, 0.0 hi, 0.2 si, 0.0 st

MiB Mem : 1847.1 total, 1294.7 free, 241.5 used, 310.9 buff/cache

MiB Swap: 0.0 total, 0.0 free, 0.0 used. 1522.6 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

1660 root 20 0 716288 159264 81096 D 16.6 8.4 1:32.93 kodi.bin

So it's obvious there is a memory leak somewhere/sometimes.

Well, I am using Timeshift by default (permanently enabled) while I am watching Live TV. So it's always somehow related with the described issues. I am rarely watching recordings so all described issues are related to Live TV + DVB-T2.

There were always issues with Timeshift but I don't know if a memory leak was there as well in past as I started discovering the issues more deeply several weeks ago. And some issues were hard to address on the same device. Now, separated to 2 devices, some issues are more visible. But it looks to me some issues (that forced me to analyzing it) appeared in recent Nightly builds with updated kernel release.

In general, I am observing the (not just memory) issues depends on:

- moving the Timeshift point

- data stream speed (more often at higher data rates)

- uptime / time of using (watching the same channel, usually several hours are needed to encounter the trouble).

And now I can confirm the DVB-T adapter firmware reload happens due to Tvheadend 4.2 service's (unexpected) restart on 'server' RPi 4B/4GB device (without OOM killer action).

BTW. this is a current top output on 'client' RPi 4B/2GB after almost 19 hours:

top - 12:58:08 up 18:42, 1 user, load average: 1.69, 1.43, 1.41

Tasks: 156 total, 1 running, 155 sleeping, 0 stopped, 0 zombie

%Cpu(s): 6.2 us, 3.4 sy, 0.2 ni, 90.0 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st

MiB Mem : 1847.1 total, 963.3 free, 540.3 used, 343.4 buff/cache

MiB Swap: 0.0 total, 0.0 free, 0.0 used. 1203.6 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

1660 root 20 0 1054956 480324 100192 D 32.1 25.4 236:23.04 kodi.bin

After restarting (just) Kodi:

top - 13:26:12 up 19:10, 1 user, load average: 1.61, 1.37, 1.35

Tasks: 155 total, 1 running, 154 sleeping, 0 stopped, 0 zombie

%Cpu(s): 7.3 us, 3.1 sy, 0.2 ni, 89.4 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st

MiB Mem : 1847.1 total, 1284.2 free, 237.2 used, 325.7 buff/cache

MiB Swap: 0.0 total, 0.0 free, 0.0 used. 1525.1 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

10968 root 20 0 751632 155516 80752 D 21.5 8.2 0:17.31 kodi.bin

BTW. another quick memory leak encountered on 'client' RPi 4B/2GB (in the meantime upgraded so running LE Nightly 20220212-d9a07e9):

top - 22:58:46 up 5:45, 1 user, load average: 1.63, 1.43, 1.31

Tasks: 155 total, 2 running, 153 sleeping, 0 stopped, 0 zombie

%Cpu(s): 0.0 us, 16.7 sy, 0.0 ni, 83.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

MiB Mem : 1847.1 total, 37.2 free, 1605.0 used, 204.8 buff/cache

MiB Swap: 0.0 total, 0.0 free, 0.0 used. 152.7 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

1662 root 20 0 2076732 1.5g 84236 R 50.0 82.5 71:02.91 kodi.bin

The memory suddenly started decreasing - about 0,5 MB per second.

After moving Timeshift back (Left Arrow IR Remote key) it stabilized:

top - 22:59:22 up 5:45, 1 user, load average: 1.62, 1.45, 1.32

Tasks: 155 total, 1 running, 154 sleeping, 0 stopped, 0 zombie

%Cpu(s): 5.6 us, 2.8 sy, 0.6 ni, 91.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

MiB Mem : 1847.1 total, 753.4 free, 906.3 used, 187.4 buff/cache

MiB Swap: 0.0 total, 0.0 free, 0.0 used. 853.0 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

1662 root 20 0 1358388 843396 84192 D 16.9 44.6 71:12.32 kodi.bin

Kodi on 'client' RPi 4B/2G crashed after switching to another TV channel, at the same time on the 'server' RPi 4B/4G Tvheadend 4.2 server restarted.

http://ix.io/3PzG

The issue described in post #19 still happens on LE Nightly 20220508.