SNMP OIDs for obtaining or calculating used memory

    MRTG with the new 'Available Memory' OID is now running a number of days.

    The below graph shows the weekly graph of the Used memory in percentage.

    There is constantly an increase of memory usage, but I was not able to identify a process (top -b -o +%MEM | head -n 20) that could be responsible for this increase.

    Graph details:

    - MRTG started last Sunday evening.

    - The RPi3 was running mostly idle for two days. During this period memory usage increased about 3 percent per day.

    - Kodi crashed on Tuesday (1-Dec-2020) evening at 2215. This decreased memory usage with about 6 to 7 percent. After this memory usage is flat for a long time.

    - Wednesday evening I played a bit with the Plex server and started a scan of my library which was still busy when I went to sleep.

    - Thursday morning at 7:42am there was another Kodi crash.

    - On Thursday at about 11am started a download that lasted till about 5pm

    - After the download memory increased about 8 percent a day.

    - At 2pm memory usage decreased with about 6 percent, but have not seen a reason (see spoiler for details).

    Top output 4-Dec-2020 2pm and 2:30pm

    - 13:59:33 up 6 days, 11:42, 1 user, load average: 0.03, 0.08, 0.10

    Tasks: 117 total, 1 running, 116 sleeping, 0 stopped, 0 zombie

    %Cpu(s): 2.9 us, 2.9 sy, 1.4 ni, 92.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

    MiB Mem : 747.9 total, 48.0 free, 486.5 used, 213.3 buff/cache

    MiB Swap: 0.0 total, 0.0 free, 0.0 used. 186.5 avail Mem

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

    25482 root 20 0 673628 179296 21224 S 12.5 23.4 620:12.02 kodi.bin

    608 root 30 10 340016 95884 1264 S 0.0 12.5 111:23.94 python

    405 root 20 0 328660 54852 8544 S 0.0 7.2 41:43.79 Plex Media Serv

    637 root 35 15 194612 51220 4180 S 6.2 6.7 36:04.24 Plex Script Hos

    9545 root 20 0 161860 15648 2132 S 0.0 2.0 266:01.81 tvheadend

    22220 root 20 0 5116 3968 3572 S 0.0 0.5 0:00.21 sshd

    581 root 20 0 30396 3336 1796 S 0.0 0.4 0:16.77 smbd

    916 root 20 0 125428 3128 448 S 0.0 0.4 8:26.13 Plex Tuner Serv

    1924 root 20 0 7312 2696 880 S 0.0 0.4 11:24.85 snmpd

    598 root 20 0 6084 2392 600 S 0.0 0.3 2:07.96 python

    22424 root 20 0 6020 2128 1700 R 6.2 0.3 0:00.03 top

    579 root 20 0 17560 2096 1060 S 0.0 0.3 1:39.76 nmbd

    2031 root 20 0 22540 1968 1564 S 0.0 0.3 0:13.87 systemd-journal

    Used memory %: 75.0576


    - 14:29:34 up 6 days, 12:12, 1 user, load average: 0.38, 0.78, 0.75

    Tasks: 124 total, 1 running, 123 sleeping, 0 stopped, 0 zombie

    %Cpu(s): 4.3 us, 2.9 sy, 1.4 ni, 90.0 id, 1.4 wa, 0.0 hi, 0.0 si, 0.0 st

    MiB Mem : 747.9 total, 32.2 free, 441.1 used, 274.6 buff/cache

    MiB Swap: 0.0 total, 0.0 free, 0.0 used. 230.3 avail Mem

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

    25482 root 20 0 673884 167368 9260 S 6.2 21.9 624:34.58 kodi.bin

    608 root 30 10 340016 95884 1264 S 0.0 12.5 111:33.71 python

    405 root 20 0 328660 57636 11328 S 6.2 7.5 41:50.92 Plex Media Serv

    637 root 35 15 194612 48596 1556 S 0.0 6.3 36:11.31 Plex Script Hos

    9545 root 20 0 162884 15980 2060 S 6.2 2.1 274:32.91 tvheadend

    581 root 20 0 30396 3328 1788 S 0.0 0.4 0:16.78 smbd

    916 root 20 0 125428 3128 448 S 0.0 0.4 8:27.78 Plex Tuner Serv

    1924 root 20 0 7312 2684 868 S 0.0 0.4 11:27.06 snmpd

    2031 root 20 0 23056 2576 2172 S 0.0 0.3 0:14.25 systemd-journal

    22220 root 20 0 5116 2408 2012 S 0.0 0.3 0:00.23 sshd

    598 root 20 0 6084 2392 600 S 0.0 0.3 2:08.32 python

    579 root 20 0 17560 2016 980 S 0.0 0.3 1:39.95 nmbd

    23006 root 20 0 6028 2008 1580 R 6.2 0.3 0:00.03 top

    Used memory % 69.1941

    Enabling debug with TVH didn't additional information at the time of the crash:

    Still the __throw_length_error() errors from Thread 1 in kodi or do you have noticed in the meantime any OOM condition logged in system or from any process?

    For the crash on 1-Dec 22:15 yes, see below. For the crash on 2-Dec 7:42 I forgot to obtain the log file via SMB.

    (at the time I took the logs one of my tuners was continuously generating errors in dmesg that took my attention. A power-cycle of the tuner solved that)

    Code
    Dec 01 22:09:41 LibreELEC connmand[314]: ntp: adjust (slew): +0.000045 sec
Dec 01 22:14:51 LibreELEC kodi.sh[613]: terminate called after throwing an instance of 'std::length_error'
Dec 01 22:14:51 LibreELEC kodi.sh[613]:   what():  basic_string::_S_create
Dec 01 22:14:55 LibreELEC tvheadend[276]: htsp: 127.0.0.1 [ admin | Kodi Media Center ]: Disconnected
Dec 01 22:14:55 LibreELEC kernel: vchiq: 0: ignoring ERROR from callback to service 5004
Dec 01 22:15:00 LibreELEC kodi.sh[613]: Aborted (core dumped)
Dec 01 22:15:10 LibreELEC kodi.sh[613]: Crash report available at /storage/.kodi/temp/kodi_crashlog_20201201221502.log

    I have not seen a OOM error so far.

    Yesterday I stopped the Plex server (systemctl stop plex.service) to see if that makes a difference.

    (Before I had MRTG running I did that, but still had a crash.)

    Yes, indeed. Even with 3% memory usages increase each day it should run for a week or 2, before it is getting into problems.

    And I might have found a reason for the crashes.

    Planning to leave Plex Media Server (PMS) stopped for ~ a week or so longer to see if the frequent crashes stopped.

    On the Plex Media Server (PMS) forum, for ARM, I found the following note "There are some unresolved memory issues with the DLNA server support (at least on ARM). Disabling DLNA is currently recommended."

    In the PMS logs for the 4 crashes I checked I do see "DEBUG - NetworkServiceBrowser: Parsing SSDP schema for xxx" reasonably close to when the crashes happen (in the minute before or minute after a crash. I have not yet started PMS to verify the setting, but likely it is enabled (as ~ indicated by the log messages).


    Also with PMS stopped there was a memory usages increase of 3%/day for 2 days, and it nearly flattened for the last 2 days.

    Lets see in a few days.

    Edit 2020-Dec-11, Dec-13:

    I have not observed a crash after stopping Plex Media Server.

    Now thinking to start PMS again this weekend. Will leave it as it is for some time.

    There is still an increase of about (average) 1.5% a day over the last (nearly) 7 days.

    Compared the Used Memory (calculated from details obtained from 'free -k') and memory used by kodi.bin, python, tvheadend (obtained from the memory column of 'top') every 30min (after stopping PMS).

    Edited 5 times, last by RomMon (December 13, 2020 at 11:08 AM).

    There was a new crash of kodi (on 14-Dec at 21:54:14).

    This time I 'saw' it happen.

    • The time on the LCD was 2 minutes behind.
    • Login via ssh took a long time, and by the time I had a login prompt I was only able to execute one command before kodi crased.

    dmesg has more messages than with previous crashes starting with

    [Mon Dec 14 21:51:54 CET 2020] rcu: INFO: rcu_sched self-detected stall on CPU

    Memory utilization (system, kodi, python and tvheadend):

    The daily and weelkly MRTG graphs:

    Hi,

    Just an update on my findings so far.

    In the last days of last year I started to use Plex live TV & PVR i.s.o. TVHeadend (mainly for a daily recording of new and weather).

    In the graph you can see that the constant increase of memory usages moved from Kodi to Plex Media Server...

    I also found a memory release issue with SABnzb-2, not releasing its memory.

    In a week or so I hope to move to a x86_64 system.