SNMP OIDs for obtaining or calculating used memory

  • MRTG with the new 'Available Memory' OID is now running a number of days.


    The below graph shows the weekly graph of the Used memory in percentage.


    There is constantly an increase of memory usage, but I was not able to identify a process (top -b -o +%MEM | head -n 20) that could be responsible for this increase.

    Graph details:

    - MRTG started last Sunday evening.

    - The RPi3 was running mostly idle for two days. During this period memory usage increased about 3 percent per day.

    - Kodi crashed on Tuesday (1-Dec-2020) evening at 2215. This decreased memory usage with about 6 to 7 percent. After this memory usage is flat for a long time.

    - Wednesday evening I played a bit with the Plex server and started a scan of my library which was still busy when I went to sleep.

    - Thursday morning at 7:42am there was another Kodi crash.

    - On Thursday at about 11am started a download that lasted till about 5pm

    - After the download memory increased about 8 percent a day.

    - At 2pm memory usage decreased with about 6 percent, but have not seen a reason (see spoiler for details).


    Enabling debug with TVH didn't additional information at the time of the crash:

    Tronsmart Ara X5 plus fried by lightning

    RPi3

    (removed from the dust with LibreELEC 8.0.1 still working)

  • Still the __throw_length_error() errors from Thread 1 in kodi or do you have noticed in the meantime any OOM condition logged in system or from any process?

  • For the crash on 1-Dec 22:15 yes, see below. For the crash on 2-Dec 7:42 I forgot to obtain the log file via SMB.

    (at the time I took the logs one of my tuners was continuously generating errors in dmesg that took my attention. A power-cycle of the tuner solved that)

    Code
    Dec 01 22:09:41 LibreELEC connmand[314]: ntp: adjust (slew): +0.000045 sec
    Dec 01 22:14:51 LibreELEC kodi.sh[613]: terminate called after throwing an instance of 'std::length_error'
    Dec 01 22:14:51 LibreELEC kodi.sh[613]:   what():  basic_string::_S_create
    Dec 01 22:14:55 LibreELEC tvheadend[276]: htsp: 127.0.0.1 [ admin | Kodi Media Center ]: Disconnected
    Dec 01 22:14:55 LibreELEC kernel: vchiq: 0: ignoring ERROR from callback to service 5004
    Dec 01 22:15:00 LibreELEC kodi.sh[613]: Aborted (core dumped)
    Dec 01 22:15:10 LibreELEC kodi.sh[613]: Crash report available at /storage/.kodi/temp/kodi_crashlog_20201201221502.log

    I have not seen a OOM error so far.


    Yesterday I stopped the Plex server (systemctl stop plex.service) to see if that makes a difference.

    (Before I had MRTG running I did that, but still had a crash.)

    Tronsmart Ara X5 plus fried by lightning

    RPi3

    (removed from the dust with LibreELEC 8.0.1 still working)

  • IMO the crash is a result of memory corruption or other programming error and not low memory.

  • Yes, indeed. Even with 3% memory usages increase each day it should run for a week or 2, before it is getting into problems.


    And I might have found a reason for the crashes.

    Planning to leave Plex Media Server (PMS) stopped for ~ a week or so longer to see if the frequent crashes stopped.


    On the Plex Media Server (PMS) forum, for ARM, I found the following note "There are some unresolved memory issues with the DLNA server support (at least on ARM). Disabling DLNA is currently recommended."


    In the PMS logs for the 4 crashes I checked I do see "DEBUG - NetworkServiceBrowser: Parsing SSDP schema for xxx" reasonably close to when the crashes happen (in the minute before or minute after a crash. I have not yet started PMS to verify the setting, but likely it is enabled (as ~ indicated by the log messages).



    Also with PMS stopped there was a memory usages increase of 3%/day for 2 days, and it nearly flattened for the last 2 days.

    Lets see in a few days.


    Edit 2020-Dec-11, Dec-13:

    I have not observed a crash after stopping Plex Media Server.

    Now thinking to start PMS again this weekend. Will leave it as it is for some time.


    There is still an increase of about (average) 1.5% a day over the last (nearly) 7 days.

    Compared the Used Memory (calculated from details obtained from 'free -k') and memory used by kodi.bin, python, tvheadend (obtained from the memory column of 'top') every 30min (after stopping PMS).

    Tronsmart Ara X5 plus fried by lightning

    RPi3

    (removed from the dust with LibreELEC 8.0.1 still working)

    Edited 5 times, last by RomMon ().

  • There was a new crash of kodi (on 14-Dec at 21:54:14).

    This time I 'saw' it happen.

    • The time on the LCD was 2 minutes behind.
    • Login via ssh took a long time, and by the time I had a login prompt I was only able to execute one command before kodi crased.

    dmesg has more messages than with previous crashes starting with

    [Mon Dec 14 21:51:54 CET 2020] rcu: INFO: rcu_sched self-detected stall on CPU


    Memory utilization (system, kodi, python and tvheadend):


    The daily and weelkly MRTG graphs:

    Files

    Tronsmart Ara X5 plus fried by lightning

    RPi3

    (removed from the dust with LibreELEC 8.0.1 still working)

  • Hi,


    Just an update on my findings so far.

    In the last days of last year I started to use Plex live TV & PVR i.s.o. TVHeadend (mainly for a daily recording of new and weather).


    In the graph you can see that the constant increase of memory usages moved from Kodi to Plex Media Server...


    I also found a memory release issue with SABnzb-2, not releasing its memory.


    In a week or so I hope to move to a x86_64 system.

    Tronsmart Ara X5 plus fried by lightning

    RPi3

    (removed from the dust with LibreELEC 8.0.1 still working)