SNMP OIDs for obtaining or calculating used memory

    I would like to monitor the memory usage of my RPi3 running LE-9.2.4.

    (My RPi3 is crashing very often and I want to find if memory is the reason)

    For this I would like to obtain or calculate the used memory via snmp and graph it with mrtg.


    Is it correct to use?:

    .1.3.6.1.4.1.2021.4.5.0 - .1.3.6.1.4.1.2021.4.6.0


    I'm a bit unsure because I saw someone using 'total' and 'available' from the 'free' command output e.g.:

    Code
    total_mem=$(free -k | grep Mem | awk '{print $2}')

    avail_mem=$(free -k | grep Mem | awk '{print $7}')

    memory_usage=$(awk "BEGIN {percent_used=($total_mem-$avail_mem)/($total_mem)*100; print percent_used}")

    echo $memory_usage

    54.0915

    While my calculation would be comparable with (but than with snmp):

    Code
    total_mem=$(free -k | grep Mem | awk '{print $2}')

    free_mem=$(free -k | grep Mem | awk '{print $4}')

    memory_usage=$(awk "BEGIN {percent_used=($total_mem-$free_mem)/($total_mem)*100; print percent_used}")

    echo $memory_usage

    96.3987





    e.g.

    Code
    total_mem=`snmpwalk -v 1 -c libreelec 192.168.178.22 -O qv .1.3.6.1.4.1.2021.4.5.0`

    free_mem=`snmpwalk -v 1 -c libreelec 192.168.178.22 -O qv .1.3.6.1.4.1.2021.4.6.0`

    used_mem=`expr $total_mem - $free_mem`

    echo used memory = $total_mem - $free_mem =  $used_mem

    used memory = 765828 - 36260 = 729568

    The free command gives the following output:


    With snmp I obtain the following:



    Tronsmart Ara X5 plus fried by lightning

    RPi3

    (removed from the dust with LibreELEC 8.0.1 still working)

    From /proc/meminfo you should use "MemAvailable" because it is ignoring "Buffers" and "Cached".


    The snmp results seem not be using this because memAvailReal == memTotalFree == 32736.

    Indeed via snmp the value for memAvailReal looks to be wrong.


    Last night I had a script running capturing the output of top every 30min, and see that actually buffers and cached is jumping up and down.

    The crashes look to corelate with high buffers/cached memory usage, (and lower free memory value).


    In the attached image you see MRTG graphs from the last ~3 days.

    The spikes in the eth0 graph indicate the crashes of kodi (at startup of kodi there seem to be some information being downloaded).

    (I have not yet looked at the reason for the re-appearing higher CPU usage each night between 2 and 6.)



    Tronsmart Ara X5 plus fried by lightning

    RPi3

    (removed from the dust with LibreELEC 8.0.1 still working)

    Kodi crashes usually do create a crash log in /storage/.kodi/temp.


    OOM conditions are logged in dmesg and journal.

    Yes, there are plenty of crash logs. But I'm not sure what to look for in these files. I attached the last 3.

    And dmesg (with timestamps i.s.o. uptime).



    Also there is normally one a specific message in dmesg vchiq: 0: ignoring ERROR from callback to service 83004 at the time of a crash.

    journalctl looks to give indeed some more details that might help terminate called after throwing an instance of 'std::length_error'.

    Files

    Tronsmart Ara X5 plus fried by lightning

    RPi3

    (removed from the dust with LibreELEC 8.0.1 still working)

    Not having any experience with RPIs I can't tell anything about the vchiq error.


    But some general recommendations.


    1) While looking for errors enable debug logging with:

    Code: /storage/.kodi/userdata/advancedsettings.xml
    <advancedsettings>

    <loglevel hide="false">1</loglevel> 

    <debug>

        <showloginfo>false</showloginfo> <!-- Change this to "false" to hide the on-screen debug log text -->

    </debug>

    </advancedsettings>



    2) Due to some errors in LubreELEC-Settings addon the automatic update to 9.2.6 is failing. Please update manually - or better install the test version from Testing LibreELEC-settings addon · Issue #4547 · LibreELEC/LibreELEC.tv · GitHub to test the automatic update fix.

    1) I enabled debug logging.


    2) For me LE-9.2.6 was already downloaded in ~/.update/

    (I even deleted it one or 2 times yesterday to stay on 9.2.4 a bit longer)

    Just rebooted and I'm on 9.2.6 now.


    This is b.t.w. on a different sd-card. Yesterday I made a backup and restored it on a cleanly fully (not quick) formatted sd-card.

    Also on this sd-card I already had a similar crash earlier today (before debug was enabled) at Nov 22 13:28 CET.

    Also this time when 'free-memory' became small.



    Tronsmart Ara X5 plus fried by lightning

    RPi3

    (removed from the dust with LibreELEC 8.0.1 still working)

    That didn't take long... new crash at Nov 22 18:55 CET.


    Files

    Tronsmart Ara X5 plus fried by lightning

    RPi3

    (removed from the dust with LibreELEC 8.0.1 still working)

    Beside the crash there is nothing visible in the log.


    Tvheadend is logging into /storage/.kodi/userdata/addon_data/service.tvheadend42/debug.txt. Maybe you see something there.

    I just enabled debug in TVheadend. Hopefully it gives more info with a next crash.



    Just looking at the build-environment to build packages. For net-snmp there is a patch

    ~/LibreELEC.tv/packages/addons/service/net-snmp/patches/net-snmp-0003-config.sub.patch that is changing "net-snmp-5.7.3/config.sub" while the current version in use (package.mk) is 5.8.

    Maybe this patch should be applied to have 'MemAvailable' providing the correct value?

    Tronsmart Ara X5 plus fried by lightning

    RPi3

    (removed from the dust with LibreELEC 8.0.1 still working)

    Wow, nice find thanks!


    I will have a look if I can apply those patches and compile net-snmp 5.9.



    I have a build-environment and tried to compile an 9.2.6 image, but this resulted in 9.2-devel images. Do you know if that is expected?


    Code
    git clone https://github.com/LibreELEC/LibreELEC.tv.git

    git checkout 9.2.6

    

    PREFER_PACKAGE_MIRROR=yes PROJECT=RPi ARCH=arm DEVICE=RPi2 tools/download-tool

    PROJECT=RPi ARCH=arm DEVICE=RPi2 make image
    Code
    ls -lat ~/LibreELEC.tv/target/

    

    

    -rw-rw-r--  1 <user> <user>       125 Nov 24 11:06 LibreELEC-RPi2.arm-9.2-devel-20201124042019-6bd7e98.img.gz.sha256

    -rw-rw-r--  1 <user> <user> 131958036 Nov 24 11:05 LibreELEC-RPi2.arm-9.2-devel-20201124042019-6bd7e98.img.gz

    -rw-rw-r--  1 <user> <user>       122 Nov 24 11:05 LibreELEC-RPi2.arm-9.2-devel-20201124042019-6bd7e98.tar.sha256

    -rw-rw-r--  1 <user> <user> 146513920 Nov 24 11:05 LibreELEC-RPi2.arm-9.2-devel-20201124042019-6bd7e98.tar

    -rw-r--r--  1 <user> <user> 133611520 Nov 24 11:05 LibreELEC-RPi2.arm-9.2-devel-20201124042019-6bd7e98.system

    -rw-r--r--  1 <user> <user>   7734000 Nov 24 11:04 LibreELEC-RPi2.arm-9.2-devel-20201124042019-6bd7e98.kernel

    Tronsmart Ara X5 plus fried by lightning

    RPi3

    (removed from the dust with LibreELEC 8.0.1 still working)

    You always get a devel-DATE-GITHASH image. Only official release builds do have official release numbers.

    I created a patch file and compiled net-snmp again but don't see a change in behavior.



    Code
    cd ~/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.2-devel/.stamps

    rm -r net-snmp

    cd  ~/LibreELEC.tv/packages/addons/service/net-snmp/patches

    vi net-snmp-0004-memavailable.patch

    cp ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk.orig

    sed -i 's/PKG_VERSION="5.8"/PKG_VERSION="5.9"/g' ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk

    sed -i 's/PKG_SHA256="b2fc.*"/PKG_SHA256="04303a66f85d6d8b16d3cc53bde50428877c82ab524e17591dfceaeb94df6071"/g' ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk

    sed -i 's/PKG_REV="106"/PKG_REV="107"/g' ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk

    cd ~/LibreELEC.tv/

    PROJECT=RPi ARCH=arm DEVICE=RPi2 scripts/create_addon net-snmp




    Files

    Tronsmart Ara X5 plus fried by lightning

    RPi3

    (removed from the dust with LibreELEC 8.0.1 still working)

    A new object memSysAvail is created by the patch. Did you update your UCD-SNMP-MIB.txt?


    BTW: net-snmp-0002-net-snmp-create-v3-user.in.patch and net-snmp-0003-config.sub.patch should be updated to apply clean to 5.9

    Yes I created a new .zip file with all the add-on files.


    I re-created the patches, now they get applied without issues.


    I added again the additional failure logging mentioned here. I don't see error messages if I run the following:


    ./snmpd -f -Lf /storage/.kodi/temp/snmpd.log -C -c /storage/.kodi/userdata/addon_data/service.net-snmp/share/snmp/snmpd.conf -M /storage/.kodi/addons/service.net-snmp/share/snmp/mibs -p /var/run/snmpd.pid -V

    or

    ./snmpd -f -Leo -C -c /storage/.kodi/userdata/addon_data/service.net-snmp/share/snmp/snmpd.conf -M /storage/.kodi/addons/service.net-snmp/share/snmp/mibs -p /var/run/snmpd.pid -V

    (not any error message...)


    The logging I get (due to the -V option):

    Tronsmart Ara X5 plus fried by lightning

    RPi3

    (removed from the dust with LibreELEC 8.0.1 still working)

    EDIT: According to Wikipedia you do need SNMPv2c to read 64bit counters.


    No Idea. Wild guess:


    1) "Some" additional configuration required on server or client side. Having no experience with snmp I can't help.


    2) The single patch is not enough to get memSysAvail. In this case try building the git HEAD.

    Edit,

    Finished setting up MRTG with the new OID.

    (I kept the 'total memory - free memory' graph also.)

    I did find a config issue in TVH causing recordings to fail due to higher than configured assigned directory size used after an 'on demand' recording.

    But don't see a direct relation with the crashed of kodi.


    (adding a graph from the last ~day, and patches used toward the end of this post)

    /Edit



    Yes, you found the issue! Indeed version 2c is required.

    Now I need to adjust the MRTG config to make use of the new OID to display the actual used memory (memTotalRealX - memSysAvail).


    Here the output all with version 2c without the additional patch (current latest official package revision 106)



    And with the patches for memSysAvail (custom package revision 109):


    Code
    # free

                  total        used        free      shared  buff/cache   available

    Mem:         765828      347828       32876       12076      385124      344704

    Swap:             0           0           0


    Edit

    Here a graph including 'Used Memory' obtained by substracting the new 'memSysAvail' OID from the total memory.

    And the patches used for 5.9 are in '5.9 patches.zip'.


    Changes to 'packages/addons/service/net-snmp/package.mk':

    Code
    sed -i 's/PKG_VERSION="5.8"/PKG_VERSION="5.9"/g' ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk

    

    sed -i 's/PKG_SHA256="b2fc.*"/PKG_SHA256="04303a66f85d6d8b16d3cc53bde50428877c82ab524e17591dfceaeb94df6071"/g' ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk

    

    sed -i 's/PKG_REV="106"/PKG_REV="107"/g' ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk

    /Edit

    Images

    Files

    Tronsmart Ara X5 plus fried by lightning

    RPi3

    (removed from the dust with LibreELEC 8.0.1 still working)

    Edited once, last by RomMon: Adding used patches and a graph ().