Posts by RomMon

    Hi,


    Just an update on my findings so far.

    In the last days of last year I started to use Plex live TV & PVR i.s.o. TVHeadend (mainly for a daily recording of new and weather).


    In the graph you can see that the constant increase of memory usages moved from Kodi to Plex Media Server...


    I also found a memory release issue with SABnzb-2, not releasing its memory.


    In a week or so I hope to move to a x86_64 system.

    nzbget is indeed an option. Used that in the past on an ADSL router.


    I did a downgrade to SABnzb version 1.1.1 and see it uses much less memory, and as a result the not freed memory use much less of an issue.

    It does look to have the same issue unfortunately.

    In my script I also capture the content of /tmp with df -hT | grep 'Filesystem\|tmpfs' >> $dumpfile.

    Verified the value for /tmp, and this is 0% used for the last event.


    In my config, the download folders, and in special 'wait_for_dfolder' is selected, and I have one server configured.

    Don't think something else non-default is selected.


    I do have hd-idle running with the following parameters hd-idle -i 120 -l /storage/.kodi/temp/hd-idle.log

    The hd-idle logging indicates the disk was spinning-down a number of times during the download period.

    Download period (incl. post processing) 2020-12-31 01:45:46 to 2020-12-31 04:03:09.

    below the logs (with 2 spin-up events before start, and after DL).

    Based on the 'stopped' period, I think I better increase the period before spinning-down to 300 seconds (i.s.o. 120 seconds now).



    Edit test with hd-idle set to 300 seconds idle time.


    Still same behaviour:


    With a idle time of 300 s the hd isn't spinning-down/up during the download/post-processing.

    DL was from 19:11:34 to 20:15:07

    hd-idle.log:

    Code
    date: 2021-01-01, time: 19:11:43, disk: sda, running: 331, stopped: 781
    date: 2021-01-01, time: 23:03:15, disk: sda, running: 4171, stopped: 9721

    The download goes to a USB connected device:

    Temporary Download Folder:

    /var/media/sda1-usb-ST1000VN_000-1HJ/downloads//incomplete

    Completed Download Folder:

    /var/media/sda1-usb-ST1000VN_000-1HJ/downloads/


    A 'systemctl restart service.sabnzbd.service' corrects the issue.

    I only recently started monitoring memory usage, and observed the same issue with all 3 downloads I started.

    I'm using SABnzb 2.3.9 from the Thoradia add-ons on LibreELEC 9.2.6 on a RPi3b.


    This works very well except that it appears that SABnzb doesn't release its memory after download/par processing/unpacking.

    I'm monitoring memory usage for a high amount of Kodi crashes (still an issue), and now also see that SABnzb doesn't release its memory after use.


    The graphs below are made from the output of 'top' sorted by memory usage collected every 30 minutes.


    Graph of first occurrence:

    (what you see it total memory usage in % in blue, Kodi in orange, Python in grey, and TVH in yellow. Added with some additional time events including the actual download period of SABnzb. And time goes from right t=0 to left. Just look at the most left part for the grey line.)


    Graph of second occurrence:

    (In above graph again total memory usage in % in blue, Kodi in purple/orange, Python in grey, and TVH in yellow. Added with some additional time events including the actual download period of SABnzb. Look at the grey line which doesn't go down after the download/processing finished.)



    Just wonder of other have observed the same...


    Happy NewYear All.

    There was a new crash of kodi (on 14-Dec at 21:54:14).

    This time I 'saw' it happen.

    • The time on the LCD was 2 minutes behind.
    • Login via ssh took a long time, and by the time I had a login prompt I was only able to execute one command before kodi crased.

    dmesg has more messages than with previous crashes starting with

    [Mon Dec 14 21:51:54 CET 2020] rcu: INFO: rcu_sched self-detected stall on CPU


    Memory utilization (system, kodi, python and tvheadend):


    The daily and weelkly MRTG graphs:

    Yes, indeed. Even with 3% memory usages increase each day it should run for a week or 2, before it is getting into problems.


    And I might have found a reason for the crashes.

    Planning to leave Plex Media Server (PMS) stopped for ~ a week or so longer to see if the frequent crashes stopped.


    On the Plex Media Server (PMS) forum, for ARM, I found the following note "There are some unresolved memory issues with the DLNA server support (at least on ARM). Disabling DLNA is currently recommended."


    In the PMS logs for the 4 crashes I checked I do see "DEBUG - NetworkServiceBrowser: Parsing SSDP schema for xxx" reasonably close to when the crashes happen (in the minute before or minute after a crash. I have not yet started PMS to verify the setting, but likely it is enabled (as ~ indicated by the log messages).



    Also with PMS stopped there was a memory usages increase of 3%/day for 2 days, and it nearly flattened for the last 2 days.

    Lets see in a few days.


    Edit 2020-Dec-11, Dec-13:

    I have not observed a crash after stopping Plex Media Server.

    Now thinking to start PMS again this weekend. Will leave it as it is for some time.


    There is still an increase of about (average) 1.5% a day over the last (nearly) 7 days.

    Compared the Used Memory (calculated from details obtained from 'free -k') and memory used by kodi.bin, python, tvheadend (obtained from the memory column of 'top') every 30min (after stopping PMS).

    For the crash on 1-Dec 22:15 yes, see below. For the crash on 2-Dec 7:42 I forgot to obtain the log file via SMB.

    (at the time I took the logs one of my tuners was continuously generating errors in dmesg that took my attention. A power-cycle of the tuner solved that)

    Code
    Dec 01 22:09:41 LibreELEC connmand[314]: ntp: adjust (slew): +0.000045 sec
    Dec 01 22:14:51 LibreELEC kodi.sh[613]: terminate called after throwing an instance of 'std::length_error'
    Dec 01 22:14:51 LibreELEC kodi.sh[613]:   what():  basic_string::_S_create
    Dec 01 22:14:55 LibreELEC tvheadend[276]: htsp: 127.0.0.1 [ admin | Kodi Media Center ]: Disconnected
    Dec 01 22:14:55 LibreELEC kernel: vchiq: 0: ignoring ERROR from callback to service 5004
    Dec 01 22:15:00 LibreELEC kodi.sh[613]: Aborted (core dumped)
    Dec 01 22:15:10 LibreELEC kodi.sh[613]: Crash report available at /storage/.kodi/temp/kodi_crashlog_20201201221502.log

    I have not seen a OOM error so far.


    Yesterday I stopped the Plex server (systemctl stop plex.service) to see if that makes a difference.

    (Before I had MRTG running I did that, but still had a crash.)

    MRTG with the new 'Available Memory' OID is now running a number of days.


    The below graph shows the weekly graph of the Used memory in percentage.


    There is constantly an increase of memory usage, but I was not able to identify a process (top -b -o +%MEM | head -n 20) that could be responsible for this increase.

    Graph details:

    - MRTG started last Sunday evening.

    - The RPi3 was running mostly idle for two days. During this period memory usage increased about 3 percent per day.

    - Kodi crashed on Tuesday (1-Dec-2020) evening at 2215. This decreased memory usage with about 6 to 7 percent. After this memory usage is flat for a long time.

    - Wednesday evening I played a bit with the Plex server and started a scan of my library which was still busy when I went to sleep.

    - Thursday morning at 7:42am there was another Kodi crash.

    - On Thursday at about 11am started a download that lasted till about 5pm

    - After the download memory increased about 8 percent a day.

    - At 2pm memory usage decreased with about 6 percent, but have not seen a reason (see spoiler for details).


    Enabling debug with TVH didn't additional information at the time of the crash:

    Edit,

    Finished setting up MRTG with the new OID.

    (I kept the 'total memory - free memory' graph also.)

    I did find a config issue in TVH causing recordings to fail due to higher than configured assigned directory size used after an 'on demand' recording.

    But don't see a direct relation with the crashed of kodi.


    (adding a graph from the last ~day, and patches used toward the end of this post)

    /Edit



    Yes, you found the issue! Indeed version 2c is required.

    Now I need to adjust the MRTG config to make use of the new OID to display the actual used memory (memTotalRealX - memSysAvail).


    Here the output all with version 2c without the additional patch (current latest official package revision 106)



    And with the patches for memSysAvail (custom package revision 109):


    Code
    # free
                  total        used        free      shared  buff/cache   available
    Mem:         765828      347828       32876       12076      385124      344704
    Swap:             0           0           0


    Edit

    Here a graph including 'Used Memory' obtained by substracting the new 'memSysAvail' OID from the total memory.

    And the patches used for 5.9 are in '5.9 patches.zip'.


    Changes to 'packages/addons/service/net-snmp/package.mk':

    Code
    sed -i 's/PKG_VERSION="5.8"/PKG_VERSION="5.9"/g' ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk
    
    sed -i 's/PKG_SHA256="b2fc.*"/PKG_SHA256="04303a66f85d6d8b16d3cc53bde50428877c82ab524e17591dfceaeb94df6071"/g' ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk
    
    sed -i 's/PKG_REV="106"/PKG_REV="107"/g' ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk

    /Edit

    Yes I created a new .zip file with all the add-on files.


    I re-created the patches, now they get applied without issues.


    I added again the additional failure logging mentioned here. I don't see error messages if I run the following:


    ./snmpd -f -Lf /storage/.kodi/temp/snmpd.log -C -c /storage/.kodi/userdata/addon_data/service.net-snmp/share/snmp/snmpd.conf -M /storage/.kodi/addons/service.net-snmp/share/snmp/mibs -p /var/run/snmpd.pid -V

    or

    ./snmpd -f -Leo -C -c /storage/.kodi/userdata/addon_data/service.net-snmp/share/snmp/snmpd.conf -M /storage/.kodi/addons/service.net-snmp/share/snmp/mibs -p /var/run/snmpd.pid -V

    (not any error message...)


    The logging I get (due to the -V option):

    I created a patch file and compiled net-snmp again but don't see a change in behavior.



    Code
    cd ~/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.2-devel/.stamps
    rm -r net-snmp
    cd  ~/LibreELEC.tv/packages/addons/service/net-snmp/patches
    vi net-snmp-0004-memavailable.patch
    cp ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk.orig
    sed -i 's/PKG_VERSION="5.8"/PKG_VERSION="5.9"/g' ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk
    sed -i 's/PKG_SHA256="b2fc.*"/PKG_SHA256="04303a66f85d6d8b16d3cc53bde50428877c82ab524e17591dfceaeb94df6071"/g' ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk
    sed -i 's/PKG_REV="106"/PKG_REV="107"/g' ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk
    cd ~/LibreELEC.tv/
    PROJECT=RPi ARCH=arm DEVICE=RPi2 scripts/create_addon net-snmp




    Wow, nice find thanks!


    I will have a look if I can apply those patches and compile net-snmp 5.9.



    I have a build-environment and tried to compile an 9.2.6 image, but this resulted in 9.2-devel images. Do you know if that is expected?


    Code
    git clone https://github.com/LibreELEC/LibreELEC.tv.git
    git checkout 9.2.6
    
    PREFER_PACKAGE_MIRROR=yes PROJECT=RPi ARCH=arm DEVICE=RPi2 tools/download-tool
    PROJECT=RPi ARCH=arm DEVICE=RPi2 make image
    Code
    ls -lat ~/LibreELEC.tv/target/
    
    
    -rw-rw-r--  1 <user> <user>       125 Nov 24 11:06 LibreELEC-RPi2.arm-9.2-devel-20201124042019-6bd7e98.img.gz.sha256
    -rw-rw-r--  1 <user> <user> 131958036 Nov 24 11:05 LibreELEC-RPi2.arm-9.2-devel-20201124042019-6bd7e98.img.gz
    -rw-rw-r--  1 <user> <user>       122 Nov 24 11:05 LibreELEC-RPi2.arm-9.2-devel-20201124042019-6bd7e98.tar.sha256
    -rw-rw-r--  1 <user> <user> 146513920 Nov 24 11:05 LibreELEC-RPi2.arm-9.2-devel-20201124042019-6bd7e98.tar
    -rw-r--r--  1 <user> <user> 133611520 Nov 24 11:05 LibreELEC-RPi2.arm-9.2-devel-20201124042019-6bd7e98.system
    -rw-r--r--  1 <user> <user>   7734000 Nov 24 11:04 LibreELEC-RPi2.arm-9.2-devel-20201124042019-6bd7e98.kernel

    I just enabled debug in TVheadend. Hopefully it gives more info with a next crash.



    Just looking at the build-environment to build packages. For net-snmp there is a patch

    ~/LibreELEC.tv/packages/addons/service/net-snmp/patches/net-snmp-0003-config.sub.patch that is changing "net-snmp-5.7.3/config.sub" while the current version in use (package.mk) is 5.8.

    Maybe this patch should be applied to have 'MemAvailable' providing the correct value?

    1) I enabled debug logging.


    2) For me LE-9.2.6 was already downloaded in ~/.update/

    (I even deleted it one or 2 times yesterday to stay on 9.2.4 a bit longer)

    Just rebooted and I'm on 9.2.6 now.


    This is b.t.w. on a different sd-card. Yesterday I made a backup and restored it on a cleanly fully (not quick) formatted sd-card.

    Also on this sd-card I already had a similar crash earlier today (before debug was enabled) at Nov 22 13:28 CET.

    Also this time when 'free-memory' became small.



    Yes, there are plenty of crash logs. But I'm not sure what to look for in these files. I attached the last 3.

    And dmesg (with timestamps i.s.o. uptime).



    Also there is normally one a specific message in dmesg vchiq: 0: ignoring ERROR from callback to service 83004 at the time of a crash.

    journalctl looks to give indeed some more details that might help terminate called after throwing an instance of 'std::length_error'.