Posts by RomMon

  • Can timeout command be added to LE?

    Thanks.

    I was looking for a solution that shows the APR requests/replies as I'm struggling with an issue that is only present for limited time (~15-30min).

    During that time I'm not able to connect to the LE RPi3 from one pc (no ping, no ssh), but works ok from another.

    Suspect my WiFi-Mesh network is causing some asymmetric behaviour after a 'mobility event'.

    I compiled a new LE-9.2.8 image with the timeout command available.

    Waiting for a next event...

  • Current journalctl logging few hours only. Possible to extent?

    Is it possible to extent the logging of journalctl to a day or longer?

    Or make it permanent (e.g. similar to mkdir -p /var/log/journal)?

    (I tried mkdir ~/.kodi/temp/journal and restart journald with systemctl restart systemd-journald.service)

    found the logs in /run/log/journal/<dir>

    But is covering the same as

    Code
    # journalctl |head -n 1
-- Logs begin at Thu 2022-09-15 13:18:40 CEST, end at Thu 2022-09-15 18:22:02 CEST. --

    Edit:

    Forgot to add, I'm on LE-9.2.8 on a RPi3b

  • SNMP OIDs for obtaining or calculating used memory

    Hi,

    Just an update on my findings so far.

    In the last days of last year I started to use Plex live TV & PVR i.s.o. TVHeadend (mainly for a daily recording of new and weather).

    In the graph you can see that the constant increase of memory usages moved from Kodi to Plex Media Server...

    I also found a memory release issue with SABnzb-2, not releasing its memory.

    In a week or so I hope to move to a x86_64 system.

  • Thoradia Add-ons

    nzbget is indeed an option. Used that in the past on an ADSL router.

    I did a downgrade to SABnzb version 1.1.1 and see it uses much less memory, and as a result the not freed memory use much less of an issue.

    It does look to have the same issue unfortunately.

  • Thoradia Add-ons

    In my script I also capture the content of /tmp with df -hT | grep 'Filesystem\|tmpfs' >> $dumpfile.

    Verified the value for /tmp, and this is 0% used for the last event.

    In my config, the download folders, and in special 'wait_for_dfolder' is selected, and I have one server configured.

    Don't think something else non-default is selected.

    I do have hd-idle running with the following parameters hd-idle -i 120 -l /storage/.kodi/temp/hd-idle.log

    The hd-idle logging indicates the disk was spinning-down a number of times during the download period.

    Download period (incl. post processing) 2020-12-31 01:45:46 to 2020-12-31 04:03:09.

    below the logs (with 2 spin-up events before start, and after DL).

    Based on the 'stopped' period, I think I better increase the period before spinning-down to 300 seconds (i.s.o. 120 seconds now).


    Edit test with hd-idle set to 300 seconds idle time.

    Still same behaviour:

    With a idle time of 300 s the hd isn't spinning-down/up during the download/post-processing.

    DL was from 19:11:34 to 20:15:07

    hd-idle.log:

    Code
    date: 2021-01-01, time: 19:11:43, disk: sda, running: 331, stopped: 781
date: 2021-01-01, time: 23:03:15, disk: sda, running: 4171, stopped: 9721

  • Thoradia Add-ons

    The download goes to a USB connected device:

    Temporary Download Folder:

    /var/media/sda1-usb-ST1000VN_000-1HJ/downloads//incomplete

    Completed Download Folder:

    /var/media/sda1-usb-ST1000VN_000-1HJ/downloads/

    A 'systemctl restart service.sabnzbd.service' corrects the issue.

    I only recently started monitoring memory usage, and observed the same issue with all 3 downloads I started.

  • Thoradia Add-ons

    I'm using SABnzb 2.3.9 from the Thoradia add-ons on LibreELEC 9.2.6 on a RPi3b.

    This works very well except that it appears that SABnzb doesn't release its memory after download/par processing/unpacking.

    I'm monitoring memory usage for a high amount of Kodi crashes (still an issue), and now also see that SABnzb doesn't release its memory after use.

    The graphs below are made from the output of 'top' sorted by memory usage collected every 30 minutes.

    Graph of first occurrence:

    (what you see it total memory usage in % in blue, Kodi in orange, Python in grey, and TVH in yellow. Added with some additional time events including the actual download period of SABnzb. And time goes from right t=0 to left. Just look at the most left part for the grey line.)

    Graph of second occurrence:

    (In above graph again total memory usage in % in blue, Kodi in purple/orange, Python in grey, and TVH in yellow. Added with some additional time events including the actual download period of SABnzb. Look at the grey line which doesn't go down after the download/processing finished.)


    Just wonder of other have observed the same...

    Happy NewYear All.

  • SNMP OIDs for obtaining or calculating used memory

    There was a new crash of kodi (on 14-Dec at 21:54:14).

    This time I 'saw' it happen.

    • The time on the LCD was 2 minutes behind.
    • Login via ssh took a long time, and by the time I had a login prompt I was only able to execute one command before kodi crased.

    dmesg has more messages than with previous crashes starting with

    [Mon Dec 14 21:51:54 CET 2020] rcu: INFO: rcu_sched self-detected stall on CPU

    Memory utilization (system, kodi, python and tvheadend):

    The daily and weelkly MRTG graphs:

  • SNMP OIDs for obtaining or calculating used memory

    Yes, indeed. Even with 3% memory usages increase each day it should run for a week or 2, before it is getting into problems.

    And I might have found a reason for the crashes.

    Planning to leave Plex Media Server (PMS) stopped for ~ a week or so longer to see if the frequent crashes stopped.

    On the Plex Media Server (PMS) forum, for ARM, I found the following note "There are some unresolved memory issues with the DLNA server support (at least on ARM). Disabling DLNA is currently recommended."

    In the PMS logs for the 4 crashes I checked I do see "DEBUG - NetworkServiceBrowser: Parsing SSDP schema for xxx" reasonably close to when the crashes happen (in the minute before or minute after a crash. I have not yet started PMS to verify the setting, but likely it is enabled (as ~ indicated by the log messages).


    Also with PMS stopped there was a memory usages increase of 3%/day for 2 days, and it nearly flattened for the last 2 days.

    Lets see in a few days.

    Edit 2020-Dec-11, Dec-13:

    I have not observed a crash after stopping Plex Media Server.

    Now thinking to start PMS again this weekend. Will leave it as it is for some time.

    There is still an increase of about (average) 1.5% a day over the last (nearly) 7 days.

    Compared the Used Memory (calculated from details obtained from 'free -k') and memory used by kodi.bin, python, tvheadend (obtained from the memory column of 'top') every 30min (after stopping PMS).

  • SNMP OIDs for obtaining or calculating used memory

    For the crash on 1-Dec 22:15 yes, see below. For the crash on 2-Dec 7:42 I forgot to obtain the log file via SMB.

    (at the time I took the logs one of my tuners was continuously generating errors in dmesg that took my attention. A power-cycle of the tuner solved that)

    Code
    Dec 01 22:09:41 LibreELEC connmand[314]: ntp: adjust (slew): +0.000045 sec
Dec 01 22:14:51 LibreELEC kodi.sh[613]: terminate called after throwing an instance of 'std::length_error'
Dec 01 22:14:51 LibreELEC kodi.sh[613]:   what():  basic_string::_S_create
Dec 01 22:14:55 LibreELEC tvheadend[276]: htsp: 127.0.0.1 [ admin | Kodi Media Center ]: Disconnected
Dec 01 22:14:55 LibreELEC kernel: vchiq: 0: ignoring ERROR from callback to service 5004
Dec 01 22:15:00 LibreELEC kodi.sh[613]: Aborted (core dumped)
Dec 01 22:15:10 LibreELEC kodi.sh[613]: Crash report available at /storage/.kodi/temp/kodi_crashlog_20201201221502.log

    I have not seen a OOM error so far.

    Yesterday I stopped the Plex server (systemctl stop plex.service) to see if that makes a difference.

    (Before I had MRTG running I did that, but still had a crash.)

  • SNMP OIDs for obtaining or calculating used memory

    MRTG with the new 'Available Memory' OID is now running a number of days.

    The below graph shows the weekly graph of the Used memory in percentage.

    There is constantly an increase of memory usage, but I was not able to identify a process (top -b -o +%MEM | head -n 20) that could be responsible for this increase.

    Graph details:

    - MRTG started last Sunday evening.

    - The RPi3 was running mostly idle for two days. During this period memory usage increased about 3 percent per day.

    - Kodi crashed on Tuesday (1-Dec-2020) evening at 2215. This decreased memory usage with about 6 to 7 percent. After this memory usage is flat for a long time.

    - Wednesday evening I played a bit with the Plex server and started a scan of my library which was still busy when I went to sleep.

    - Thursday morning at 7:42am there was another Kodi crash.

    - On Thursday at about 11am started a download that lasted till about 5pm

    - After the download memory increased about 8 percent a day.

    - At 2pm memory usage decreased with about 6 percent, but have not seen a reason (see spoiler for details).

    Top output 4-Dec-2020 2pm and 2:30pm

    - 13:59:33 up 6 days, 11:42, 1 user, load average: 0.03, 0.08, 0.10

    Tasks: 117 total, 1 running, 116 sleeping, 0 stopped, 0 zombie

    %Cpu(s): 2.9 us, 2.9 sy, 1.4 ni, 92.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

    MiB Mem : 747.9 total, 48.0 free, 486.5 used, 213.3 buff/cache

    MiB Swap: 0.0 total, 0.0 free, 0.0 used. 186.5 avail Mem

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

    25482 root 20 0 673628 179296 21224 S 12.5 23.4 620:12.02 kodi.bin

    608 root 30 10 340016 95884 1264 S 0.0 12.5 111:23.94 python

    405 root 20 0 328660 54852 8544 S 0.0 7.2 41:43.79 Plex Media Serv

    637 root 35 15 194612 51220 4180 S 6.2 6.7 36:04.24 Plex Script Hos

    9545 root 20 0 161860 15648 2132 S 0.0 2.0 266:01.81 tvheadend

    22220 root 20 0 5116 3968 3572 S 0.0 0.5 0:00.21 sshd

    581 root 20 0 30396 3336 1796 S 0.0 0.4 0:16.77 smbd

    916 root 20 0 125428 3128 448 S 0.0 0.4 8:26.13 Plex Tuner Serv

    1924 root 20 0 7312 2696 880 S 0.0 0.4 11:24.85 snmpd

    598 root 20 0 6084 2392 600 S 0.0 0.3 2:07.96 python

    22424 root 20 0 6020 2128 1700 R 6.2 0.3 0:00.03 top

    579 root 20 0 17560 2096 1060 S 0.0 0.3 1:39.76 nmbd

    2031 root 20 0 22540 1968 1564 S 0.0 0.3 0:13.87 systemd-journal

    Used memory %: 75.0576


    - 14:29:34 up 6 days, 12:12, 1 user, load average: 0.38, 0.78, 0.75

    Tasks: 124 total, 1 running, 123 sleeping, 0 stopped, 0 zombie

    %Cpu(s): 4.3 us, 2.9 sy, 1.4 ni, 90.0 id, 1.4 wa, 0.0 hi, 0.0 si, 0.0 st

    MiB Mem : 747.9 total, 32.2 free, 441.1 used, 274.6 buff/cache

    MiB Swap: 0.0 total, 0.0 free, 0.0 used. 230.3 avail Mem

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

    25482 root 20 0 673884 167368 9260 S 6.2 21.9 624:34.58 kodi.bin

    608 root 30 10 340016 95884 1264 S 0.0 12.5 111:33.71 python

    405 root 20 0 328660 57636 11328 S 6.2 7.5 41:50.92 Plex Media Serv

    637 root 35 15 194612 48596 1556 S 0.0 6.3 36:11.31 Plex Script Hos

    9545 root 20 0 162884 15980 2060 S 6.2 2.1 274:32.91 tvheadend

    581 root 20 0 30396 3328 1788 S 0.0 0.4 0:16.78 smbd

    916 root 20 0 125428 3128 448 S 0.0 0.4 8:27.78 Plex Tuner Serv

    1924 root 20 0 7312 2684 868 S 0.0 0.4 11:27.06 snmpd

    2031 root 20 0 23056 2576 2172 S 0.0 0.3 0:14.25 systemd-journal

    22220 root 20 0 5116 2408 2012 S 0.0 0.3 0:00.23 sshd

    598 root 20 0 6084 2392 600 S 0.0 0.3 2:08.32 python

    579 root 20 0 17560 2016 980 S 0.0 0.3 1:39.95 nmbd

    23006 root 20 0 6028 2008 1580 R 6.2 0.3 0:00.03 top

    Used memory % 69.1941

    Enabling debug with TVH didn't additional information at the time of the crash:

  • SNMP OIDs for obtaining or calculating used memory

    Edit,

    Finished setting up MRTG with the new OID.

    (I kept the 'total memory - free memory' graph also.)

    I did find a config issue in TVH causing recordings to fail due to higher than configured assigned directory size used after an 'on demand' recording.

    But don't see a direct relation with the crashed of kodi.

    (adding a graph from the last ~day, and patches used toward the end of this post)

    /Edit


    Yes, you found the issue! Indeed version 2c is required.

    Now I need to adjust the MRTG config to make use of the new OID to display the actual used memory (memTotalRealX - memSysAvail).

    Here the output all with version 2c without the additional patch (current latest official package revision 106)

    And with the patches for memSysAvail (custom package revision 109):

    Code
    # free
              total        used        free      shared  buff/cache   available
Mem:         765828      347828       32876       12076      385124      344704
Swap:             0           0           0

    Edit

    Here a graph including 'Used Memory' obtained by substracting the new 'memSysAvail' OID from the total memory.

    And the patches used for 5.9 are in '5.9 patches.zip'.

    Changes to 'packages/addons/service/net-snmp/package.mk':

    Code
    sed -i 's/PKG_VERSION="5.8"/PKG_VERSION="5.9"/g' ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk

sed -i 's/PKG_SHA256="b2fc.*"/PKG_SHA256="04303a66f85d6d8b16d3cc53bde50428877c82ab524e17591dfceaeb94df6071"/g' ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk

sed -i 's/PKG_REV="106"/PKG_REV="107"/g' ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk

    /Edit

  • SNMP OIDs for obtaining or calculating used memory

    Yes I created a new .zip file with all the add-on files.

    I re-created the patches, now they get applied without issues.

    I added again the additional failure logging mentioned here. I don't see error messages if I run the following:

    ./snmpd -f -Lf /storage/.kodi/temp/snmpd.log -C -c /storage/.kodi/userdata/addon_data/service.net-snmp/share/snmp/snmpd.conf -M /storage/.kodi/addons/service.net-snmp/share/snmp/mibs -p /var/run/snmpd.pid -V

    or

    ./snmpd -f -Leo -C -c /storage/.kodi/userdata/addon_data/service.net-snmp/share/snmp/snmpd.conf -M /storage/.kodi/addons/service.net-snmp/share/snmp/mibs -p /var/run/snmpd.pid -V

    (not any error message...)

    The logging I get (due to the -V option):

  • SNMP OIDs for obtaining or calculating used memory

    I created a patch file and compiled net-snmp again but don't see a change in behavior.


    Code
    cd ~/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.2-devel/.stamps
rm -r net-snmp
cd  ~/LibreELEC.tv/packages/addons/service/net-snmp/patches
vi net-snmp-0004-memavailable.patch
cp ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk.orig
sed -i 's/PKG_VERSION="5.8"/PKG_VERSION="5.9"/g' ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk
sed -i 's/PKG_SHA256="b2fc.*"/PKG_SHA256="04303a66f85d6d8b16d3cc53bde50428877c82ab524e17591dfceaeb94df6071"/g' ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk
sed -i 's/PKG_REV="106"/PKG_REV="107"/g' ~/LibreELEC.tv/packages/addons/service/net-snmp/package.mk
cd ~/LibreELEC.tv/
PROJECT=RPi ARCH=arm DEVICE=RPi2 scripts/create_addon net-snmp