[RPi4] Out of Memory Error

    hi, i have a rpi4 for my father. he connects using a vpn connection to my router and so he can use my tv stream. i configured everything and all looks good, but after about 1h he has to reconnect everytime. so i checked some logs and noticed, that there is probably some sort of memory leak. i ordered the 1gb version, thought thats enough, now i regret that. i assume with 2 or more gb of memory that would not occur, but its just a guess.

    the interesting part of the journal is here:

    the dmesg is attached.

    anyone an idea what we could do (software side). i already thought about asking in a repair cafe if anyone could replace the memory with a 4gb chip. but not sure if they have the equipment for smd/bga

    I doubt it's a memory leak, but without understanding the configuration of the device in more detail it's hard to comment. NB: If the problem is a leak then a 2GB or 4GB device will not fix the problem, only delay the failure.

    Run "pastekodi" and "pastecrash" and share the URL's generated.

    The 4gb version would not fix the problem, but it would be an ok workaround, because i assume if 1gb is enough for 1h watching, 4gb should be enough for 4h watching :P

    he usually watches for 2-3h

    i will provide those files/urls asap.

    edit: i just called him, i will gather those files with a remote session on friday

    Edited once, last by olze (July 17, 2024 at 3:03 PM).

    One of my Pi4 4GB that runs 24/7 and only gets reboots when updated usually starts at 9-10% memory usage and after about one week its up to 25%, i dont use it for anything else than locale files and occasionally the Retrospect addon.

    Edit; After updating to latest LE12 nightly its up from 9 to 15% over night without me using it.

    Edited once, last by MatteN (July 19, 2024 at 11:56 AM).

    Code
    RPi5:~ # uptime
 17:00:37 up 12 days, 20:36,  load average: 2.62, 2.65, 2.67
RPi5:~ # free -m
               total        used        free      shared  buff/cache   available
Mem:            8052        2177        4711         201        1461        5875
Swap:              0           0           0

    This is LE13 (master) not LE12 but it's the same Kodi version in the image (not Piers, yet). Remind me in another couple of days and let's see if memory has substantially increased.

    I'm already having a kodi memory monitoring script logging the kodi RSS size in KiBytes:

    Bash
    #!/bin/sh
L=/storage/kodi_rss.log
while sleep 60
do
  P=$(pidof kodi.bin)
  test -z "$P" && continue
  echo $(date "+%Y-%m-%d %H:%M:%S,")$((4*$(cut -f 2 -d ' ' </proc/$P/statm))) >>$L
done

    Create a script file and start it from autostart.sh in the background.

    It is for Generic with 4Ki pages. If using other values, likely 16Ki for RPI4 today, change the '4'.

    Quote from chewitt
    Code
    RPi5:~ # uptime
 17:00:37 up 12 days, 20:36,  load average: 2.62, 2.65, 2.67
RPi5:~ # free -m
               total        used        free      shared  buff/cache   available
Mem:            8052        2177        4711         201        1461        5875
Swap:              0           0           0

    This is LE13 (master) not LE12 but it's the same Kodi version in the image (not Piers, yet). Remind me in another couple of days and let's see if memory has substantially increased.

    Is this default kodi without anything "special"? If yes, i assume he must go with the 4gb version.

    ^ minor changes to log uptime and background commands else they block boot. I do have a large media library so scrolling around will consume some RAM with caching and kids do play old SNES games somtimes (although they've been sent to grandparents for the next two weeks). The rest of our family usage is rather ordinary: watching media from a NAS in the LAN and streams from a remote Tvheadend instance.

    Code
    2024-07-29 15:26:39 up 10 min 538352
2024-07-29 15:27:39 up 11 min 543008
2024-07-29 15:28:39 up 12 min 542848
2024-07-29 15:29:39 up 13 min 544544
...
2024-08-03 11:15:11 up 4 days 803584
2024-08-03 11:16:11 up 4 days 799680
2024-08-03 11:17:11 up 4 days 799888
2024-08-03 11:18:11 up 4 days 810112

    Defnitely an increase in RAM consumption over time, but figuring out what is beyond my Linux skillset.

    HiassofT  mglae any thoughts?

    Please test with those builds, hopefully they'lll help to narrow it down a bit:

    RPi4: https://www.horus.com/~hias/tmp/libr…455-ddbb512.tar
    RPi5: https://www.horus.com/~hias/tmp/libr…425-ddbb512.tar

    They are current LE12 builds and have kodi compiled with debug info plus the leak sanitizer enabled (for the self builders: include the top 2 commits from this branch https://github.com/HiassofT/Libre…le12-sanitizers and build with DEBUG=kodi)

    After stopping kodi (systemctl stop kodi) the sanitizer will output detailled info about leaked memory. For now look at the summary (run systemctl status kodi | grep SUMMARY after stopping kodi).

    First stop kodi pretty immediately after startup to get a baseline. Here on RPi5 I see about 900k leaked in a bit over 2k allocations:

    Code
    Aug 03 12:25:43 LibreELEC kodi.sh[1097]: SUMMARY: LeakSanitizer: 936782 byte(s) leaked in 2027 allocation(s).

    Then reboot, let kodi run for some time until you see increased kodi memory usage / less free memory, stop kodi and check the summary again.

    Please report the values you got for baseline and in the "increased memory" case.

    Here in local tests (just letting kodi idle on the main screen) I couldn't see much difference between 30s, 6min and 15min running time.

    After 6min I got

    Code
    Aug 03 12:22:33 LibreELEC kodi.sh[980]: SUMMARY: LeakSanitizer: 920304 byte(s) leaked in 2030 allocation(s).

    And after 15min I got

    Code
    Aug 03 13:03:04 LibreELEC kodi.sh[1198]: SUMMARY: LeakSanitizer: 920296 byte(s) leaked in 2029 allocation(s).

    Hopefully your tests will show significantly more leaked memory, then we can start looking further into the actual allocations.

    so long,

    Hias

    Forgot to add: if you test with above builds then it's best to enable "Persistent Logs" in Settings->LibreELEC->System (30 MB size should hopefully be enough) - don't forget to reboot after changing this.

    Then you can grab the full sanitizer output, including alloc stacktraces, with journalctl -u kodi -b > your-alloc-info.txt - best do this after each kodi stop and reboot afterwards so each boot contains only one kodi start/stop cycle.

    so long,

    Hias

    The thing is... i see my dad maybe 4 times a year. Thats where this machine is. So i can not flash a new image to his sd card and do a lot of stuff. I will see what is possible. I have to do all remote.

    Can i flash such an image with a command and all settings keep the same?

    You can simply download the tar to the /storage/.update/ folder and reboot - but testing remotely could be tricky if something goes south...

    so long,

    Hias

    • New

    i upgraded to the version mentioned above but i get tons of those errors and libreelec resp. kodi does not start:

    • New

    I updated the image but when i grepped for SUMMARY, nothing appeared.

    So i checked with systemctl status kodi, when its normal running. Thats the output:

    The last line is a bit irritating. Is there something else missing? When i disable kodi, it looks like this: