LE9 on RaspberryPi 3 freezes - running out of memory

1st Official Post

    Hey,

    I am using LibreELEC on a RaspberryPi 3 (regular, not the "b+" version).

    Right now it runs the following software:

    • LibreELEC 9.0.1
    • TVHeadend Server & Client (installed as Kodi Addon)
    • Linuxserver.io Docker (installed as Kodi Addon)
    • Linuxserver.io ngnix/letsencrypt (installed as Kodi Addon)
    • Linuxserver.io MariaDB (installed as Kodi Addon)
    • Linuxserver.io Portainer (installed as Kodi Addon)
    • Linuxserver.io Nextcloud (installed as Kodi Addon)
    • two additional custom docker with custom Python-Scripts.

    The following hardware is connected to the system:

    • 1TB external HDD connected through a USB-Hub with external power supply
    • DVB-C2 USB Tuner for TVHeadend, connected through the same external USB-Hub

    The system is usually stable, as long as I don't try to perform any "heavy" task such as:
    Using the Nextcloud docker to open a folder with more than 10 images (thumbnail generation) or performing a system backup with the integrated backup solution.

    However, these tasks don't always crash the system. Sometimes these tasks run completely fine and sometimes the system crashes "out of the blue" while no tasks are running at all (e.g. in the middle of the night).

    If the system crashes, most software just freezes. Kodi does not respond to any input, ssh disconnects with "timeout/host not reachable". All the dockers are unreachable.

    But strangely, TVHeadend still does it's job like recording a show due to a pre-set timer, its webinterface still works and I can stream a tv channel to a different client.

    Things I have tested:
    I used SD cards of two different manufactures (Samsung Evo and SanDisk). <-- freezes occur with both cards.

    I switched Hardware: Tested an two different Raspberry Pi 3 <-- freezes occur with both hardwares.

    I tested two different power adapters, both made for the RasPi (no phone chargers) with 5V and 3000mA <-- freezes occur with both power supplies.

    I have these freezes everyday. I used to run OSMC earlier but with it I never had something similar before. Kodi used to crash if I used Nexcloud too heavily, once in a while, but the system never became unusable. Kodi just restarted itself and that's it. Happened maybe once a month.

    This is the kodi.log right after the most recent crash, which was caused by Nextcloud.

    I simply checked some settings on the Nextcloud-Webinterface when the system crashed.

    External Content pastebin.com
    Content embedded from external sources will not be displayed without your consent.
    Through the activation of external content, you agree that personal data may be transferred to third party platforms. We have provided more information on this in our privacy policy.

    This is the php-error log of the nextcloud docker:

    Code
    [07-Mar-2019 06:08:31] NOTICE: fpm is running, pid 289
[07-Mar-2019 06:08:31] NOTICE: ready to handle connections
[07-Mar-2019 07:07:24] NOTICE: Terminating ...
[07-Mar-2019 07:07:24] NOTICE: exiting, bye-bye!
[07-Mar-2019 07:09:25] NOTICE: fpm is running, pid 290
[07-Mar-2019 07:09:25] NOTICE: ready to handle connections
[07-Mar-2019 16:06:31] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

    And this the nginx log form the nextcloud docker:

    I really appreciate any help, as I have no clue what else could cause this.

    I have just noticed, that the system crashes always when I try to do a backup using the build-in backup mechanism.

    I will now enable debug mode and try to do a backup again and then post a new kodi.log as soon as the system crashes.

    Ok, so with debug enabled, I see that the memory runs completely full. The pi freezes at about 5MB of memory left.

    This is a fresh kodi.log with debug enabled. Extracted from SD card right after the crash

    External Content pastebin.com
    Content embedded from external sources will not be displayed without your consent.
    Through the activation of external content, you agree that personal data may be transferred to third party platforms. We have provided more information on this in our privacy policy.

    Before the crash, I made these two:

    This is the dmesg

    http://ix.io/1crj

    This is the journalctl

    http://ix.io/1crk

    Does anyone has an idea?

    • Official Post

    Well I think you have answered your own question. It has run out of memory and cannot swap anything to HDD.


    LE is designed as a just enough for Kodi and such has no need of a swap file. Considering what you are running is like asking a Mini Cooper to outperform a Ferrari.

    Your only real option is to offload some of the stuff to another machine or compile LE with swap support added.

    • Official Post

    "WARNING: [pool www] server reached pm.max_children setting (5), consider raising it"

    ^ that setting needs to be raised in the php-fpm config, but whether that's the cause or simply a symptom of simply running out of RAM is anyone's guess. For the quantity of containers you're running an RPi3 is inadequate. You'd be a lot better off with a basic NUC or a basic Synology NAS and then use the RPi3 as a dumb client device.

    Well I read so many people saying that LibreElec and Nextcloud on a RasPi3 run well together, using the docker containers.

    Therefore I was wandering if I am doing anything wrong.

    Do you know any stable configuration for running Nextcloud and LibreElec on a RasPi3 ?

    • Official Post

    I think it's more about what you do with Nextcloud. There are always scenarios, when you will run out of memory when using Nextcloud. User chewitt describes the most performant solution, but if you have to stay with your current hardware, you can try to install an external swap partition (SSD) to expand the RAM. I don't suggest it, because it's probably slow.