LE9 on RaspberryPi 3 freezes - running out of memory

the-unknown · March 7, 2019 at 8:39 PM

Hey,

I am using LibreELEC on a RaspberryPi 3 (regular, not the "b+" version).

Right now it runs the following software:

LibreELEC 9.0.1
TVHeadend Server & Client (installed as Kodi Addon)
Linuxserver.io Docker (installed as Kodi Addon)
Linuxserver.io ngnix/letsencrypt (installed as Kodi Addon)
Linuxserver.io MariaDB (installed as Kodi Addon)
Linuxserver.io Portainer (installed as Kodi Addon)
Linuxserver.io Nextcloud (installed as Kodi Addon)
two additional custom docker with custom Python-Scripts.

The following hardware is connected to the system:

1TB external HDD connected through a USB-Hub with external power supply
DVB-C2 USB Tuner for TVHeadend, connected through the same external USB-Hub

The system is usually stable, as long as I don't try to perform any "heavy" task such as:
Using the Nextcloud docker to open a folder with more than 10 images (thumbnail generation) or performing a system backup with the integrated backup solution.

However, these tasks don't always crash the system. Sometimes these tasks run completely fine and sometimes the system crashes "out of the blue" while no tasks are running at all (e.g. in the middle of the night).

If the system crashes, most software just freezes. Kodi does not respond to any input, ssh disconnects with "timeout/host not reachable". All the dockers are unreachable.

But strangely, TVHeadend still does it's job like recording a show due to a pre-set timer, its webinterface still works and I can stream a tv channel to a different client.

Things I have tested:
I used SD cards of two different manufactures (Samsung Evo and SanDisk). <-- freezes occur with both cards.

I switched Hardware: Tested an two different Raspberry Pi 3 <-- freezes occur with both hardwares.

I tested two different power adapters, both made for the RasPi (no phone chargers) with 5V and 3000mA <-- freezes occur with both power supplies.

I have these freezes everyday. I used to run OSMC earlier but with it I never had something similar before. Kodi used to crash if I used Nexcloud too heavily, once in a while, but the system never became unusable. Kodi just restarted itself and that's it. Happened maybe once a month.

This is the kodi.log right after the most recent crash, which was caused by Nextcloud.

I simply checked some settings on the Nextcloud-Webinterface when the system crashed.

External Content pastebin.com

Content embedded from external sources will not be displayed without your consent.

Through the activation of external content, you agree that personal data may be transferred to third party platforms. We have provided more information on this in our privacy policy.

This is the php-error log of the nextcloud docker:

Code

[07-Mar-2019 06:08:31] NOTICE: fpm is running, pid 289
[07-Mar-2019 06:08:31] NOTICE: ready to handle connections
[07-Mar-2019 07:07:24] NOTICE: Terminating ...
[07-Mar-2019 07:07:24] NOTICE: exiting, bye-bye!
[07-Mar-2019 07:09:25] NOTICE: fpm is running, pid 290
[07-Mar-2019 07:09:25] NOTICE: ready to handle connections
[07-Mar-2019 16:06:31] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

And this the nginx log form the nextcloud docker:

Code

2019/03/03 14:48:04 [error] 308#308: *794 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 172.18.0.1, server: _, request: "GET /nextcloud/owncloud/status.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "nextcloud.example.com:443"
2019/03/03 14:48:35 [error] 308#308: *797 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 172.18.0.1, server: _, request: "GET /nextcloud/status.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "nextcloud.example.com:443"
2019/03/03 14:48:36 [error] 308#308: *800 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 172.18.0.1, server: _, request: "GET /nextcloud/owncloud/status.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "nextcloud.example.com:443"
2019/03/03 15:17:45 [error] 308#308: *848 open() "/config/www/nextcloud/favicon.ico" failed (2: No such file or directory), client: 172.18.0.1, server: _, request: "GET /favicon.ico HTTP/1.1", host: "nextcloud.example.com:443"
2019/03/07 15:55:20 [error] 307#307: *1 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught Doctrine\DBAL\DBALException: Failed to connect to the database: An exception occured in driver: SQLSTATE[HY000] [2002] Host is unreachable in /config/www/nextcloud/lib/private/DB/Connection.php:64
Stack trace:
#0 /config/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect()
#1 /config/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion()
#2 /config/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform()
#3 /config/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(623): Doctrine\DBAL\Connection->getDatabasePlatform()
#4 /config/www/nextcloud/lib/private/DB/Connection.php(151): Doctrine\DBAL\Connection->setTransactionIsolation(2)
#5 /config/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DriverManager.php(172): OC\DB\Connection->__construct(Array,...
PHP message: PHP Fatal error:  Uncaught Doctrine\DBAL\DBALException: Failed to connect to the database: An exception occured in driver: SQLSTATE[HY000] [2002] Host is unreachable in /config/www/nextcloud/lib/private/DB/Connection.php:64

Display More

I really appreciate any help, as I have no clue what else could cause this.

the-unknown · March 8, 2019 at 6:10 AM

I have just noticed, that the system crashes always when I try to do a backup using the build-in backup mechanism.

I will now enable debug mode and try to do a backup again and then post a new kodi.log as soon as the system crashes.

the-unknown · March 8, 2019 at 7:17 AM

Ok, so with debug enabled, I see that the memory runs completely full. The pi freezes at about 5MB of memory left.

This is a fresh kodi.log with debug enabled. Extracted from SD card right after the crash

External Content pastebin.com

Content embedded from external sources will not be displayed without your consent.

Through the activation of external content, you agree that personal data may be transferred to third party platforms. We have provided more information on this in our privacy policy.

Before the crash, I made these two:

This is the dmesg

http://ix.io/1crj

This is the journalctl

http://ix.io/1crk

Does anyone has an idea?

the-unknown · March 9, 2019 at 11:40 AM

no one?

**Iridium** · March 9, 2019 at 12:19 PM Official Post

Well I think you have answered your own question. It has run out of memory and cannot swap anything to HDD.

LE is designed as a just enough for Kodi and such has no need of a swap file. Considering what you are running is like asking a Mini Cooper to outperform a Ferrari.

Your only real option is to offload some of the stuff to another machine or compile LE with swap support added.

**chewitt** · March 9, 2019 at 2:35 PM Official Post

"WARNING: [pool www] server reached pm.max_children setting (5), consider raising it"

^ that setting needs to be raised in the php-fpm config, but whether that's the cause or simply a symptom of simply running out of RAM is anyone's guess. For the quantity of containers you're running an RPi3 is inadequate. You'd be a lot better off with a basic NUC or a basic Synology NAS and then use the RPi3 as a dumb client device.

the-unknown · March 11, 2019 at 11:15 PM

Well I read so many people saying that LibreElec and Nextcloud on a RasPi3 run well together, using the docker containers.

Therefore I was wandering if I am doing anything wrong.

Do you know any stable configuration for running Nextcloud and LibreElec on a RasPi3 ?

**Da Flex** · March 11, 2019 at 11:56 PM Official Post

I think it's more about what you do with Nextcloud. There are always scenarios, when you will run out of memory when using Nextcloud. User chewitt describes the most performant solution, but if you have to stay with your current hardware, you can try to install an external swap partition (SSD) to expand the RAM. I don't suggest it, because it's probably slow.