LE10 rpi4-4gb oom errors

Setup

• RPi4 4GB
• 12TB USB 3.0 Drive
• LE 10 beta
• Docker

I am encountering constant oom events when there is plenty of available ram. The oom-killer can fire as often as every 30 seconds, and it loves to kill kodi. I know I can tell it to avoid killing kodi, but the underlying problem would persist. I've encountered this before on 9.2.6, but not nearly as often as on the new build.

I do a fresh install of LE10, attach my external USB3 drive, add my media libraries and add any docker image that generates lots of disk io (transmission, rsync, medusa, etc), and after a few minutes the oom-killer will begin decimating my processes every few minutes. My rpi4 memory usage never really goes over 1 GB or so (out of 4GB). I have seen the oom events occur even before docker and Kodi loads. I don't believe that memory fragmentation is a problem, as I always have plenty of 4096kB blocks of DMA and HiMem. I have been beating my head against this, and I believe it is possibly related to USB3 disk writes trigger OOM reaper on Pi 4. Any suggestions on how best to try to debug this?

I really love the design philosophy of LE and I would rather not shift to a normal distro, but other users report a 64 bit kernel alleviates may oom errors. Would it even be possible for me to compile a 64 bit version of LE for the rpi4 knowing that this would force software x264 decoding and loss of widevine?

Things I have tried:

• Enabling cgroups and limiting the docker images to only 256MB (limiting works, but oom persists)
• "sysctl vm.overcommit_memory=2" This causes the system to crash during boot
• cron job of "echo 4 > /proc/sys/vm/drop_caches" every minute. This reduces the frequency of oom events, but it doesn't stop them.
• mucking about with "vm.extfrag_threshold, vm.vfs_cache_pressure, swappiness": no real change
• Adding a massive swap file: It never gets used.
• Fresh LE10 install on both a large SD card and a usb 3.0 attached ssd.
• Even without docker, I believe I have seen one or two oom events, but they are rare.

Thanks!

[12007.624637] Mem-Info:
[12007.624649] active_anon:2019 inactive_anon:108517 isolated_anon:0
[12007.624649]  active_file:71240 inactive_file:80161 isolated_file:39
[12007.624649]  unevictable:0 dirty:4 writeback:0
[12007.624649]  slab_reclaimable:10261 slab_unreclaimable:9625
[12007.624649]  mapped:48527 shmem:4136 pagetables:1389 bounce:0
[12007.624649]  free:673253 free_pcp:155 free_cma:116364
[12007.624659] Node 0 active_anon:8076kB inactive_anon:434068kB active_file:284960kB inactive_file:320644kB unevictable:0kB isolated(anon):0kB isolated(file):156kB mapped:194108kB dirty:16kB writeback:0kB shmem:16544kB writeback_tmp:0kB kernel_stack:3288kB all_unreclaimable? yes

[12007.624671] DMA free:485840kB min:16384kB low:20480kB high:24576kB reserved_highatomic:4096KB active_anon:0kB inactive_anon:0kB active_file:1352kB inactive_file:6260kB unevictable:0kB writepending:0kB present:786432kB managed:652332kB mlocked:0kB pagetables:640kB bounce:0kB free_pcp:256kB local_pcp:52kB free_cma:465456kB

[12007.624677] lowmem_reserve[]: 0 0 3187 0

[12007.624714] HighMem free:2207172kB min:512kB low:21000kB high:41488kB reserved_highatomic:0KB active_anon:8076kB inactive_anon:434068kB active_file:283520kB inactive_file:314416kB unevictable:0kB writepending:16kB present:3263488kB managed:3263488kB mlocked:0kB pagetables:4916kB bounce:0kB free_pcp:364kB local_pcp:0kB free_cma:0kB

[12007.624720] lowmem_reserve[]: 0 0 0 0
[12007.624751] DMA: 500*4kB (UE) 452*8kB (UEC) 358*16kB (UEC) 12*32kB (UEC) 41*64kB (UEC) 9*128kB (UEC) 17*256kB (UC) 8*512kB (UC) 3*1024kB (C) 2*2048kB (C) 111*4096kB (C) = 485776kB
[12007.624878] HighMem: 14135*4kB (UM) 11331*8kB (UM) 5085*16kB (UM) 1180*32kB (UM) 376*64kB (UM) 85*128kB (UM) 55*256kB (UM) 13*512kB (UM) 9*1024kB (UM) 2*2048kB (UM) 457*4096kB (M) = 2207172kB
[12007.625005] 150713 total pagecache pages
[12007.625012] 0 pages in swap cache
[12007.625018] Swap cache stats: add 0, delete 0, find 0/0
[12007.625024] Free swap  = 0kB
[12007.625030] Total swap = 0kB
[12007.625036] 1012480 pages RAM
[12007.625042] 815872 pages HighMem/MovableOnly
[12007.625047] 33525 pages reserved
[12007.625053] 131072 pages cma reserved

Edit: Disregard this, this only delayed the problems for an extra hour or so before the oom-killer kicked in.

I have tried tons of things to fix this, and I believe I have finally cracked it. I am fairly certain that adding these two lines to my autostart.sh has stopped the condition that summons the oom-killer. Naturally, there is still some underlying problem that is merely sidestepped here.

sysctl vm.dirty_background_ratio=5
sysctl vm.dirty_ratio=8

Details:

I found that before the oom killer was invoked, the kernel process kswapd0 was using %100 cpu. I found that if the caches were flushed via echo 4 > /proc/sys/vm/drop_caches, it resolved the problem temporarily. This led me to mess with the cache ratios to force the system to keep the caches as empty as possible.

Quote from SupermanLeRetour

For what it's worth, I mitigated the issue by moving the external HDD to a USB 2.0 port and limiting my download speed to 10 MB/s. It seems the best way to prevent the issue it to not do a lot of I/O, so not ideal at all but at least I can use the system again. I'm still having trouble unrar-ing a large 30 GB files though.

I am not even sure it is limited to USB usage. I setup a second rpi4 as an nfs, and ran a docker torrent client on Libreelec accessing its share over samba, and I still got the error. This is backed up by this comment on github where a user is able to reliably trigger the error with a script that employs a using a virtual filesystem. Oh, I see that you are already participating in that issue thread.

My rpi4 running raspbian with the 64-bit kernel handles these docker containers without breaking a sweat. I would very much like to address this somehow, but at this point, I am not sure where to start. It seems that this is unlikely to be fixed in the 32-bit kernel upstream when arm_64bit=1 works in most use cases.