- RPi4 4GB
- 12TB USB 3.0 Drive
- LE 10 beta
I am encountering constant oom events when there is plenty of available ram. The oom-killer can fire as often as every 30 seconds, and it loves to kill kodi. I know I can tell it to avoid killing kodi, but the underlying problem would persist. I've encountered this before on 9.2.6, but not nearly as often as on the new build.
I do a fresh install of LE10, attach my external USB3 drive, add my media libraries and add any docker image that generates lots of disk io (transmission, rsync, medusa, etc), and after a few minutes the oom-killer will begin decimating my processes every few minutes. My rpi4 memory usage never really goes over 1 GB or so (out of 4GB). I have seen the oom events occur even before docker and Kodi loads. I don't believe that memory fragmentation is a problem, as I always have plenty of 4096kB blocks of DMA and HiMem. I have been beating my head against this, and I believe it is possibly related to USB3 disk writes trigger OOM reaper on Pi 4. Any suggestions on how best to try to debug this?
I really love the design philosophy of LE and I would rather not shift to a normal distro, but other users report a 64 bit kernel alleviates may oom errors. Would it even be possible for me to compile a 64 bit version of LE for the rpi4 knowing that this would force software x264 decoding and loss of widevine?
Things I have tried:
- Enabling cgroups and limiting the docker images to only 256MB (limiting works, but oom persists)
- "sysctl vm.overcommit_memory=2" This causes the system to crash during boot
- cron job of "echo 4 > /proc/sys/vm/drop_caches" every minute. This reduces the frequency of oom events, but it doesn't stop them.
- mucking about with "vm.extfrag_threshold, vm.vfs_cache_pressure, swappiness": no real change
- Adding a massive swap file: It never gets used.
- Fresh LE10 install on both a large SD card and a usb 3.0 attached ssd.
- Even without docker, I believe I have seen one or two oom events, but they are rare.
[12007.624637] Mem-Info: [12007.624649] active_anon:2019 inactive_anon:108517 isolated_anon:0 [12007.624649] active_file:71240 inactive_file:80161 isolated_file:39 [12007.624649] unevictable:0 dirty:4 writeback:0 [12007.624649] slab_reclaimable:10261 slab_unreclaimable:9625 [12007.624649] mapped:48527 shmem:4136 pagetables:1389 bounce:0 [12007.624649] free:673253 free_pcp:155 free_cma:116364 [12007.624659] Node 0 active_anon:8076kB inactive_anon:434068kB active_file:284960kB inactive_file:320644kB unevictable:0kB isolated(anon):0kB isolated(file):156kB mapped:194108kB dirty:16kB writeback:0kB shmem:16544kB writeback_tmp:0kB kernel_stack:3288kB all_unreclaimable? yes [12007.624671] DMA free:485840kB min:16384kB low:20480kB high:24576kB reserved_highatomic:4096KB active_anon:0kB inactive_anon:0kB active_file:1352kB inactive_file:6260kB unevictable:0kB writepending:0kB present:786432kB managed:652332kB mlocked:0kB pagetables:640kB bounce:0kB free_pcp:256kB local_pcp:52kB free_cma:465456kB [12007.624677] lowmem_reserve: 0 0 3187 0 [12007.624714] HighMem free:2207172kB min:512kB low:21000kB high:41488kB reserved_highatomic:0KB active_anon:8076kB inactive_anon:434068kB active_file:283520kB inactive_file:314416kB unevictable:0kB writepending:16kB present:3263488kB managed:3263488kB mlocked:0kB pagetables:4916kB bounce:0kB free_pcp:364kB local_pcp:0kB free_cma:0kB [12007.624720] lowmem_reserve: 0 0 0 0 [12007.624751] DMA: 500*4kB (UE) 452*8kB (UEC) 358*16kB (UEC) 12*32kB (UEC) 41*64kB (UEC) 9*128kB (UEC) 17*256kB (UC) 8*512kB (UC) 3*1024kB (C) 2*2048kB (C) 111*4096kB (C) = 485776kB [12007.624878] HighMem: 14135*4kB (UM) 11331*8kB (UM) 5085*16kB (UM) 1180*32kB (UM) 376*64kB (UM) 85*128kB (UM) 55*256kB (UM) 13*512kB (UM) 9*1024kB (UM) 2*2048kB (UM) 457*4096kB (M) = 2207172kB [12007.625005] 150713 total pagecache pages [12007.625012] 0 pages in swap cache [12007.625018] Swap cache stats: add 0, delete 0, find 0/0 [12007.625024] Free swap = 0kB [12007.625030] Total swap = 0kB [12007.625036] 1012480 pages RAM [12007.625042] 815872 pages HighMem/MovableOnly [12007.625047] 33525 pages reserved [12007.625053] 131072 pages cma reserved