Kodi 21 alpha 3 memory leak?

Noticing with the latest LE12 nightly that there appears to be a memory leak, seeing oom killer on RPi4 quite a bit more frequently.

http://ix.io/4Gxm - journalctl oom-killer reports

http://ix.io/4Gxn - pmap of kodi.bin (3.8G virt, 1.1G rss at the moment -- will probably be killed soon)

Seems like a similar issue on Allwinner H6 boards. Anyone else seeing this problem?

RPi4 - Built on commit 09641de8d37c324555542c1ba62992ce0d89d5e3 (looks like RPi4 is using Kodi commit aaef1a0d1e37ff87361011fd48c89d9e69ec9a9d, just before alpha 3)

OPi3lts - Built on commit 699f92cf62ce318a80d28d6e06ab55661e6c6c03 (this is using 21 alpha 3)

Alpha 2 was OK, didn't notice any problems with that.

Quote from HiassofT

Might be worth testing with a clean kodi installation without addons.

Pretty basic addons here, on the RPi4 it's just the Libretro stuff, pvr.hts, youtube and plutotv. On the OPi3 it's even less just pvr.hts and youtube, plus the usual dependencies.

I suppose it could be pvr.hts or Youtube/InputStream Adaptive. Nothing sketchy, can't be bothered with that crap it breaks all the time and ISPs are going to be banning those illegal streams in the US anyways.

I have zachmorris repo on the RPi4 for IAGL, but really haven't used any of the RetroPlayer stuff, the nostalgia wore off pretty quickly :).

I think it is related to pvr.hts. Rolled back to 21.0.1-Omega and it seems better, but it will take a few days of regular usage to be sure. Which is odd, because I see no functional change. I also rolled back InputStream Adaptive to 21.1.3.1.

At idle it is a ~500 byte memory leak every minute (it's pretty regular), 30MB leaked per hour. It is either add-on, "feature" related, or based on whether you actually start doing something. The aw device is just sitting idle at the home screen, nothing done there, and it's not showing any leakage. The RPi4 device has been used to playback PVR, videos, youtube, and it is routinely leaking ~500 bytes every minute.

memory monitoring

Here is the script mm.sh:

http://ix.io/4GIb

You set permissions to 0700 and run it with ./mm.sh |tee -a mm.csv

I disabled YouTube add-on, and interestingly that seems to be the cause. So it is either related to recent Python interpreter bumps or something has changed to cause this resource leak.

So there is a 60-second loop in YouTube add-on for checking whether httpd is running, which it is enabled if you use MPEG-DASH or API configuration page. I typically use MPEG-DASH, turned that off in settings and the leak is gone. Turned it back on (w/ enable/disable toggle of Youtube addon) and the leak returns.

I suspect it is perhaps related to the act of "pinging the httpd server" that is causing the problem, the loop does two things:

1. uses requests to do an http get to the internal http service running for youtube addon

2. restarts httpd service when request fails (not 204)

I really doubt it's #2 because I don't think Youtube would work at all, IIRC ISA uses httpd to communicate back to the requesting player for some stuff (I remember that from the Netflix addon), it is probably requests library is doing some sort of resource leakage?

Memory leak associated to Youtube when MPEG-DASH is enabled · Issue #520 · anxdpanic/plugin.video.youtube (github.com)

I couldn't reproduce it with a simple requests.get loop in Python, so it's perhaps related to the embedded interpreter or Kodi built-ins bound to the embedded interpreter.

This was the kludge test script I tried via command line:

#!/usr/bin/python3
import sys
sys.path.append("/storage/.kodi/addons/script.module.requests/lib/")
sys.path.append("/storage/.kodi/addons/script.module.urllib3/lib/")
sys.path.append("/storage/.kodi/addons/script.module.chardet/lib/")
sys.path.append("/storage/.kodi/addons/script.module.idna/lib/")
sys.path.append("/storage/.kodi/addons/script.module.certifi/lib/")
import requests
import time

while True:
   response = requests.get("http://google.com/generate_204")
   print(response)
   time.sleep(10)

Kodi issue:

xbmcaddon.Addon python binding memory leak after 21.0alpha2 · Issue #23786 · xbmc/xbmc (github.com)

Reverting this PR seems to solve it as workaround:

[AddonSettings] Fix setting persistency when acessing raw Settings interface by enen92 · Pull Request #23558 · xbmc/xbmc (github.com)

Quote from MaxMustermann

Thank you so much for your enormous efforts to track this bug down. It's so annoying to have to restart Kodi every few hours because it runs OOM. Can't wait to have a normally working instance again.

I expect it will be worse with the more Python add-ons you use, until it is fixed by the Kodi team. I was only using a few Python addons (with YouTube really being the primary one), everything else is binary addons (pvr.hts, for example) which seem unaffected by this change.