Massive performance deterioration after time

I'm running LibreElec 10.0 beta (9.95.4) on a Raspberry Pi4b (4GB). My goal is to watch 4k dumps with audio passthrough (7.1) for all available formats to an AVR. Movies are streamed from an external HDD (NTFS) which is connected to a USB 3.0 port.

To get some stable playback performance, I overclocked the CPU to 2300 MHz and the GPU to 750 MHz (the following problem is not tied to the overclock though, as it also occurs with stock clock values). This works greatly when booting freshly into the system. Video playback is smooth. However, after a few minutes the picture starts to lag behind the sound. First this only looks like minor lags: video micro-pausing, followed by sped up video playback where the player tries to run through the frames to catch up with the audio. This gets so bad over time that videos become unwatchable and a buffering ring is displayed every few seconds. Ultimately, total "CPU-KODI" usage just creeps higher and higher until it persistently reaches 100+%.

I SSHed into the system and monitored system usage with top. I realized that mount.ntfs and kswapd0 would hit 100% CPU usage every time the lag occurred (see attached screenshot). The timespan between those spikes becomes shorter and shorter over time.

Going by top, no swapping seems to be actually taking place (MiB Swap = 0.0). Also, reported memory usage by Kodi never exceeds 750MB. Is there anything I can do on my end to prevent those spikes in CPU usage as a workaround? My first idea was to reformat the external HDD to a Linux-compatible format, but that would break compatibility with my TV and other devices. The other obvious solution seems to be to deactivate the swapping service. Although I think that I have enough RAM available to not require swapping, this seems to be not advisable by the sources I found. But may be those spikes are just the symptom of an underlying problem.

I actually don't need joystick support (yet) and deactivated it. Doesn't change anything about the problem. Do you need me to supply a new log after a restart?

No problem

I'm not sure if this one will be more useful though. I activated debug logging, rebooted and logged the first five minutes of playing a movie. The log ended up being 30 MB in size. It was flooded with ffmpeg details. I had to manually edit it down because I couldn't compress it below a 1 MB zip file. If you have a suggestion to produce a more useful log then please let me know.

Alright, I deactivated BT and Airplay, rebooted and played some 3 minutes of a 4k movie until the lagging became very obvious to create the log. Still pretty huge though.

I also tested 1080p footage and the same issue occurs, albeit less intense. Usually, when playing 1080p, CPU usage hovers between 30 to 40%, but I was able to screenshot a spike where all services hit 100% or higher. The playback recovers much faster and no buffering circle is triggered, but the playback is jerky every few seconds to a point where it's no fun watching at all.

Update: For testing purposes I also gave LibreElec 9.2.7 a shot (no overclock this time). None of the issues occured. 1080p and 4k playback was smooth. CPU usage was around 25% for 1080p and 50% to 60% for 4k. While the playback is dead stable, unfortunately all colors for HDR/BT2020 content are way off making LE 10 the only viable option.

Quote from elonesna

I see very strange the little amount of free memory (393.4 / 3825.0 MB on your device - 2867.3 / 3832.2 MB on my device). This may be a good reason for your device to be slow. If I know more I would try to find some addon or kodi settings responsible for this.

Actually, top reports 2867MB free and also Kodi itself never reports more than 750MB to be in use.

Anyway, I tested some more: I put a 4k movie directly onto the SD-card LibreElec is running from and removed my external NTFS-formatted drive where the movie originally was played from. All issues are gone. Neither kswapd0 nor mount.ntfs pop up in top any more. CPU usage is between 50 to 60% without any overclocks. So I think this might be an issue pertaining to NTFS compatibility.

Quote from Da Flex

There are some network and cache settings you can play with at advancedsettings.xml.

Thanks! I'll give that a shot.

I tested some more with playing 4k content off of different media and got some strange results:

- 128 GB Sandisk Extreme USB stick (NTFS) - no issues
- 32 GB Sandisk SDXC flash memory card (ExFAT) - no issues

elonesna: You're correct about the cache/buffer filling up. I monitored this for a while. Usually, cache/buff fills up until 150-200MB of free space remains and hovers around this amount. With the aforementioned media, this doesn't have any bearing on playback. Everything remains buttery smooth with the Sandisk stick and card despite the full cache.

The fact that flash memory doesn't seem to be impacted while a physical HDD is, puzzles me as well. The HDD is brand new. Write speeds range from 100-190MB/s, depending on whether the data is located on the inner or outer sectors of the platter. There's little to no fragmentation. I'm wondering whether my external HDD is triggering some extra functions that impact playback performance. It has a USB-hub built in, though no devices have been connected to it. Let's see if the buffer options in advancedsettings.xml will have an impact.

Quote from Da Flex

There are reports that other connected USB devices can slow down the data rate. So exclusively connect your HDD to a blue USB port.

I did. In fact, I tried all ports, just to make sure there wasn't a fluke with a single one of them. Always the same result. I will now look into disabling kswapd0. It's not the services fault or mount.ntfs, but clearly LibreElecs, as can be seen from the top screenshots as also LE's CPU usage goes through the roof when those services do. There must have been changes between 9.2.7 and 9.95.4 regarding NTFS handling, as the former works with the HDD without a hitch.

Quote from Da Flex

Another approach related to NTFS: If LE mounts the HDD as read-only, chances are that some meta data can't be stored on HDD. Please check whether it's mounted as read-only or read-write device.

Thanks for the hint! I wasn't quite sure how to do that, so I used the built-in file manager to copy a file to the HDD and that worked. May be forcing a read-only mount could help?

In the meantime, I tested another external NTFS HDD (older model, USB 2.0 only). It ran absolutely smoothly. I also tested attaching a USB-hub to the Raspi and all of the functioning drives one by one to see if the HDD's built-in hub was somehow at fault, but no problems occurred. Furthermore, I connected those drives to the problematic drive's USB hub, but no drive was affected by the described issue.

Quote from Da Flex

Thanks for the info! I would re-format the problematic NTFS device then. Maybe use GParted for this task, because it can detect HDD errors safely.

I'll give that a shot, but it's an 8 TB drive and moving everything off will take all day. I'll report back once done.

Quote from HiassofT

Before you go ahead and re-format the drive I'd recommend posting the journal after the issue occurred. If there are kernel or ntfs issues they should be logged there.

Run the following command and post the URL:

Code

journalctl -a | pastebinit

so long,

Hias

Thanks. Here's the log:

http://ix.io/3rob