Posts by camelreef

BTW, this is fixed from my POV.

I like the ConnMan implementation.
The manual config straight into a file could probably do with a GUI for more general use, but I personally don't mind.
The ability to turn the VPN on and off from the LE GUI is great through, other family members can decide between geoblocking avoidance or local content without hand-holding just by flipping that graphical switch!

hmmmm... My forced EDID loading was suddenly messing things up and killing audio.

I've let LibreELEC deal naturally with the EDID data sent by both TV and AVR and audio has returned.

The double whammy is that the problem that hed led me to force saved EDID files has also disappeared. When? Who knows, I haven't tried things without forced EDID in months...

Github bug updated.

Quote from ghtester

The same issue here with LE 11 Nightly 20220607 (kernel 5.15.43). No audio on HDMI output (vc4-hdmi-0 MAI PCM i2s-hifi-0). Tried to change several Audio settings options without any positive effect.

Test with attached Lenovo USB Soundbar works fine (both available Analog and S/PDIF audio outputs).

As I am also experiencing an occassional screen flashing during browsing GUI menu when HDMI audio output is selected, the issue looks to be more deep and maybe the latest vc4-hdmi drivers are bad or there's a kernel issue.

Without any hint from developers where to look / what to try to get the HDMI audio working again, I had to revert back to latest working LE 11 Nightly 20220528-7faa80c (kernel 5.15.38).

More action about this here: https://github.com/LibreELEC/LibreELEC.tv/issues/6575

Quote from MatteN

One of my Pi 4 4GB runs LibreELEC-RPi4.arm-11.0-nightly-20220607-9d4c846 now, no problems with sound through HDMI to LG 65CX.

That's the funny thing. I have two identical RPi4 4GB, one has an issue, the other not.

Still no sound with LibreELEC-RPi4.arm-11.0-nightly-20220607-9d4c846

I opened this issue in Github: https://github.com/LibreELEC/LibreELEC.tv/issues/6575

I can add a +1 to that issue, on a 4GB RPi4, dual headed (hdmi0 to 4K TV, hdmi1 to HDMI 1.4 Receiver).

Reverting to 20220528-7faa80c gives me the audio back.

NO AUDIO - 20220601-0e57a30 debug log: http://ix.io/3ZbJ

GOOD AUDIO - 20220528-7faa80c debug log: http://ix.io/3ZbO

I have the same Pi model and revision (Raspberry Pi 4 Model B Rev 1.1) solely connected to a TV that doesn't exhibit the issue at all when using a post 20220528-7faa80c image.

GOOD AUDIO - 20220601-0e57a30 debug log: http://ix.io/3ZbP

GOOD AUDIO - 20220528-7faa80c debug log: http://ix.io/3ZbU

Both Pis were on the January Bootloader EEPROM when generating those debug logs. Updating to the April Bootloader EEPROM didn't change a thing.

I'm hoping that these will help see something. Ask for more data if needed!

Nico

Hello!

What is this hotdog build on the nightlies website?

https://test.libreelec.tv/hotdog/

Thanks!

Right... I've been playing with the add-ons with no real positive result.

Then, after one of the nightly updates, the problem has gone.

A sign that you guys are doing a great job! Thanks!

Quote from popcornmix

One doesn't seem to be getting symbols for functions, so hard to say what's happening.

The other shows openssl is repsonsible. That will be user code (not kernel) so is isn't the issue I was referring to.

Is that the on with the watchdog addon?

In either case I'd be tempted to (temporarily) disable that addon (and potentially others) to try to narrow it down.

I'll explore which as what addons. But no, the machine with OpenSSL doing strange things doesn't have the watchdog.

This is the list of addons. I'll work towards making the list as similar as possible on both ends and see what happens).

Ah... I don't know what to make of this! Two machines, two different results...

Not ignoring, but the 2 machines have decided to behave since! It will come, it always does...

Quote from popcornmix

Run "perf top" when it's laggy. Leave for a minute to settle.

If you see "vc4_atomic_commit_tail"/"clk_request_start" appearing in list (and increasing in percentage), then you have the same issue.

Noted, I will report back.

Thanks!

popcornmix OK... Can I do anything to help prove the theory or provide hard data beyond my vague description?

I'm filing this on the bugs board, but it's really a placeholder.

So, here is the problem I am experiencing: I have noticed that the Kodi interface slows down quite dramatically after a few hours of uptime on both of my RPi 4.

Data points:

• both machines are RPi 4 4 GB
• One Rpi is dual headed HDMI, HDMI0 to 4K/HDR TV for video, HDMI1 to older receiver for audio
• The other RPi is single HDMI to an older 720p TV
• Both wired to the network
• Both running the latest LE11 nightlies as they get posted
• Both machines point to a shared MySQL DB for Kodi
• Both run the tvheadend add-on
• One runs the library watchdog service
• A restart of the Kodi service or a reboot restores normal GUI responsiveness
• It's only the GUI that gets slow. Playback of any type of video is fine
• No apparent memory leak
• On one Pi (running the watchdog addon) I've noticed that the kodi process regularly gets stuck hogging a core (a restart of the kodi service fixes that). But the other Pi exhibits nothing like that. Both experience the same GUI slow downs, so I would say that the CPU usage is unrelated. Normally CPU usage is very low, and still I get the GUI slow down after some time.
• GUI slowdowns are probably not linked to usage. They happen when the systems have been left alone and idle, with the TVs off for a long time.

I can run an hours long debug log and post it, but I'm afraid that I would be providing a massive haystack. I'm not refusing to do it, if told to take that path anyway.

Is there anything more targeted that I could monitor and log that would be more useful/efficient for you guys?

Quote from CvH

btw dump write performance is just the half truth, read/write a lot files is also horrible slower compared to ext4

I'm ready to accept that.

The use case is a few large files being constantly written to with small chunks of data. That must not be any better!

In any case, if the dump benchmark is bad, the real world case of multiple read/writes will be worse.

Quote from heitbaum

Looking at the R&D we have now undertaken - I think the sentence could be written something like:

using samba to serve files from kernel ntfs3 …… whereas using samba to serve files from ext4 ….. In earlier kernel versions …. using ntfs-3g, samba was able to serve files from the ntfs external drive … (without issue) …..

I’m not sure that we have validated anything else yet? What does

Code

time dd if=file.mov of=/dev/null bs=4m 

(or similar) show from a performance perspective? From each of the file system types?

Your summary is a good one. However, I would say that the ability to serve files using Samba is only a consequence of the root cause, i.e. the difference of disk I/O performance provided by ext4, ntfs3 and ntfs-3g mounted filesystems, at least in my context.

The clear highlight is that I've first experienced the issue while staging a kit in January 2022, when I had a copycat production system having no issue, as it was in a software state predating December 15 2021, when the kernel update including the switch from ntfs-3g to ntfs3 arrived. Updating the software to a version including that switch also put the production system in a situation where the issue manifested itself.

This is on the LE machine, no Samba involved, cache dropped, repeatable numbers:

WD Element 2.5" external spindle, USB 3.0, 5 TB, NTFS formatted partition, ntfs3 kernel driver, RPi 4, LE 11 nightly-20220303-70c3c5d

kodiplayer:/var/media/content # sync && echo 3 > /proc/sys/vm/drop_caches && dd if=/dev/zero of=testfile bs=128k count=16k && sync && echo 3 > /proc/sys/vm/drop_caches && dd if=testfile of=/dev/null bs=128k && rm testfile
16384+0 records in
16384+0 records out
2147483648 bytes (2.0GB) copied, 186.714454 seconds, 11.0MB/s
16384+0 records in
16384+0 records out
2147483648 bytes (2.0GB) copied, 274.038229 seconds, 7.5MB/s

WD Element 2.5" external spindle, USB 3.0, 5 TB, ext4 formatted partition, RPi 4, LE 11 nightly-20220303-70c3c5d

kodiplayer:/var/media/content # sync && echo 3 > /proc/sys/vm/drop_caches && dd if=/dev/zero of=testfile bs=128k count=32k && sync && echo 3 > /proc/sys/vm/drop_caches && dd if=testfile of=/dev/null bs=128k && rm testfile
32768+0 records in
32768+0 records out
4294967296 bytes (4.0GB) copied, 40.273163 seconds, 101.7MB/s
32768+0 records in
32768+0 records out
4294967296 bytes (4.0GB) copied, 42.869481 seconds, 95.5MB/s

I should have done some tests over Samba before, using the NTFS filesystem, but it stupidly did not occur to me, sorry. However, it would most probably have embarrassed Samba, which would have defaulted as explained by the samba guys, and no meaningful data would have been gathered.

Quote from Samba bug

If the aio write size setting does not help, the next step will be "sync always = yes" and "strict sync = yes". We need to avoid that the kernel accumulates unwritten data in RAM, which can lead to these large timeouts that Samba has no control over. If you do the "sync always = yes", my hope is that the disk slowness is smoothed out over the time of all writes and is not batched when the kernel decides that it's enough to flush everything to the very slow disk.

I ended up ignoring the Samba options, preferring to fix the root cause, bad disk I/O, instead of adapting Samba to it.

Here is a benchmark over Samba, cache dropped, repeatable numbers. This is the current working solution, all is working well, disk I/O has enough performance, Samba is happy, the services higher up are happy:

RPi 3B+, Samba mount with default system options, up-to-date DietPi <-> WD Element 2.5" external spindle, USB 3.0, 5 TB, ext4 formatted partition, Samba share with default system options, RPi 4, LE 11 nightly-20220303-70c3c5d

root@grabber:/mnt/content# sync && echo 3 > /proc/sys/vm/drop_caches && dd if=/dev/zero of=testfile bs=128k count=16k && sync && echo 3 > /proc/sys/vm/drop_caches && dd if=testfile of=/dev/null bs=128k && rm testfile
16384+0 records in
16384+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 64.9038 s, 33.1 MB/s
16384+0 records in
16384+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 74.991 s, 28.6 MB/s

Does this help?

Quote from JoeAverage

isn't it already done by moving to kernel 5.15 ?

https://www.phoronix.com/scan.php?page=…-For-Linux-5.15

It has been done on Dec. 15. No need to go outside, I've already provided the link to the LE PR above.

Clearly ntfs3 is not that great for me. It's either because of my context, or because there is an issue in its current implementation in LE, as ntfs3, a kernel driver, is supposed to provide better performance than the older ntfs-3g, a user space driver.