MAJOR bug with large file USB copy speed on LE 9.2.2

1st Official Post

    This is a MAJOR bug in LE reported many times before, but it still persists.

    I have horribly slow file copy speeds on USB disks attached to LE and reading the recommended solution in thread here I can't understand how to fix it.

    Here's my setup:

    1) LE 9.2.2 is running on Intel PC (i7 / 16gb) Lenovo "Tiny"

    2) My 1tb Evo 860 SSD is housed in external USB 3.1 Gen2 enclosure and formatted to gpt / ext4 - connected to the LE PC via external USB 3.0 port

    3) If I copy large files on LE directly (dd) it starts at 120 mb/sec and then quickly drops to 1 mb/sec and then stops completely.

    4) If I copy the same file on LE from Thunar on my MX laptop across SMB wifi connection Ii get a slow, but steady speed of 12 mb/sec

    5) If I plug in the same drive directly into my laptop (which runs MX Linux) - I can copy a 10Gb file to/from my laptop at 400 mb/sec

    It's got even worse - when I play a high-bitrate mkv (e.g. DTS-HD MA) - Kodi plays for about 5 min, then stops playing with Error msg - "the source is too slow for continuous playback" ..

    The etc/fstab on my LE PC is unwritable and has no entries at all.

    Using PuTTY the dmesg -c on LE returns several pages of verbose text, and I don't know how to copy it from PuTTy into a text file to publish it here.

    For comparison, the etc/fstab on my MX laptop reads:

    # Pluggable devices are handled by uDev, they are not in fstab 

    UUID=6fd8aee6-70cb-431e-ab8d-36b210214c23 / ext4 defaults 1 1 

    UUID=32A3-E19E /boot/efi vfat defaults,noatime,dmask=0002,fmask=0113 0 0 

    UUID=01cb3ced-d2c3-4843-9c6e-c0e0bb860e3d swap swap defaults 0 0

    My understanding of the solution recommended in the thread here is as follows:

    1) Disable auto-mount of USB disks (really? - how?)

    2) Edit etc/fstab and/or add "quirks" line to the /boot/cmdline.txt (actually, I don't have a /boot/ folder on LE file system)

    Could someone please explain - step by step, in plain English - which LE system files need to be edited and how?

    Many thanks in advance!

    Edited 2 times, last by bpr323 (June 30, 2020 at 5:01 AM).

    Hello!

    The solution in the Link you mentioned is for NTFS, Not for EXT4 Disks.


    Can you Check If it also Happens If KODi is running or Not (systemctl Stop KODi)?


    What I found Out, it depends on the fact If it is your System Disk or Not (KODi can Block the whole system, makes No difference If it's USB or SATA).

    You can also force a Stall/halt of the entire PC Copy process or playback, If you Open a movie, EPG, Add-on, music or TV Show Info Page. The Copy process (also a Video or Music Playback running in the Background) slows down and finally stops.

    Feels Like there is No Multitasking anymore, one process in KODi can Block/slow down the entire system including Other Services Like tvheadend, smb. This Starts with KODi V18.

    I tought it's a KODi Bug because it only Happens as Long as KODi is running. If I Stop KODi everything is okay.

    Regards

    Nicolas

    Edited 2 times, last by Nicolas (June 29, 2020 at 10:01 AM).

    Thanks for replying Nicholas!

    No, I don't think it's Kodi to blame - it's Libreelec, for sure.

    1) I stopped Kodi # systemctl stop kodi

    2) Ran the first file (9Gb) copy from SSD1 to SSD2 -> very fast, 500 mb/s and completed Ok.

    3) Ran the second file -> it started at 120 mb/sec then stalled and stopped.

    when I play a high-bitrate mkv (e.g. DTS-HD MA) - Kodi plays for about 5 min, then stops playing with Error msg - "the source is too slow for continuous playback" ..

    It appears that some buffer on LE gets filled up and chokes the whole file system.

    Edited once, last by bpr323 (June 30, 2020 at 5:02 AM).

    • Official Post
    Quote from bpr323

    No, I don't think it's Kodi to blame - it's Libreelec, for sure.

    LibreELEC = Linux + Kodi , so I'm a bit confused there since you stopped the Kodi application. All that remains is a Linux OS.


    But, I'll try and have a go with 2 SSDs here and see what comes out of my copy experience with a RPi-4B.

    Quote from Klojum

    LibreELEC = Linux + Kodi , so I'm a bit confused there since you stopped the Kodi application. All that remains is a Linux OS.


    But, I'll try and have a go with 2 SSDs here and see what comes out of my copy experience with a RPi-4B.

    Hello Klojum

    LibreELEC = Linux + Kodi clear but, I thought it is a KODi Bug so I posted it there 2 Times but it seems noone has the Same Problem...

    I only saw that behavior on x86 Hardware since KODi V18.

    bpr323 you are Right! Same behavior Here.

    I personaly don't use the terminal Nor the integrated Filemanager from KODi to moved Big Files. So this never happened to me before.

    Just tried it on:

    Intel NUC ( j3455 SATA SSD to nvme USB3 SSD)

    PC Gigabyte mainboard i5 4460 (SATA SSD to SATA HD)

    Acer Nitro Laptop i7 6700hq (USB3 Stick to NVME SSD)

    ASRock j3455m (SATA HD to SATA HD)

    In all Tests KODi was running but idle.

    Makes No difference If it is USB, SATA or NVME.

    But If I Copy over the Network (also from and to the Same PC), which is the way I Always use, everything is okay as Long as I don't start an Action within KODi (Grab EPG CPU load 36%, Update Database CPU load 55%, search for new Media etc) or open an Info Screen.

    Regards

    Nicolas

    Edited once, last by Nicolas (June 29, 2020 at 2:40 PM).

    Quote from Klojum

    LibreELEC = Linux + Kodi , so I'm a bit confused there since you stopped the Kodi application. All that remains is a Linux OS.


    But, I'll try and have a go with 2 SSDs here and see what comes out of my copy experience with a RPi-4B.

    Hi Klojum, thank you so much for responding!

    Libreelec = "just enough OS for Kodi" ... I don't know about other os platforms, but Linux in LE is definitely broken.

    I have no issues with Kodi, this is the best (and only) renderer for DLNA playback. I'm not using its "media server" at all, the disks are not used by Kodi library.

    I've outlined 3 use-cases for USB attached storage in my post, but to summarise:

    1) USB disks are attached to LE box, file manager is Dolphin (or Krusader) running in docker container on LE as native app, which I access using its webUI (browser) from my MX Linux laptop. File copy runs super fast the 1st time, the next file stalls and the whole system grinds to a halt - including MiniDLNA scraping.

    2) As above, but file copy is done from the MX Linux laptop using Thunar across SMB or SFTP. File copy runs steady at a slow, but reliable speed of 12-17 mb/sec

    3) The USB disks are attached directly to MX Linux laptop - crazy speeds up to 420 mb/sec

    4) Previously I've tried Midnight Commander running as native app on LE, but driven remotely from MX Linux laptop - same "crashing" issues.

    There's definitely something broken with the way linux on Libreelex mounts the disks and/or treats large files.

    Edited once, last by bpr323 (June 30, 2020 at 2:54 AM).

    Here's my LibreELEC DLNA Home Theatre conceptual diagram cook book.

    1) The USB disks are attached to LE Linux box, and LE is installed on a USB stick. The host is Lenovo M720g "Tiny" FF

    2) Transmission and Krusader run on the LE box, managed via WebUI remotely on my MX Linux laptop.

    All heavy traffic is offloaded from my laptop to LE host box.

    The reason I prefer MiniDLNA and don't use Kodi Library is that Kodi doesn't let me browse my audio as "folders".

    MiniDLNA is super-fast and piss easy to configure scraping your USB disks (when you know how).

    Furthermore, I control media browsing and playback from my Android phone - BubbleUPNP app has excellent UI including thumbnails and artwork. Basically, BubbleUPNP "tells" Kodi renderer which files to play from the MiniDLNA media server.

    Imagine, you can instantly play your music without turning on your TV or Laptop screen! No more fluffy dice :)

    Edited 7 times, last by bpr323 (June 30, 2020 at 3:54 AM).

    My SSD USB drives sit in USB 3.1 Gen2 enclosure, so theoretically the speed should be 10,000M.

    The USB ports on Lenovo M720q are also USB 3.1 Gen2 compatible.

    However, when I run <# lsub -t> the output shows my max speed is only 5,000M

    Edited 3 times, last by bpr323 (July 1, 2020 at 1:05 AM).

    Hi Flex, thanks for replying!

    Here's the Kodi debug log attached.

    I doubt its a Kodi issue, rather a Linux programming error which you cannot see from the Kodi log.

    The most common are "large audio sync error", where Kodi doesn't get enough data bandwidth from the media source.

    Possibly, the clue for the underlying root cause is provided in the log here:

    [301] 2020-07-01 19:29:26.313 T:139930189871232 DEBUG: DBus method call to org.freedesktop.DBus.Peer.Ping at /org/freedesktop/UDisks2 of org.freedesktop.UDisks2 failed: org.freedesktop.DBus.Error.ServiceUnknown - The name org.freedesktop.UDisks2 was not provided by any .service files

    [302] 2020-07-01 19:29:26.314 T:139930189871232 DEBUG: DBus method call to org.freedesktop.UDisks.EnumerateDevices at /org/freedesktop/UDisks of org.freedesktop.UDisks failed: org.freedesktop.DBus.Error.ServiceUnknown - The name org.freedesktop.UDisks was not provided by any .service files

    2020-07-01 19:29:26.314 T:139930189871232 DEBUG: Selected UDev as storage provider

    UPDATE - I have 4 SSD's (1Tb 860 Evo) connected to external USB 3.1 Gen2 (USB-C Thunderbolt) ports on Lenovo, and 1 SSD (SSD-5) connected internally to SATA-III interface on the motherboard.

    If I copy/cut/paste large 10Gb file across different folders on the same internal disk (SSD-5) it goes flawlessly. But if I move same 10Gb file to one of the attached USB disks, or from one USB disks to another - that's when the first time file copy works fine, but on a second attempt the file copy slows down and grinds to a halt.

    Therefore, the problem appears to be caused by the way USB attached disks are mounted by UDEV on LE (etc/fstab has no entries at all).

    Maybe there's a missing driver in LE for USB 3.1 Gen2 PCIE interface? Gen2 increases the speed from 6Gb/s to 10Gb/s. There is no problem with the controller in the USB 3.1 enclosure. If I connect this enclosure with two SSD's to a USB 3.0 or USB 3.1 (-C) port on my MX Linux laptop, everything is super fast, no slowing or choking on large >10Gb files.

    Quote from Da Flex

    I haven't found related errors at your log. We already had users, who suffered low speed on external devices. Often a better cable was the answer. If you already tested the cable, then advancedsettings.xml is worth to play with. Especially have a look at chapter 2.8.4 Cache.

    I fail to see how your response (looking at Kodi log) is related to slow copy speeds when Kodi is not even turned on??

    "a better cable was the answer" ?? Have you even read my post above regarding attaching the same SSD+enclosure+cable to a different device (MX Linux laptop) and getting speeds >300 mb/s ?? You've got to be joking, right?

    Because you run on PC hardware, I think you have more chance to solve the problem by installing any Linux OS you like ( MX, Mint, Ubuntu... ) configure as you want and after that install Kodi on top...

    p.s.

    On rpi LE, I am successfully customizing my Kodi installation by unsquashfs, editing and squashfs again the SYSTEM.

    Спасибо, Занятой!

    Where on my LE can I "find the name of the embedded udev rule that mounts and sets options" - any clues?

    My current LE /storage/.config/udev.rules.d/ folder is empty - what should I call the "new rule.*" so that the modified rule is recognised?

    Many thanks in advance, mate!

    My LE etc/fstab file has two lines:

    /dev/cdrom /media/cdrom iso9660 noauto,ro 0 0

    /dev/usbdisk /media/usb vfat noauto,ro 0 0

    # blkid returns as per screenshot below

    <removed by admin>

    If etc/fstab holds the answer - how should I fix the current two lines in it?