Slow USB3 ntfs/smb write Speeds

1st Official Post

    Hi

    now i know this is not the most ideal place to ask this question but i couldn't get an answer on the Raspberry forums so i am hoping you guys can help me out real quick.

    as the topic says i am having problems with write speeds on USB3/NTFS connected drives with Raspbian on a 4gb RPI4 sharing via SMB on a Gbit LAN.

    Running my normal environment LibreELEC smb write speeds from my windows PC are about 60mb/s (just tested it again)

    Now in order to be able to do play around with some stuff i installed Raspbian on another SDcard. But now write speeds dropped to about 20mb/s.

    to eliminate possible error sources i tested the following on Raspbian:

    - Network with iperf: ~900mbit/s
    - HDD read with hdparm: ~100mb/s
    - HDD write with sync; dd if=/dev/zero of=/media/[user/MyUSB]/tempfile bs=1M count=1024; sync: ~30mb/s

    so i suppose the culprit here is not necessarily the network(?) but somehow the way Raspbian accesses the USB3 HDD?

    I read in another thread that the way Raspbian's auto-mount works (which i am using) is not optimal for USB3 NTFS write speeds? Does LE mount the drives differently? Are they shared differently via smb?

    i have compared the outputs of blkid/demsg/mount of both LE and Raspbian but couldn't really make much out of it.

    So since i am still a Linux beginner and i do not want to just blindly copy paste mount commands or edit fstab, i would very much appreciate sb to point me in the right direction

    tl;dr: what magic does LE employ to achieve these super fast transfer rates? :)

    Cheers and thanks!

    thanks for the fast reply!


    sorry should have mentioned that -- this was the first thing i tried , alas, didn't help

    as mentioned above, this is what i read but i am hesitant

    Quote from AmigaGamer post_id=1494579 time=1562510211 user_id=99660

    Another thing that can really affect the performance is the way you mount the Samba filesystems.

    on my desktop PC running Buster i can only get about 40-50mb/s if the filesystem is mounted in at user level (gvfs-fuse)

    e.g. through File managers or comand line

    Code
    gio mount smb://user@server/share

    however if i mount via fstab or a traditional mount line i get a full gigabit transfer rates with a lower cpu overhead , e.g "

    Code
    sudo mount -t cifs -o ro username=USER,uid=$(id -u),gid=$(id -g) //server/share /home/pi/SAMBAshare"

    is the latter way how LE mounts it?


    :edit

    hmmm... i tried disabling auto-mount in file explorer and mount the drive manually in fstab

    1024+0 records in

    1024+0 records out

    1073741824 bytes (1.1 GB, 1.0 GiB) copied, 544.147 s, 2.0 MB/s

    ;(

    :edit²

    reconfigured fstab to use ntfs-3g

    now i am back at 30mb/s


    :edit³

    reconnected LibreElec to run the same test

    1024+0 records in

    1024+0 records out

    1073741824 bytes (1.0GB) copied, 13.911960 seconds, 73.6MB/s

    why are you guys ntfs wizards and Raspbian/(i.e. me) is not /shrug

    Edited 2 times, last by phob08 (April 15, 2020 at 2:42 PM).

    okay so i am getting desperate here trying to find the problem...^^

    i took a fresh U3 Samsung Evo Sd Card and installed Raspbian from scratch and ran the USB3 write test right out of the box: alas 30mb/s

    so i switched back to the LibreElec Sd Card (70mb/s USB3 write speed) and tried to compare both configurations to find out why LibreElec does such a better job at writing. There is only one thing i could find that was different(?): the ntfs-3g driver revision

    Raspbian reports:

    Quote

    ntfs-3g 2017.3.23AR.3 integrated FUSE 28 - Third Generation NTFS Driver


    LibreElec:

    Quote

    ntfs-3g 2017.3.23 external FUSE 29 - Third Generation NTFS Driver


    i am not sure what this means but could it possibly be the soúrce of my troubles? It is so weird since READ speeds are normal it is just the WRITE speeds that are too slow (30mb/s vs 70mb/s, see above)

    Again, any feedback or other ideas are highly appreciated, i am really running out of ideas on how pinpoint the problem

    Check if ntfs-3g's big_writes option is set - you should see it via "ps ax | grep ntfs".

    This option is known to speed up writes a lot and it's enabled by default in LibreELEC.

    Code
    xbmc:~ # ps ax | grep ntfs
684 root 0:00 mount.ntfs /dev/sdc1 /media/Sat-Hias -o big_writes,fmask=0133,uid=0,gid=0,utf8

    so long,

    Hias

    HiassofT to the rescue!

    as per usual, thanks for the reply!

    on RASPBIAN ps ax | grep ntfs returns:

    Code
    1084 ?        Ss     0:02 /sbin/mount.ntfs /dev/sda1 /media/[USERNAME/USB-DRIVE] -o rw,nodev,nosuid,uid=1000,gid=1000,uhelper=udisks2
2755 pts/0    S+     0:00 grep --color=auto ntfs

    so no it is not explicitly mentioned as enabled. I came across this idea as well but figured from what i understood from the ntfs3g documentation that the big writes option would be enabled by default anyway


    ill check out the LE sdcard later to compare if it is explicitly enabled there.

    Quote from HiassofT

    Check if ntfs-3g's big_writes option is set - you should see it via "ps ax | grep ntfs".

    This option is known to speed up writes a lot and it's enabled by default in LibreELEC.

    Code
    xbmc:~ # ps ax | grep ntfs
684 root 0:00 mount.ntfs /dev/sdc1 /media/Sat-Hias -o big_writes,fmask=0133,uid=0,gid=0,utf8

    so long,

    Hias

    my god could somebody please nominate this guy for president?! this was the third time that he provided the key clue for my stupid RPI issues

    added big writes to the mount table

    made sure it was reported in grep

    Quote

    dd if=/dev/zero of=/media/[NTFS-MOUNT W BIG WRITES]/tempfile bs=1M count=512; sync

    512+0 records in

    512+0 records out

    536870912 bytes (537 MB, 512 MiB) copied, 5.3942 s, 99.5 MB/s

    thanks so much

    Quote from phob08

    added big writes to the mount table

    made sure it was reported in grep

    I just signed up to this forum just to say THANK YOU.

    I was suffering from the same issue, very poor write speed on external NTFS hard drive which was connected to USB 3.0 port on my Raspberry Pi 4.

    Previous speed:

    Code
    ubuntu@raspberrypi:/mnt/extdrive$ dd if=/dev/zero of=test.bin bs=10M count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 26.4035 s, 39.7 MB/s

    After adding big_writes speed:

    Code
    ubuntu@raspberrypi:/mnt/extdrive$ dd if=/dev/zero of=test.bin bs=10M count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 6.62319 s, 158 MB/s

    As you can see, speed boosted from 40 MB/s to 158 MB/s (approx 4x faster :love:). They should make this feature ON by default. :rolleyes:

    Really, thank you both of you. :heart:

    I too, have the same issue with horrible USB copy speeds on LE but I didn't understood from this post (or the linked Rasoery sticky) how to fix it.

    I do not have an RPI nor USB attached drives formatted in msdos/ntfs.

    Here's my setup:

    1) LE is running on a "normal" Intel PC (i7 / 16gb) and LE is installed on a USB stick plugged into it.

    2) I have 2 pairs of 1tb Evo 860 SSD's in external enclosures (all formatted to gpt / ext4) connected to the LE PC via external USB 3.0 ports

    3) If I plug one of these enclosures into my laptop (which runs MX Linux) - I get copy speeds close to 400 mb/sec, but on LE it's down to 1 mb/sec

    a) Entering "ps ax | grep ntfs" (using SSH via Putty to LE) returns:

    11520 root 0:00 grep ntfs

    b) The etc/fstab on my LE is unwritable and empty.

    For comparison, the etc/fstab on my laptop reads:

    # Pluggable devices are handled by uDev, they are not in fstab 

    UUID=6fd8aee6-70cb-431e-ab8d-36b210214c23 / ext4 defaults 1 1 

    UUID=32A3-E19E /boot/efi vfat defaults,noatime,dmask=0002,fmask=0113 0 0 

    UUID=01cb3ced-d2c3-4843-9c6e-c0e0bb860e3d swap swap defaults 0 0

    c) The dmesg -c returns several pages of obscure text, and I don't know how to copy it from PuTTy into a text file on my laptop.

    My understanding of the solution recommended by other users above is as follows:

    1) Disable auto-mount of USB disks

    2) Edit etc/fstab and/ or add a "quirks" to the /boot/cmdline.txt (I don't have a "boot" folder on LE)

    Could someone with a "success story" please explain here, step by step, in plain English - which LE files need to be edited and how?

    Many thanks in advance!

    Thanks, but unfortunately I'm not sure I'm following the solution on this.. I'm just too much of a dabbler in Linux, would be my guess.

    I have an RPI 3 (not 4) running LibreElec 9.2.1 and Kodi build 18.6 dated 2020-03-03.

    I have an external 1TB drive plugged into it to use as a local media source. All works fine with that configuration.

    If I plug a USB flash drive into it and attempt to transfer a large file from the flash drive to the HDD, it will successfully transfer smaller files (say, up to 500-800 MB) but larger files start with fast transfer speeds (often 20 MB/s or so), but then after copying anywhere from 5 to 20% or so, things often just stop. I can leave it overnight and nothing changes.

    This is not consistent - sometimes it starts the file transfer, pauses, slows down, then speeds up again, and sometimes even completes the file transfer. But most often I have to try several times to get it to finish. When it fails, it has usually created a file on my HDD, often 50 MB or so.

    If I try any file over 1GB or so, forget it. It's almost impossible. And if I try to transfer multiple files - to, say, set up file transfers and let them run overnight, that almost never works.

    Although I'm not a Linux expert, I can Putty into the box and see the file structure.

    If I "ps ax | grep ntfs" on the machine I get

    /storage/.kodi/userdata$ ps ax | grep ntfs

    392 root 0:05 mount.ntfs /dev/sdb1 /media/Media -o big_writes,fmask=0133,uid=0,gid=0,utf8

    1164 root 0:00 grep ntfs

    Which seems to imply to me that "big_writes" is enabled, so the solutions here are already implemented.

    My fstab appears to be empty.

    Am I reading this incorrectly? Is there something here that I need to do to further troubleshoot this?