Slow USB3 ntfs/smb write Speeds

phob08 · April 15, 2020 at 12:30 PM

Hi

now i know this is not the most ideal place to ask this question but i couldn't get an answer on the Raspberry forums so i am hoping you guys can help me out real quick.

as the topic says i am having problems with write speeds on USB3/NTFS connected drives with Raspbian on a 4gb RPI4 sharing via SMB on a Gbit LAN.

Running my normal environment LibreELEC smb write speeds from my windows PC are about 60mb/s (just tested it again)

Now in order to be able to do play around with some stuff i installed Raspbian on another SDcard. But now write speeds dropped to about 20mb/s.

to eliminate possible error sources i tested the following on Raspbian:

- Network with iperf: ~900mbit/s
- HDD read with hdparm: ~100mb/s
- HDD write with sync; dd if=/dev/zero of=/media/[user/MyUSB]/tempfile bs=1M count=1024; sync: ~30mb/s

so i suppose the culprit here is not necessarily the network(?) but somehow the way Raspbian accesses the USB3 HDD?

I read in another thread that the way Raspbian's auto-mount works (which i am using) is not optimal for USB3 NTFS write speeds? Does LE mount the drives differently? Are they shared differently via smb?

i have compared the outputs of blkid/demsg/mount of both LE and Raspbian but couldn't really make much out of it.

So since i am still a Linux beginner and i do not want to just blindly copy paste mount commands or edit fstab, i would very much appreciate sb to point me in the right direction

tl;dr: what magic does LE employ to achieve these super fast transfer rates?

Cheers and thanks!

**Klojum** · April 15, 2020 at 12:38 PM Official Post

Perhaps this is a solution for your USB connections: STICKY: If you have a Raspberry Pi 4 and are getting bad speeds transferring data to/from USB3.0 SSDs, read this - Raspberry Pi Forums

phob08 · April 15, 2020 at 12:42 PM

thanks for the fast reply!

sorry should have mentioned that -- this was the first thing i tried , alas, didn't help

as mentioned above, this is what i read but i am hesitant

Quote from AmigaGamer post_id=1494579 time=1562510211 user_id=99660
Another thing that can really affect the performance is the way you mount the Samba filesystems.

on my desktop PC running Buster i can only get about 40-50mb/s if the filesystem is mounted in at user level (gvfs-fuse)

e.g. through File managers or comand line
Code
gio mount smb://user@server/share
however if i mount via fstab or a traditional mount line i get a full gigabit transfer rates with a lower cpu overhead , e.g "
Code
sudo mount -t cifs -o ro username=USER,uid=$(id -u),gid=$(id -g) //server/share /home/pi/SAMBAshare"

is the latter way how LE mounts it?

:edit

hmmm... i tried disabling auto-mount in file explorer and mount the drive manually in fstab

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB, 1.0 GiB) copied, 544.147 s, 2.0 MB/s

:edit²

reconfigured fstab to use ntfs-3g

now i am back at 30mb/s

:edit³

reconnected LibreElec to run the same test

1024+0 records in

1024+0 records out

1073741824 bytes (1.0GB) copied, 13.911960 seconds, 73.6MB/s

why are you guys ntfs wizards and Raspbian/(i.e. me) is not

phob08 · April 16, 2020 at 11:25 AM

okay so i am getting desperate here trying to find the problem...^^

i took a fresh U3 Samsung Evo Sd Card and installed Raspbian from scratch and ran the USB3 write test right out of the box: alas 30mb/s

so i switched back to the LibreElec Sd Card (70mb/s USB3 write speed) and tried to compare both configurations to find out why LibreElec does such a better job at writing. There is only one thing i could find that was different(?): the ntfs-3g driver revision

Raspbian reports:

Quote

ntfs-3g 2017.3.23AR.3 integrated FUSE 28 - Third Generation NTFS Driver

LibreElec:

Quote

ntfs-3g 2017.3.23 external FUSE 29 - Third Generation NTFS Driver

i am not sure what this means but could it possibly be the soúrce of my troubles? It is so weird since READ speeds are normal it is just the WRITE speeds that are too slow (30mb/s vs 70mb/s, see above)

Again, any feedback or other ideas are highly appreciated, i am really running out of ideas on how pinpoint the problem

**HiassofT** · April 16, 2020 at 12:29 PM

Check if ntfs-3g's big_writes option is set - you should see it via "ps ax | grep ntfs".

This option is known to speed up writes a lot and it's enabled by default in LibreELEC.

Code

xbmc:~ # ps ax | grep ntfs
684 root 0:00 mount.ntfs /dev/sdc1 /media/Sat-Hias -o big_writes,fmask=0133,uid=0,gid=0,utf8

so long,

Hias

phob08 · April 16, 2020 at 12:51 PM

HiassofT to the rescue!

as per usual, thanks for the reply!

on RASPBIAN ps ax | grep ntfs returns:

Code

1084 ?        Ss     0:02 /sbin/mount.ntfs /dev/sda1 /media/[USERNAME/USB-DRIVE] -o rw,nodev,nosuid,uid=1000,gid=1000,uhelper=udisks2
2755 pts/0    S+     0:00 grep --color=auto ntfs

so no it is not explicitly mentioned as enabled. I came across this idea as well but figured from what i understood from the ntfs3g documentation that the big writes option would be enabled by default anyway

ill check out the LE sdcard later to compare if it is explicitly enabled there.

**Klojum** · April 16, 2020 at 3:36 PM Official Post

Not to be very picky, but there is a difference between MB/s , Mb/s and mb/s.

I hope you mean the correct one.

phob08 · April 16, 2020 at 4:30 PM

Not quite sure what you mean but i was refering to Megabytes per second (not megabits)

phob08 · April 17, 2020 at 3:44 PM

Quote from HiassofT
Check if ntfs-3g's big_writes option is set - you should see it via "ps ax | grep ntfs".

This option is known to speed up writes a lot and it's enabled by default in LibreELEC.
Code
xbmc:~ # ps ax | grep ntfs
684 root 0:00 mount.ntfs /dev/sdc1 /media/Sat-Hias -o big_writes,fmask=0133,uid=0,gid=0,utf8
so long,

Hias

my god could somebody please nominate this guy for president?! this was the third time that he provided the key clue for my stupid RPI issues

added big writes to the mount table

made sure it was reported in grep

Quote

dd if=/dev/zero of=/media/[NTFS-MOUNT W BIG WRITES]/tempfile bs=1M count=512; sync

512+0 records in

512+0 records out

536870912 bytes (537 MB, 512 MiB) copied, 5.3942 s, 99.5 MB/s

thanks so much

torme1983 · June 17, 2020 at 12:39 AM

Quote from phob08

added big writes to the mount table

made sure it was reported in grep

I just signed up to this forum just to say THANK YOU.

I was suffering from the same issue, very poor write speed on external NTFS hard drive which was connected to USB 3.0 port on my Raspberry Pi 4.

Previous speed:

Code

ubuntu@raspberrypi:/mnt/extdrive$ dd if=/dev/zero of=test.bin bs=10M count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 26.4035 s, 39.7 MB/s

After adding big_writes speed:

Code

ubuntu@raspberrypi:/mnt/extdrive$ dd if=/dev/zero of=test.bin bs=10M count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 6.62319 s, 158 MB/s

As you can see, speed boosted from 40 MB/s to 158 MB/s (approx 4x faster ). They should make this feature ON by default.

Really, thank you both of you.

bpr323 · June 27, 2020 at 1:34 PM

I too, have the same issue with horrible USB copy speeds on LE but I didn't understood from this post (or the linked Rasoery sticky) how to fix it.

I do not have an RPI nor USB attached drives formatted in msdos/ntfs.

Here's my setup:

1) LE is running on a "normal" Intel PC (i7 / 16gb) and LE is installed on a USB stick plugged into it.

2) I have 2 pairs of 1tb Evo 860 SSD's in external enclosures (all formatted to gpt / ext4) connected to the LE PC via external USB 3.0 ports

3) If I plug one of these enclosures into my laptop (which runs MX Linux) - I get copy speeds close to 400 mb/sec, but on LE it's down to 1 mb/sec

a) Entering "ps ax | grep ntfs" (using SSH via Putty to LE) returns:

11520 root 0:00 grep ntfs

b) The etc/fstab on my LE is unwritable and empty.

For comparison, the etc/fstab on my laptop reads:

# Pluggable devices are handled by uDev, they are not in fstab

UUID=6fd8aee6-70cb-431e-ab8d-36b210214c23 / ext4 defaults 1 1

UUID=32A3-E19E /boot/efi vfat defaults,noatime,dmask=0002,fmask=0113 0 0

UUID=01cb3ced-d2c3-4843-9c6e-c0e0bb860e3d swap swap defaults 0 0

c) The dmesg -c returns several pages of obscure text, and I don't know how to copy it from PuTTy into a text file on my laptop.

My understanding of the solution recommended by other users above is as follows:

1) Disable auto-mount of USB disks

2) Edit etc/fstab and/ or add a "quirks" to the /boot/cmdline.txt (I don't have a "boot" folder on LE)

Could someone with a "success story" please explain here, step by step, in plain English - which LE files need to be edited and how?

Many thanks in advance!

tmaddison · August 24, 2020 at 4:52 PM

Thanks, but unfortunately I'm not sure I'm following the solution on this.. I'm just too much of a dabbler in Linux, would be my guess.

I have an RPI 3 (not 4) running LibreElec 9.2.1 and Kodi build 18.6 dated 2020-03-03.

I have an external 1TB drive plugged into it to use as a local media source. All works fine with that configuration.

If I plug a USB flash drive into it and attempt to transfer a large file from the flash drive to the HDD, it will successfully transfer smaller files (say, up to 500-800 MB) but larger files start with fast transfer speeds (often 20 MB/s or so), but then after copying anywhere from 5 to 20% or so, things often just stop. I can leave it overnight and nothing changes.

This is not consistent - sometimes it starts the file transfer, pauses, slows down, then speeds up again, and sometimes even completes the file transfer. But most often I have to try several times to get it to finish. When it fails, it has usually created a file on my HDD, often 50 MB or so.

If I try any file over 1GB or so, forget it. It's almost impossible. And if I try to transfer multiple files - to, say, set up file transfers and let them run overnight, that almost never works.

Although I'm not a Linux expert, I can Putty into the box and see the file structure.

If I "ps ax | grep ntfs" on the machine I get

/storage/.kodi/userdata$ ps ax | grep ntfs

392 root 0:05 mount.ntfs /dev/sdb1 /media/Media -o big_writes,fmask=0133,uid=0,gid=0,utf8

1164 root 0:00 grep ntfs

Which seems to imply to me that "big_writes" is enabled, so the solutions here are already implemented.

My fstab appears to be empty.

Am I reading this incorrectly? Is there something here that I need to do to further troubleshoot this?

Slow USB3 ntfs/smb write Speeds

Similar Threads

S805x build (lafrite)

Pi4 with LE 9.2.1 - slow boot and USB3 SSD problem