Slow file transfers to RPi5 over scp or rsync

    Hi!

    A few days ago I replaced my trusty RPi4b LibreELEC device, with a RPi5. Basically I pulled the cables from the back of the rpi4, and plugged them into the rpi5 (except the power lead, I use the new official rpi5 brick for the new device). All devices are connected with ethernet cables to my router, and my PC is set up to have passwordless ssh access.

    With the old I used to transfer files to it using scp and rsync. This went at around 80-90MB/s, which was fine. With the new rpi5, I found that transfers are capped at around ~23MB/s, much slower:

    Bash
    [~] $ # the old rpi4:
[~] $ scp -r -P 666 ~/largefile [email protected]:"/storage/"
largefile                                                         100%  781MB  91.2MB/s   00:08
[~] $ # the new rpi5:
[~] $ scp -r -P 666 ~/largefile [email protected]:"/storage/"
largefile                                                         100%  781MB  24.2MB/s   00:32

    After hours searching the internet, posting on other forums, and (I'm ashamed to say) even asking an AI chatbot, I found that if I manually limit the bandwidth, using the -l switch of the scp command, the speed goes up:

    After some testing the sweet spot seems to be around 100MB/s (819200 kbps). Higher than that, and the speed absolutely tanks (and apparently, setting no limit is equivalent to a much higher limit).

    So, that sort of solves the problem I guess, I can manually specify the bandwidth limit when running scp. But... unfortunately rsync does not have a working bandwidth limit. It supports a --bwlimit option, but that only works to lower the speed below the ~23MB/s, it will not help to increase it. If I use other (system level) methods to make sure rsync can not send more 100MB/s over the network, it does work, but these methods are cumbersome and not workable as standard method.

    Does anyone have any suggestions to fix this? I'm not exactly sure what the problem is, it seems to me that my PC is sending out data to the RPi5 much faster than it can handle. Do the pi's negotiate the speed somehow when a connection is established? Is there any setting in LibreELEC to make sure the RPi5 tells my PC not to push out the data too fast? With the -l flag on scp, it shows the RPi5 can handle receiving data at ~100MB/s, but if I go over just a little bit, it chokes on it spectacularly. Maybe there are settings that make it handle the data-overload better?

    I have added the old RPi4 back to the network (next to the new RPi5) and it still behaves completely normal, with fast transfers over both rsync and scp without any explicit bandwidth limiting.

    Thanks!

    I think I am on the verge of solving my own problem. It has to do with the TCP window size. I first noticed if I disable TCP window scaling altogether:

    Code
    # sysctl -w net.ipv4.tcp_window_scaling=0

    All file transfer max out my 1Gbit connection. Of course, disabling window scaling is not exactly beneficial to transmissions to/from remote servers. So I re-enabled window scaling and started messing with the maximum window size.

    • If I set it to 256KB, everything is fast.
    • setting to 512KB drops the speed to about 75MB/s
    • setting to 1MB drops it to around 40MB/s
    • setting it to 4MB (which was the default) drops it to ~22MB/s (which was my initial situation)

    Not entirely sure why this isn't a problem for the rpi4 though (it also uses window scaling, with the same maximum values), maybe its slower (sharing the usb bus, while the rpi5 is pci) and never actually gets to scale up?

    I do not know what is causing this to be an issue, the rpi5, or my router, or somthing else, but I'm fairly happy with this solution. Could someone, just to be sure, post the default values for their TCP window scaling options please? I've now fiddled so much with them I'm not sure anymore... That would be the output of:

    Code
    # sysctl net.ipv4.tcp_window_scaling net.core.wmem_max net.ipv4.tcp_wmem net.core.rmem_max net.ipv4.tcp_rmem

    Also, any other insights or ideas are welcome!

    Thanks!

    Code
    RPi5:~ # sysctl net.ipv4.tcp_window_scaling net.core.wmem_max net.ipv4.tcp_wmem net.core.rmem_max net.ipv4.tcp_rmem
net.ipv4.tcp_window_scaling = 1
net.core.wmem_max = 4194304
net.ipv4.tcp_wmem = 16384	16384	4194304
net.core.rmem_max = 4194304
net.ipv4.tcp_rmem = 16384	131072	33554432

    Default values from LE13 at current master branch HEAD state ^