Why is copying to USB stick on Linux so damn slow?
Why is copying to USB stick on Linux so damn slow?
For context:
I'm copying the same files to the same USB drive for comparison from Windows and from my Fedora 41 Workstation.
Around 10k photos.
Windows PC: Dual Core AMD Athlon from 2009, 4GB RAM, old HDD, takes around 40min to copy the files to USB
Linux PC: 5800X3D, 64GB RAM, NVMe SSD, takes around 3h to copy the same files to the same USB stick
I've tried chagning from NTFS to exFAT but the same result. What can I do to improve this? It's really annoying.
You're viewing a single thread.
I find that it's around the same, except linux waits on updating the UI until all write buffers are flushed, whereas Windows does not.
except linux waits on updating the UI until all write buffers are flushed, whereas Windows does not.
I wish that were true here. But when I copy to USB the file manager ( XFCE/Thunar ) shows the copy is finished and closes the copy notifications way way before it's even half done, when I copy movies to a stick.
I use fast USB 3 stick on USB 3 port, and I don't get anywhere near the write speed the stick manufacturer claims. So I always open a terminal and run sync, to see when it's actually finished.I hate to the extreme when systems don't account for write cache before claiming a copy is finished, it's such an ancient problem we've had since the 90's, and I find it embarrassing that such problems still exist on modern systems.
I've ran sync and it exited already 4 times and the copy is still going
Try checking the progress here
https://unix.stackexchange.com/questions/48235/can-i-watch-the-progress-of-a-sync-operation#48245
Yes that's annoying too, I have no clue why it does that, but when the sync says "clear", I always wait a couple seconds, and run sync again a couple of times, to see if it's actually finished. And only THEN unmount the stick.
Copy to USB does not seem very solid on Linux IMO. So I also ALWAYS buy sticks with activity LED.
But even that can fool you, sometimes when I think a smaller copy is finished, because the LED stops blinking, it suddenly starts up again, after having paused for about 1½ second?!?!
That's nice but I managed to copy 300GB worth of data from the Windows PC to my Linux PC in around 3h to make a backup while I reinstall system and now I've been stuck for half a day copying the data back to the old Windows PC and I've not even finished 100GB yet... I've noticed this issue long ago but I ignored it as I never really had to copy this much data. Now it's just infuriating.
One thing I ran into, though it was a while ago, was that disk caching being on would trash performance for writes on removable media for me.
The issue ended up being that the kernel would keep flushing the cache to disk, and while it was doing that none of your transfers are happening. So, it'd end up doubling or more the copy time because the write cache wasn't actually helping removable drives.
It might be worth remounting without any caching, if it's on, and seeing if that fixes the mess.
But, as I said, this has been a few years, so that may no longer be actively the case.
This actually sounds like it could be the case, I'll explore tomorrow as I'm already in bed. Thanks for suggestion.
Edit: disabling caching yielded an improvement but very minor, writing to USB stick still sucks
Are you using two separate devices? If so another option could be LocalSend, it allows you to send files over the same network.
I used it for sending a couple hundred GBs of files. Didn’t take too too long. Also avoids unnecessary writes to flash media.
Thanks but that does not solve my problem. I'm not looking for alternative. If someone hands me USB drive to copy files for him I won't be able to use "LocalSend".