Hi everyone, I was wondering if there are any computer gurus that can tell me why when I am copying from one portable hard drive to another the speed drops from around 11.6 MB/Seconds to sometimes under 1 MB/Seconds? I am backing up my Poser library which is around 220 gigs and seems to take forever to copy the library! Cheers

I'm about as far from a guru as one can be, but I notice the same thing whenever I'm copying a large number of very tiny files. When I'm copying a few really big files, the speed is much greater. Why this is, I cannot tell you, but it seems intuitively correct that it should be the case. (Even if you have a truck to help you, a ton of loose marbles takes longer to move than one big chunk of granite.)

Poser runtimes have thousands of files, maybe even millions for some of us. It takes a long time to copy all those.

If someone has a better answer than my own "it seems intuitively correct", I'd be interested to hear it. :)

There are a million reasons why copying/moving files will be slow.  Much depends on the drive and connection.  I do a lot of this at work with very large image and video databases.  I've found that this little tool makes the process goof-proof and much more stable

http://codesector.com/teracopy.

The free version works extremelly well.  Windows native file manager will lock and hang without giving any indication that something's wrong.  With Teracopy I can set a huge file backup to run overnight and know that it'll work.  

I'm running SyncBack, the Free version
http://www.2brightsparks.com/download-syncbackfree.html

But most of those tools are pretty good, just take your pick.

The most likely answer to your question: you are using the Windows file copy method, which copies a file and then checks its proper arrival, then it takes the next file. As a result, small files will suffer an enormous overhead without transmission activities, and your transfer speed will drop dead. On top of that, the reported speed is very inaccurate.

While the Windows method is fine for unreliable long distance telephone wiring, it's a nightmare for direct UTP or USB links. Hence, modern sync or backup software offers continuous bitstreams as well, transferring all files in one go while checking in parallel.

In my case, disk transfer speeds (35Mb/s) and the connection (USB3) are the only limits. Performing multiple parallel backup processes from the same HD source or to the same HD destination can slow down the process, as the disk-head has to swap around too much.

Some reasons I have come across in the past.

Copy process may have hit a lot of smaller files, this will slow down the rate (doesn't seem likely in this case). "Zipping" the directories into one file would prevent this.

Disk fragmentation, disk head has to reseek before reading  new data - this is where defragging the disk can help (doesn't seem likely in this case).

Flow control - The source disk is much faster than the destination, the intermediate buffer fills up and the whole process has to be put on hold whilst the buffer is emptied. Not much you can do but wait. Sometimes slowing down the process can speed up the copying but unless you want to write a script to copy each file seperately its probably not worth doing.

Errors on the disk, cable or bus - corrupted data causing muliple retries (think you would be unlucky if this was the case, bad disk should cause a clicking sound as the disk heads attempt to re-read a sector).

Network sliding window problem - errors in the transmission causes the data to be resent. Initially the data will be sent in small chunks, for each chunk the destination will return a packet to say its received ok. As the transmission progresses if there are no errors early on the chunks become larger to make the copying process more efficient. The problem is if there is an error the larger chunks have to be restransmitted. The are options you can do with network set-up to improved this but  maybe not a good idea to re-tune if this is a one off as it might impact other network peformance. This is more likely when copying files from your PC to an internet based storage.

Sure there are other reasons but retries and flow control are very common.

It can be so easy that it is your antivirus program thats scanning the files, try turning it off and see if it makes any difference.

Fragmented files is the main cause. Secondly if they are BOTH external hard drives, even if you are "directly" copying one to another, you are creating tons of temporary files on the internal hard drive of the host computer. Try installing one of the external as an internal (even better if both) and you will see the copy times tremendously increase.

Quote -
Errors on the disk, cable or bus - corrupted data causing muliple retries (think you would be unlucky if this was the case, bad disk should cause a clicking sound as the disk heads attempt to re-read a sector).

Ah, so some of the windows cab and bus drivers are threatening union action? That would explain it. 

Seriously, though. The explanations have been very enlightening. Hope the OP has come back to read them.

Im going to assume your drives are USB...

Most USB speed issues start at the bios level. If USB is set to legacy, slow, or 1.1 compatability mode in the bios, max transmission speed is capped by the bios. Windows will show 18 to 35 meg a second when you first start to copy, but soon realize it is no where near that and cut the computed speed back.

Usb controller types can further compound this. Most controllers are half duplex when more than one device is on the bus. Some don't support high speed at all.

For instance, USB1.1 is 12 meg a sec in fast mode, 1.5 meg a sec in slow mode. Sound familiar?

If your motherboard has seperate usb controllers, plugging the drives into different ones can help as well. But not always...

Memory allocation for the controller(s) and cpu speed also add to this confusion. If all of them are configured to use the same memory blocks, your stuck at slow speed as soon as two devices are used at once. Doesn't matter how many controllers you have then.

Then there is the power issue. If your external drives are just powered by the USB cable, the controller in the drive may default to slow speed right off the bat.

Not all cables are created equal either, there are ones that only work right on 1.1.

There were a lot of Intel chipset drivers that were incorrectly configured as well. USB 3 ports should be routed to the xHCI ports, not the ECHI ones. Updating the motherboard drivers to the newest version will usually fix this with the acception of some Win8+ drivers for oddball chipsets.

The best solution to all of this is installing an eSATA card and getting eSATA cables and enclosures for the drives. This wont work if the drives are IDE, you will need new drives. If that is the case, get a 6gig a sec card and drives. eSATA enclosures usually do eSATA and usb, so they will be backwards compatable to PC/notbooks without eSATA.

Many thanks everyone for your assistance. Just one question: The two programs mentioned codesector and syncbackfree.Do I put one of each on my portable USB drives or the main c:/ drive? Cheers

Main drive.

Thanks! Cheers

It blows my mind...Ive been running Poser for ever (since version 1) and bought a crapload of stuff over the years yet my runtime is only 48gig. I guess I'm not keeping up with the joneses?

I wouldn't complain.  It's digital hording; you think you're going to use it soon but it never happens.  Or use content once or twice, end the project and forget to delete it.