Fast File Copy script in Windows

Posted on

A server stack is the collection of software that forms the operational infrastructure on a given machine. In a computing context, a stack is an ordered pile. A server stack is one type of solution stack — an ordered selection of software that makes it possible to complete a particular task. Like in this post about Fast File Copy script in Windows was one problem in server stack that need for a solution. Below are some tips in manage your windows server when you find problem about windows, windows-command-prompt, batch, xcopy, .

I have a process that is run every month or so. Such process identifies a set of files and constructs a proper source and destination path from several fields of different tables in DB to then copy all of these files, so I end up with potentially hundreds of thousands or millions of paths. It is possible that these paths point to a file that has been previously been copied, but has been modified.

How can I perform this copy in the fastest way possible?

I tried at first a simple huge cmd script checking if the file, per resulting path, existed in the destination path and then trying to copy if it didn’t exist, but the modified files were skipped. So, then I tried to use “xcopy” to avoid overwrite if date is the same, by using the /D switch and /Y switch to silently confirm the prompt, but in practice, a huge script with hundreds of thousands of xcopy commands takes forever and I don’t see any speed-up when some or most of these files already exist with the same date. Do you have any ideas that may help me? What can I do with a file containing hundreds of source and destination paths to make a copy of all of these? Is there a faster alternative to xcopy? or a faster date check and then call any copy command?

A more detailed explanation on how I currently get and copy files based on DB information:
There are multiple tables in DB:

Table 1:
ID, Dir_Name, …

Table 2:
ID, Dir_Name, …

Table N:
ID, Dir_Name, …

Source Folder Paths Table:
Dir_Name, Dir_FullPath

Destination Folder Paths Table:
Dir_Name, Dir_FullPath

Then I can create the set of copy commands (or just get the source and destination paths)

Select 'xcopy /d /y '|| T2.Dir_FullPath || '\' || ID || '.ext ' || T3.Dir_FullPath || '\' || ID || '.ext*' 
from Table1 T1 
left join SourceFolderPathsTable T2 on T1.Dir_Name=T2.Dir_Name
left join DestFolderPathsTable T3 T1.Dir_Name=T3.Dir_Name where T1.some_criteria=true;

Thank you!

Take a look at the command line tool “robocopy” that comes with Windows. I believe it may accept a response file, and it already can be told to only update files that have changed dates / times, plus it is about as fast as it can get, being pure command line with no GUI overhead.

Leave a Reply

Your email address will not be published. Required fields are marked *