Robocopy is an old-school command line tool that ships with Windows and Windows Server operating systems. Simply put, it copies file data from one location to another.
In my use case, I needed to move a bunch of data from an on-prem server to a cloud server: ~20 GB for the FileMaker files themselves and a whopping 105 GB of container data.
Using traditional methods (compressing and segmenting the data using 7-zip + WinSCP for the transfer), the transfer took ~6 1/2 hours. Since this was a client’s production environment, that would have meant at least 6 1/2 hours of downtime.
Enter Robocopy. This utility can be configured to effectively act like a mirror for the data between the on-prem server (the current production environment) and the cloud server (the future production environment).
When executed, Robocopy simply compared the source file directory to the target file directory and updated the target directory if there was a discrepancy between the two. Updates in the current PROD were transferred to future PROD each time we executed the Robocopy command.
In the week leading up to the cloud server cutover, we ran the Robocopy updater 5-6 times. Each time, the updater brought future PROD into parity with the current PROD.
How to Use Robocopy
To make all of this happen, we only needed to do the following:
- Map a drive on the on-prem server to the target server.
- Open port 445 in the firewall of the target server. (Robocopy uses FTP to transfer the files.)
- Execute the Robocopy utility from the command line on the on-prem server.
Building the Command
The command itself is straightforward. Here’s its basic structure:
robocopy <source> <target> <file> <options>
From Microsoft’s help page:
| Parameter | Description |
|---|---|
| <source> | Specifies the path to the source directory |
| <target> | Specifies the path to the target directory |
| <file> | Specifies the file or files to be copied. Wildcard characters (* or ?) are supported. If you don’t specify this parameter, *.* is used as the default value. |
| <options> | Specifies the options to use with the robocopy command, including copy, file, retry, logging, and job options. |
Source: Microsoft
In my case, for the Filemaker container data, the basic command looked like this:
robocopy D:\Containers\RC_Data_FMS\ Z:\RC_Data_FMS\
That worked just fine for refreshing the target, but the real power came by tapping into the additional options that are available with the tool. To speed things up, we applied a series of options to suppress logging, take advantage of multiple threads, and restrict file write retries and retry wait times.
The revised command with our options applied looked like this:
robocopy D:\Containers\RC_Data_FMS\ Z:\RC_Data_FMS\ /MIR /Z /R:5 /W:5 /NP /NFL /NDL /MT:128 /log+:robocopy.log
Copy Options
/MT:128 – this option makes the command multi-threaded, and you can spawn up to 128 threads. The default value for the /MT option is 8 threads.
USE THIS WITH CAUTION: When running from the FMServer, this will consume CPU resources and outbound Ethernet bandwidth. Before opting to use 128 threads, we monitored the server to ensure normal business operations would not be negatively impacted (during the transfer, our CPU utilization percentage hovered around 35%). Reduce the thread count to reduce this load on the server.
/MIR – this option ensures the directory tree is mirrored in the target environment and applies the appropriate security settings.
/Z — this option makes the command restartable. If something were to interrupt the file copy process (lost connection, etc.), robocopy could resume the process where it terminated rather than re-copying the entire file.
Retry Options
/R:5 – this option allows you to control how many times robocopy will retry the copy process if there is a failure. The default value is one million retries, so we limited this to speed up performance. Failures are logged (see below), so we could circle back on any file or set of files that did not get properly transferred.
/W:5 – this option gives you control over how long robocopy will wait between retries when an error is encountered. The default value here is 30 seconds, so like the retry option, we limited this value to 5 seconds to improve processing speed.
Logging Options
/log+:robocopy.log – this option logs the outcome of each discrete folder and file copy process in a file named according to the value you specify (e.g., we named the file ‘robocopy.log’). That file is created in the directory where you execute the command.
/NP – this option suppresses the progress of the copy operation (i.e., objects copied so far in the process) from the terminal window display
/NFL – this option suppresses the file names from being added to the log (unless there is an error)
/NDL – like /NFL, this option suppresses the directory names from being logged.
Performance was dramatically improved by using these logging options in concert with one another. Instead of the tool using resources to display the copy progress (across hundreds of thousands of objects in my case), resources are maximized and focused on the data transfer itself.
In the end, incrementally refreshing the container data on the target environment saved a ton of time! The original container data transfer took 5 1/2 hours. By the time the actual cutover window arrived, we were able to refresh the container data on the target server in just 9 minutes with Robocopy.
This, combined with a predefined deployment checklist and several trial runs made the whole deployment go off without a hitch. The client, expecting us to be offline for most of the evening, got an email from me about an hour after we started saying that everything was successfully migrated. It was beautiful!
Launching Robocopy for Your Application
Further information about Robocopy, along with many additional command options that are available but not covered here, can be found on the Microsoft website.
Our team of deployment experts regularly leverage insights, best practices, and tools like robocopy to ensure we launch our clients’ applications efficiently and with as little downtime as possible. To learn more about how we can help you build custom functionality and launch it flawlessly, contact our team to talk to a consultant and learn more.
