Anyone who regularly transfers large amounts of data knows that transfer times can be unmanageable when large quantities of data are involved. As a DIT (Digital Imaging Technician) I am constantly tasked with transferring large video files within limited time constraints so it is essential to be able to determine how long a particular transfer will take considering the circumstances, and balance risk with speed. It’s always a good idea to transfer data using the safest method possible but sometimes there is only time to copy and paste the data, the least safe, but fastest means of transferring.

While on the topic of data safety, hard drives are prone to failure, or getting crushed by a steamroller if you leave it in the wrong place, so it is essential to back up your data to not one but two hard drives before clearing camera drives and cards. Of course, it tends to be much safer to back up from these original camera drives than it is to copy to one drive, and then back up from the copy. Any errors that may have occurred in the original transfer would then be transferred to the backup. So we always transfer and make the safety backup copy at the same time, which means we are copying the files to two separate drives either simultaneously, or one after another.

Using a 10GB RED R3D file I ran a series of tests under a variety of parameters. The results can easily be calculated to fit any amount of data under various situations and transfer methods.

Who Should Care?

This data is useful to anyone who transfers large files and would benefit from estimating the time the transfer will take using various methods of transfer and/or various levels of data checking. The most important job as a DIT is to ensure that files are transferred and backed up safely without data loss. For footage shot on the RED, we use an application called R3D Data Manager which was designed specifically to transfer RED footage with various levels of data safety. The application will give an estimate of how long the transfer will take however, like Apple’s Finder, the times are horribly inaccurate. Our readings below were manually timed using various configurations and are thus far more accurate estimates to work from than the applications “instant estimates”.

Methodology:

I ran a series of trials using a 17″ MacBook Pro with a 2.16 MHz Intel Core Duo processor, 7200 rpm FireWire 800 and eSATA hard drives, and the transfer software R3D Data Manager version 6.4.

The variables we included in the test were the ones we see on location on every job:

– Using one FireWire 800 drive to another FireWire 800 drive to copy the file.

– A FireWire 800 drive writing to an eSATA drive using an eSATA Express Card.

– And a FireWire 800 drive copying the same file to two separate FireWire 800 drives simultaneously using 2-port FireWire 800 Express Card.

With each drive configuration I transferred the file using a variety of methods, including copy and paste, and R3D Data Manager with various levels of file checking protocols enabled. Each transfer was timed and logged for comparison. All data is in the format of minutes:seconds.

Transfer Speeds Of a 10GB File

FireWire 800 to eSATA (Single Drive)
Copy and Paste: 02:26
R3D Manager with No Checks: 02:41
R3D Manager with MD5 (Safe): 04:48
R3D Manager with SHA-1 (Safer): 05:39
R3D Manager with SHA-256 (Safest): 08:33

FireWire 800 to FireWire 800 (Single Drive)
Copy and Paste: 03:17
R3D Manager with No Checks: 04:37
R3D Manager with MD5 (Safe): 07:10
R3D Manager with SHA-1 (Safer): 07:51
R3D Manager with SHA-256 (Safest): 10:05

FireWire 800 to 2x FireWire 800 Drives (Simultaneous)
Copy and Paste: 04:38
R3D Manager with No Checks: 06:42
R3D Manager with MD5 (Safe):10:43
R3D Manager with SHA-1 (Safer): 11:04
R3D Manager with SHA-256 (Safest): 13:03

Because the slower methods with more detailed checking tend to be safer, it is always best to utilize those methods when time permits, however in situations of greater urgency, the chart can be used to determine what method would be the safest yet still finish in the time allotted. All of the times listed in this study are for a 10 GB file, making it easy to calculate the transfer time for any batch of data – so If you are working with 120 GB of data using FireWire to eSATA you can calculate that the transfer using R3D Data Manager with no checks enabled will take approximately 8 minutes and 12 seconds. We were surprised to find that transferring to two FireWire 800 drives was so much faster than copying one at a time, and it doesn’t take much longer than it would to transfer to a single drive (and, of course, switching out drives adds time to the copying process as well as meaning that you have to be there babysitting them for more of your valuable on set DIT time).

Hope that you can benefit from this!