Read & Write
I previously spoke about the differences between backup and fault tolerance, focusing on the differences between backup as a strategy and fault tolerance as a system. I want to now head into the realms of speed, drives and data rates. This came about after some discussions with a colleague concerning his own video post-production workflow, whereby some limiting factors such as the computers I/O were being overlooked.
When talking about speed, we are referring to the speed with which a hard drive transfers data between the disk(s) and the computer. The speed of this operation can be impacted by how fast the disks spin and the number of disks involved. For example the Western Digital Black drives used in our own systems spin at 7200 RPM.
When it comes to disks, keep in mind that the more data you store on a hard drive the slower it becomes.
A drive is fastest when it is empty. Makes sense right? The drive has less work to do and the mechanical components inside the drive have less distance to travel.
When a drive is totally full, it neither reads back, nor writes. I recommend keeping at least 25% free space on any drive to ensure everything runs smoothly.
Some storage devices don't suffer from this slow down because they are solid state drives (SSD) - meaning no moving parts, no RPM, and no travel time between the inside of the drive platter and the edge.
SSDs offer blazing fast read and write times not limited by technical components, however they have less storage space and - as with anything thats newer, faster, more robust - they cost more.
But no matter what choice of storage is being used, the greater controlling factor is how the drive connects to your computer. This is referred to as a data transfer rate and is commonly the bottleneck that can block performance.
For example, in practical terms, USB 2 has a limited speed on MacOS X to around 18 MB/s. And remember FireWire 800? Well it was better, limiting hard drive speeds to around 85 MB/s. Back in the day, it was that extra speed which promoted video editors on a Mac to use FireWire 800 drives for all editing purposes.
In contrast, a drive attached internally to a desktop (SATA) could transfer data around 120MB/second. So theoretical sequential read and write times for a modern hard drive extend between 120-140 MB/s from a 7200 RPM drive that is under 50% full. But we don't want to keep video on our internal drive which runs the OS.
Purpose = Speed
Where as SATA is still a viable internal connection, FireWire and USB 2 have become digital antiquities. Clearly, as multi-cam editing, video image sizes and frame rates increase, faster technologies are called upon.
Enter USB 3 and Thunderbolt.
Both these connections are very high-speed; think of them as a large pipe. Both of these pipes are really, really big – built to carry lots and lots of data all at once.
USB 3 is designed to transfer data at around 460 MB/s, yet Thunderbolt is built to transfer data at about 1,000 MB/s. Thunderbolt 2 can double that speed. Thunderbolt 3 or USB type C as its also known weights in at approx. 5000 MB/s.
So the bottle necks we had with USB and FW are now gone. We can access the full speed of a SATA drive, around 120-140 MB/s.
Speed is important; a single stream of ProRes 4444 at HD 1080p 25fps, which is one video format Final Cut Pro X can use for optimisation and render files plays at a data rate of approx 34 MB/s. As USB 2 tops out at 18 MB/s its too slow for editing that quality of video. With Thunderbolt and USB 3 suddenly the speed restrictions we faced with FireWire 800 or USB 2 are gone. A single 7200 RPM drive can now handle 3-4 streams of ProRes video at the above specification when supplied by Thunderbolt.
Like any pipe, the contents is only as fast as the pump feeding it. In this case our single solid state drive or mechanical hard drive. For example; if I connect a single hard drive to a computer using a Thunderbolt 2 connection, the hard drive is connected using a large pipe, yet the total amount of data that can be carried is based not on the size of the pipe, but both the pipes width and the pump that is feeding it. This means that if the data transfer rate of the pipe is bigger than the hard drive, the speed of the hard drive determines how much data gets sent to the computer. The bottleneck in the workflow changes from the connection (pipe), like in the USB and FW days, to the drive itself (the pump).
To complicate things a little further, not all pipes are created equally. Both USB 3 and Thunderbolt are excellent protocols. However, as I understand these protocols, Thunderbolt is more efficient at handling large media files.
Say goodbye to mechanical arms, platters and spinning discs that make up a traditional hard drive. A solid state drive can produce read and write times of approx 500 MB/s. An average of 5x faster than a mechanical drive. Combined with USB 3 or any flavour of Thunderbolt a single SSD can offer amazing performance benefits.
This comes with an additional cost; financially, but also in available capacity. SATA drives are relatively low cost in comparison, especially when considering size. A Western Digital Black 4TB 7200 HD retails for around £170. A high end SSD, one comparable with a WD Black, say a Samsung 850 PRO 1 TB would set you back over £320. If you need a large amount of storage, 8 TB+, then SSD might just be unaffordable. At least for now.
Furthermore, SSDs only achieve their speeds when you are accessing the same files over and over. That performance advantage falls off when you are constantly accessing different files.
So we come full circle and arrive at RAID. As described in my other article on Raid vs Fault Tolerance - it's clear RAID offers some distinct advantages to handling large data at speed. It's a cost effective way we can fill the huge data transfer pipes provided by Thunderbolt.
It works by combining multiple hard drives into a single unit called a RAID (Redundant Array of Inexpensive Drives).
If one hard drive transfers data at 120 MB/s, then combining two hard drives can provide nearly double that. 240 MB/s. If we group ten drives together, we can totally fill the size of a Thunderbolt 1 pipe. 20 drives and we saturate a Thunderbolt 2 connection. Theoretically this sounds perfect, the reality isn't always as simple. There are a variety of different flavours called RAID levels, with differing levels of performance. The more drives we add the actual individual drive speeds will be slightly lower.
Another colleague recently approached me to consult on his own backup and storage system. It's easy to think of this as one system which needs to do it all, in professional environments I prefer to treat it as two separate systems. One fast system for working on data to day. The other for backup. We separated backup from working storage and implemented RAID but with two tiers of fault tolerance. These tolerances are a way of ensuring work can continue if the primary, multi drive RAID array should fail.
So, let’s boil this down into a potential workflow, in my colleagues case video post production.
If you are only shooting and editing single camera narrative, a single hard drive connected via Thunderbolt is a good choice. If your computer only supports FireWire, then for single camera with two streams or less its still ok. You still can find FW 800 hard drive system such as the ones from G-Tech and LaCie, or even better look into a Thunderbolt system and use a FireWire to Thunderbolt adapter to convert the connection and further proof yourself. Transfer speeds remain at FW levels.
However, if you are doing multi-cam, stereoscopic 3D, 2K, 4K or high frame rate video, a single drive, no matter how you connect it, will not be fast enough. you need SSD as a minimum, single drive solution but I urge you to consider a RAID that contains at least 2 drives.
My colleague wanted to edit 2K and eventually 4K video.
He needs at least 60 MB/s for the ProRes 2K video, rising to 220 MB/S for the 4K video - again this depends on the ‘flavour or ProRes being used - in this case ProRes 4444 XQ 4K. But no doubt he will need room for multiple streams, audio, graphics and other assets.
He also needs space as that top end ProRes codec takes up approx 764 GB per hour. Its a working system which will only hold 2 - 3 projects at a single time, no more than 9 hours in total which requires a maximum of 8TB for 4K video.
To determine how many drives he needed, we took the number of video files / streams he will play at the same time and multiply that by the average MB/s.
In other words, an eight camera multi-cam shoot in 1080p ProRes would be 8 * 20 = 160 MB/second.
In my colleague's workflow two streams of ProRes 4444 XQ at 4K would be 2 * 220 = 440 MB/s.
Assume that you get 100 MB/s per hard drive, just to be on the safe side and allows for necessary overhead in the RAID. This means a 2 drive RAID will transfer about 200 MB/second of data. Although as shown below with the G-Tech 2 drive system you can get some pretty amazing speeds.
Taking into account the overhead of additional assets such as audio, I recommended a 6 drive system to my colleague. A Pegasus R6 - he will connect over Thunderbolt 1 but the system supports Thunderbolt 2. It provides adequate storage for multiple projects, but equally importantly is allows him to run the system / drives at half capacity, 8TB, to keep speeds high and consistent.
We decided upon a 18TB configuration which will run at 16TB when configured into a RAID 5. This provides a level of redundancy if one drive fails. If two or more drives fail all the data will be lost. If a single drive does go down, replace it and the system rebuilds itself, allowing him to be back working in a few hours. To ensure he can continue working we make sure regular backups are made via software running on the computer. Of the 16TB, 8TB is allocated for use, then an equivalent size 8TB external / separate Thunderbolt 8TB enclosure, made of 2 4TB drives configured to RAID 0 can provide an additional level of protection. This protection can be increased by expanding disk size and configuring a RAID 5, 6 1+0 etc.
With this setup we were able to achieve speeds of approx 700MB/s Writes and 800 MB/s reads.
For extra speed he could run at RAID 0 on the working volume, but thats throwing your balls to the wall. If a single drive goes down, you lose everything unless you're backing up regularly. I worked like this for a year when I needed every MB of speed on my last documentary. Never again. Despite the fact I never had an issue, a testament to the Promise system and WD drives, it wasn't worth the worrying. Despite creating a working clone and office backup.
Some users prefer RAID 6, 1+0 or another flavour of RAID. On the Pegasus R6 I found RAID 5 to be the best balance of reliability, protection, speed and storage. I suggest before you copy any data to do some tests in different configuration, finds what works best for you and your setup. Configure your system and use a speed checking utility such as the one provided by Blackmagic.
Some final words I have heard allot around this topic; Avoid Drobo systems for a working drive, perhaps even for backup. Think about how to get your data backup - offsite. USB 2 is dead, long live USB C and Thunderbolt, ethernet is not fast enough for large video files, dedicated systems for specific workloads, RAID is NOT backup but a level of operational protection, speed and mass storage.
Lets talk backup soon.
There are variations in drives and drive speeds, numbers quoted are based on my own research and tests with my own system. These numbers should be considered ranges, personally I have seen single drives and Raid systems both faster and slower than these numbers.