LTO Tape, A Data Storage Solution for the Modern Multimedia Artist
At some level, photographers and videographers in this modern era are simply data collectors, journeying about our metro areas to bring home the bits. And, on so many sad days, we are no more than data processors: ingesting, tweaking, and delivering the good gigabytes (and, perhaps, the occasional paper product).
So what happens to the gigabytes once the goods are delivered? Ideally – and as highlighted by advertising material like this: Chase Jarvis TECH, Complete Workflow and Backup for Photo + Video – data professionals take over, safely squirreling our work away on monster server racks in humming, temperature-controlled back rooms, backed up overseas and on other planets as well. Perhaps, hyperbole of the elite aside, a multi-workstation mid-size studio might upload files to a backed-up server in a spare closet.
But what’s a winning data storage and archiving plan in a humbler scenario like this: a single hard-working Mac Pro, a Mac Book Pro for field work, and a trusty Drobo or RAID box backing them up? What that smaller studio needs, and what is so difficult to find, is a good solution for long-term data storage, some safe way to archive away the processed and delivered goods to keep the computers cleared for new action.
Options for Data Storage
Certainly, there are server options, dedicated computers with hard drives arrayed in utopian rack-mounted order. These are expensive, but fast, and wonderfully accessible from multiple computers. Servers are a great option for ingesting files and managing large image catalogs in a multiple-workstation environment, cloud like in the low demand for workstation storage.
There are also many options for essentially dumb servers, RAID boxes and NAS drives designed to be managed by a workstation. 2, 4, 5, 8, or 10 drives spinning away in RAIDed unison, these can be connected via USB, FireWire, local network, eSATA, or Thunderbolt. RAID boxes are certainly capable of storing the digits, and can be backed up and readily accessible. Check out our reviews of two favorites: one from Data Robotics, one from Other World Computing.
The two RAID boxes in my own studio include a Mac Gurus “Burly Box” running via an eSATA port multiplier that has been backing up my workstation nightly for 5 years and has worked flawlessly. There have only been rare occasions for recalling files from its archive, with the odd accidental deletion and, on one memorable occasion, a failed 1 TB hard drive. It has proven reliable and very easy to run. Twice, I have upped its capacity with newer, bigger drives. At one point, it made the transition from two RAID pairs to one big quadruple RAID that is a backup copy of a RAID pair in my Mac Pro tower. To avoid instantly duplicating errors in data management, it is backed up using Econ’s Chronosync software (reviewed here) which is set to archive changes and deletions. Like so many of its hard-drive-packing brethren, it is an excellent and trust-worthy backup solution.
But, and this is the key turning point: backup is not long-term storage and it is certainly not archiving. A RAID system is no good for long-term storage or archiving for two reasons. Firstly, hard drives are known to only last a few years and, Secondly, aside from the incremental advancement of hard drive technology, it is not scalable once all the drive slots are maxed out. It might make sense to periodically buy new boxes and occasionally to move content to newer, fresher drives, but, in the end, that plan is labor intensive, subject to human and machine error, and, what’s worse, it isn’t cost effective.
For studios that have more than a few terabytes already and who plan to gather more in the years to come, there must be a plan to clear computer hard drives and store data someplace safe, a plan for data archiving, a modern-day version of the historic stacks of file boxes in office basements of yore. Are expensive hard drives, delicate and designed to last only a few years, our only option for data storage? Is there a feasible option short of setting up an expensive server system or signing a life-long monthly-fee contract with an off-site cloud?
To find that missing link, I have been pouring through options, interviewing professionals, and reading extensively on the technology and am confident in the long-term data storage option presented to you here. I first encountered the technology while shooting an editorial story at the National Accelerator Lab in Menlo Park, CA (SLAC). A very large particle collider experiment had recently concluded, leaving a whole data center worth of digits to be safely stored for thirty years or so. The data won’t be changed, just processed, studied, and analyzed. At the heart of this data-storage Goliath, is a high-tech, robot-controlled, storage unit crammed with, of all things, tapes. Tapes? I have data stored on tapes too, though instead of cutting-edge particle physics, it is C&C Music Factory and Jimmy Buffett. Could it be that cousins of these same low-brow plastic boxes are a leading candidate for safe, cost-effective data storage?
Yes they are. The current generation of the popular “Ultrium” family of data tapes is called LTO-5, featuring 1.5TB tapes that cost around $40 a piece and are designed to last 30 years. Unlike hard drives, the brains, motors, and parts to read and write data are located in the player, not the tape. So the player, the LTO-5 tape drive, is an expensive up-front cost, but the actual media is cheap. Considering a 10-pack of tapes can be bought for a bit more than a single hard drive, it’s easy to imagine the system scaling up to accommodate the fruit of the most trigger happy multimedia artist, including a duplicate tape set mailed off to Mom and Dad’s basement.
Yes, after a month of hard testing the hottest LTO backup system money can buy, I am convinced this is a good option, perhaps the First good option, for small studios to really set up a professional system of safe data storage. LTO-5 is a good option to clear internal hard drives and RAID back up for current projects, extending the life of workstations, and allowing a smooth-functioning, affordable, infinitely scalable studio data management plan.
In the following sections, I’ll explain the technology and propose a workflow for digital artists who might be in this single-work-station, high-data-valume scenario. This is one specific technology that has a specific applications, but I’ve found nothing better to fit the niche.
LTO, “Linear Tape-Open” (on the Wiki) is a tape format designed cooperatively between a small number of companies. The technology has been evolving since LTO-1 was introduced in 2000 and has outlived other tape technologies by offering ever-expanding speed and capacity. Like hard drives, drives for LTO tapes can be had in internal or external configurations and in single, double, or large library arrays. Currently, LTO-5 is the most advanced generation, offering 1.5 TB tapes (which are sometimes marketed as “3 TB” assuming hypothetical compression that will have no effect whatsoever on already-compressed JPG, TIFF, MOV, MP3, AVCHD, H.264, NEF, CR2, 3FR, PEF, MOS, CRW, DNG, IIQ, etc.) LTO-6, offering 2.5 TB per tape, is expected in the first half of 2013. The LTO standard is that tapes from previous generations are read and writable on current drives, and older tapes dating back two generations can be read.
Comparing familiar hard drives to LTO tapes, meanwhile, is like comparing CDs to audio cassette tapes. CDs can jump from track to track, while cassettes, though they can be fast-forwarded or rewound, move linearly from end to end. LTO tape drives read and write linearly, which makes “traditional” jump-happy hard drive use impractical. Given that, most operating systems don’t support the format natively (Mac OS-X doesn’t). Instead, LTO is managed via proprietary software and there are a few options. For this test, we used BRU PE, a program designed with a raft of familiar backup tools and a nice blend of drag-and-drop ease and complex customizeability. LTO tapes aren’t read like hard drives, instead data is written to tape for back up or storage and restored to active hard drives for editing.
One point of consideration for LTO technology, as it is designed with large data systems in mind, is that the only available connection to a desktop computer is via SAS. SAS, a cousin of eSATA, is a plug that looks like an overgrown, industrial USB plug. Such plugs can be added to a Mac Pro via a PCIe slot and to virtually any computer via an external PCIe case mounted with an SAS card. TOLIS Group, developer of BRU PE and integrator of the hardware system we tested, bundles PCIe card, tape drive, software, and all the appropriate cabling for a working high end LTO system.
BRU PE (or BRU Producer’s Edition) writes data sets to tapes and catalogs the included files. Files can be found in the catalog along with instructions for which tape to pull off the shelf: pop the tape in the drive and run a “restore” to have the selected files placed back on to the hard drives from which they came (or select a new destination). Though in some ways, it isn’t as convenient as connecting a Firewire hard drive and copying files across, it is more convenient than swapping hard drives out of a RAID box or computer tower, and the bits and bytes flow more quickly too, considering the system’s beefy and remarkably fast SAS connection. BRU PE embeds a small file on each tape with what is effectively a table of contents. Files can be selected and restored, then, to any computer with Bru PE, even if the original computer and tape catalog are lost. A unique feature of BRU PE is the one-click switch between super-easy drag and drop interface and a more complex and full-featured view.
BRU PE can also be set for scheduled backups and for writing to large tape libraries instead of single drives. In many scenarios, LTO is used to backup servers by rotating among multiple tapes in a library, allowing restores from various points in recent history, like Apple’s Time Machine on an industrial scale.
Screen Grabs from BRU PE
On the Bench
The bundled components from TOLIS Group are of the highest quality and presented in an impressively comprehensive package. As a photographer and filmmaker, I’ve had to spend far more time that I’d like tinkering with computer components, drivers, PCIe cards, and hard drives, but I’m by no means a computer genius. So, a complete package like this, that all works together, and that includes the appropriate software for my computer, well, that’s a great thing. I opened the Mac Pro and installed the PCIe card with two big SAS plugs and hooked up the intense-looking, solid-feeling red box, right on top of the tower. I tested the single-drive version, though many people will be attracted to the automatic backup potential of the two-drive variant (BRU PE has an option to write two identical tapes simultaneously).
Once the appropriate drivers and software was in place (and I only had to call the friendly support line once) the first tape was buzzing away after just a few minutes of familiarizing with the interface. The software offers both an incredibly easy interface and a great deal of depth for more advanced users. It isn’t as pretty or slick-looking as, say, the Chronosync software I use for RAID backup, but it does inspire confidence and is easy to navigate.
One potential negative of the system and, in fact, of LTO in general, is the dependence on SAS for interface. I have eSATA ports in my Mac Pro tower for RAID boxes, and am comfortable with the process, but would like the ability to more spontaneously load data on to other machines, the vast majority of which don’t have so exotic a connection. Also, though it is a remarkably fast way to move a lot of information, USB 3 and Thunderbolt are more modern and much more widely spread. External eSATA boxes, like the unit from Sonnet included in Bru’s Thunderbolt desktop solution, work well (we tested it) but are large and unfortunate compared to a hypothetical tape drive with USB 3 or Thunderbolt. Many video professionals, also, will have maxed out Mac Pro PCIe slots with GPUs, GUIs, SDI or audio interface cards, Red Rockets, etc. etc.
The Sonnet PCIe expansion box bundled with TOLIS’s Thunderbolt solution is a good option for keeping things relatively compact. The box we tested is bigger than you might think, with the size and infrastructure to handle full-size GPUs. If an extra port or two is the only intended addition, photographers will likely do better with TOLIS’s smaller-sized option designed for more modest-sized cards. A popular option among video editors and colorists is the Cubix line of PCIe expanders like this one which would house any double-size, heat-intensive, or power hungry cards leaving the Mac Pro’s PCIe slots open for SAS, eSATA, and other less demanding PCIe cards.
In testing, once the rig was up and running, I archived an entire year of my studio’s image files in a few hours by simply selecting the root level folders on my RAID, dragging it into the Bru PE interface and starting the process. It sounds like a placid cousin of R2-D2 in action, buzzing and whirring while the data flows, not loud, but not the forgettable white noise of a RAID either. The verified copy is then ready for archiving. Though square-shaped and smaller, the actual tapes are reminiscent in size and build-quality to old VHS tapes (be kind, rewind!) They are tough boxes that feel archival. From the backup, I was able to very easily complete a full restore in a few hours. After dragging and dropping folders or files into the Bru interface, the software calculates how many tapes will be needed, including separate numbers for both LTO-4 and LTO-5. Data sets larger than LTO-5’s 1.5 TB or LTO-4’s 800 GB can be easily written across and restored from multiple tapes.
In a separate test, I archived 1.5 TB of Final Cut X project and event folders. These too are easy to drag and drop and then restore. Want to call up only a single “event” to revisit an edit or re-render a final, no problem: find the appropriate tape via the catalog and restore it to the same folder from which it came. Simple and fast, and wonderfully safe in the meantime.
The system works and works well. The combined feeling after a month of rough testing is that the format is dependable and easy to use. As the chirping sounds of spinning tape grow familiar and the shelf of little tapes grows, the satisfaction of having data safely archived is a great feeling.
LTO drives are expensive and work only via proprietary software and an SAS connection. Computers will need a PCIe SAS card, either internally or via external PCIe extender. LTO-6 is just entering the market, offering more capacity and speed compared to LTO-5. Pricing can be found on TOLIS Group’s Website and products are expected to begin shipping in January.
LTO tapes are affordable, safe for long-term storage, durable, and easy to use. BRU PE software is straight-forward and functional, with both an easy-to-use interface, and the depth to satisfy the most professional data handler. BRU’s Desktop Bundles are complete and ready-to-install. Tapes are designed to archive for 30 years. When copying from, or restoring to RAID arrays, a whole terabyte of data moves in less than 2 hours, impressively fast.
Though a somewhat niche product, LTO tape deserves a great deal more consideration as a potential tool for data archiving and storage. It has certain limitations when compared to hard drives, including ties to proprietary software and the unusual (in desktops) SAS connection, and the inability to access data without restoring it first to a hard drive. LTO also, however, has several unique traits that will make it the perfect solution for many studios. Data stored on tapes lasts a very long time and they are very hard to beat for the cost per terabyte. If, like me, your library of video clips and RAW files are growing faster than hard drive technology, LTO tape allows an affordable way to scale up to practically unlimited storage, and keep your hard drives free for incoming projects.
To comment on data and other Photo Arts Monthly articles, join us on Facebook. Also, we appreciate the financial support we get when you use the sponsor links at the bottom of this page and links throughout the site for your shopping. Please use the below buttons to share this article far and wide; thank you!
by Matt Beardsley, photos & illustration by Matt Beardsley