Jump Starting Off-Site Storage

During a discussion about Cloud Storage on the StorageMojo blog, Pete Steege asked about the challenges of the initial load of data into a cloud storage provider.

How are storage cloud companies handling the “first backup” issue? Multiple terabytes or petabytes that need to be migrated to the cloud initially?

The incremental part of the process is a no-brainer.

One solution that we use at Bycast is to deploy two or more edge servers with attached storage at the customer's premise, and allow them to perform bulk ingest over the REST API (or via CIFS/NFS). When the ingest is complete or nearly complete, the object storage repository on disk can be physically shipped to be integrated into the "cloud", and subsequent transactions can be performed over the network.

Another approach taken by EMC's Mozy backup service is what they call "data seeding". Here, you purchase a 2 TB USB drive from EMC, store your data onto the drive, then ship it back to EMC in order to get the process going. I couldn't find any current references of this capability on their web site, so this may no longer be a supported feature.

With such hybrid models, you need the software intelligence to ensure that the data is always accessible via the cloud API, always protected from loss, corruption and unauthorized disclosure, and to ensure that the data is audited to ensure that you know that all data that you ingested locally made it into the cloud.

Despite the associated complexity, this is a very powerful approach, as one should never underestimate the bandwidth of a 747 full of disks or tapes.

Welcome, EMC

Today, EMC announced their Atmos product, perviously code-named "Maui". It is an interesting offering, and one that we are quite familiar with... given that object-based storage is what Bycast has been developing since 2000, and has had deployed and operational in customer sites since 2002.

Let's do a quick rundown of the core features that Bycast StorageGRID offers that are also offered by Atmos:

• Object based storage? Check
• Metadata with objects? Check
• Multi-site? Check
• Multi-tenancy? Check
• File-system interface? Check
• Web-services API? Check
• Policy-driven replication? Check
• Compression? Check
• Object-based De-dupe? Check
• Object versioning? Check
• MAID-style spin-down? Check
• Web-based admin? Check

There are also a couple of significant features found in StorageGRID that are not present in Atmos:

• Encryption? Unknown
• Object integrity protection? Unknown
• Clustered gateways? Unknown
• Storage hardware independent? Nope
• Storage vendor independent? Nope
• Support for tape and other high-latency media? Nope
• Six years of operation in the field? Nope
• Mature, customer-proven product? Nope

Over the last five years of deploying large-scale distributed object-based storage systems, we've learned a lot about what things work well, and where things go wrong. Based on the initial feature set, we know that Atmos will be a successful offering for EMC. There will, of course, be the usual teething pains, not unlike those that happened with Centera, but that is the nature of all 1.0 software.

So, to our friends at EMC, I say, Welcome. We're glad to see object-based storage becoming mainstream.