Vanishing Into the Infrastructure

Not the End, Just the End of a Beginning

2010-06-01T17:35:00.000-07:00

Today, I am proud to announce my new blog as part of the NetApp blogging community: Objects in Context.

I am very pleased to join the NetApp team, and to have this opportunity to start writing a second chapter. January 2010 marked the passage of a decade since I first started working on object storage, and we've seen many milestones, from the first large-scale deployments to the development of industry standards. Over the next few years, I fully expect things to accelerate — After all, the Internet changes everything, and storage is just starting to catch up.

Moving forward, I will be re-purposing this blog here to discuss topics such as information visualization, radio design, iPad development and other similar non-work-related subjects. I encourage all my readers to continue to follow my adventures in object storage over at the NetApp blog. We've got many exciting things in store, and I can assure you that in this case, one plus one is far greater than two.

CDMI Functional Areas

2010-03-30T17:07:00.000-07:00

With the 1.0 release of the Cloud Data Management Interface standard quickly wrapping up, below is my summary of the major functional areas covered in the 1.0g draft:

Object Access by Name - Sections 8 and 9

Cloud storage clients can store, list and retrieve objects by name. This is the most common method for accessing objects, and allows the placement of objects into "containers" to group together like objects, in a similar manner as directories in a filesystem.

For more details, see CDMI Tutorial - Basic Input/Output.

Object Access by ID - Sections 8 and 9

Cloud storage clients can also store and retrieve objects by ObjectID. Every named object also has an Object ID, but not all objects have a name. When unnamed objects are created, they can only be accessed by Object ID. Storing Object IDs in a database is more efficient than storing URIs, and this mode of object access is more suited when access is ID based or query based.

Data System Metadata - Section 16.4

Data System Metadata is a set of special metadata items that are interpreted by the cloud in order to allow the cloud storage client to specify the level of service, protection, placement and other characteristics for stored objects.

For more details, see CDMI Tutorial - Data Management, Part 1.

Serialization - Section 15

Serialization allows objects and containers to be transformed into a portable data object that can be deserialized back into the original objects and containers. This is useful for archiving and for system-to-system transport.

Domains - Section 10

Domains specify administrative control within a CDMI cloud, and define how users are mapped to permissions, specifies delegation of authentication and authorization, and provides usage summaries.

Queues - Section 11

Queues are special data objects that can store multiple values in a first in, first out access semantics. Queues represent a key technology for connecting together applications in the cloud, as they enable reliable inter-process communication with underlying persistent data storage.

Query - Section 11.1.3

Query is implemented as a queue interface that allows a client to express in a standardized manner a query, and for a query engine to create a response as a CDMI data object with a specific format of results.

Notification - Section 11.1.1

Notification builds on top of queues, and allows a client to subscribe to a client-defined set of notifications about operations performed against the storage cloud.

Audit - Section 11.1.2, 17

Audit builds on top of queues, and allows a client to subscribe to a client-defined set of log messages about system operations.

Exports - Section 13

Exports allows a client to specify and control how access to named objects are provided through network files protocols. Exports also allow a container to be exported as a block device.

Snapshots - Section 14

Snapshots are the ability for a client to specify that access to a set of named objects and containers should be preserved.

Retention - Section 18

Retention and Hold are a set of functions that allow a client to specify that an object may not be modified or deleted, and for how long the restrictions must remain in place.

The World's Shortest CDMI Implementation

2010-02-24T14:51:00.001-08:00

The CDMI standard is built around the concept of "capabilities", which describe which functionality a given cloud storage system provides to clients. As a result, it permits a system to only implement a small subset of the standard while still being compliant.

So, to demonstrate this, I present to you the world's shortest* CDMI implementation, written in Ruby:

# World's smallest CDMI implementation, or, how to say NO in CDMI.
require 'socket'
require 'openssl'

listener = TCPServer.new('', 2000)

# Set up TLS
ssl_context = OpenSSL::SSL::SSLContext.new()
ssl_context.cert = OpenSSL::X509::Certificate.new(File.open("cdmi_server.cert"))
ssl_context.key = OpenSSL::PKey::RSA.new(File.open("cdmi_server.key"))
ssl_listener = OpenSSL::SSL::SSLServer.new(listener, ssl_context)

while (connection = ssl_listener.accept)
    request = ""
 while(request.index("\n\n") == nil)
  request << connection.gets
 end

 print "-- CLIENT REQUEST -------------------------------------------------------------\n"
 print request
 print "-------------------------------------------------------------------------------\n"

 if(request.index("GET ") == 0)
  uri = request.slice(request.index(" ") + 1, request.length)
  uri = uri.slice(0, uri.index(" "))
  
  if(uri == "/")
   connection.puts("HTTP/1.1 200 OK\nContent-Type: application/vnd.org.snia.cdmi.container+json\nX-CDMI-Specification-Version: 1.0\n\n{\"objectURI\" : \"/\", \"objectID\" : \"AABwbQAQgTpfe4qRBsyCCw==\", \"parentURI\" : \"/\", \"capabilitiesURI\" : \"/cdmi_capabilities/\", \"completionStatus\" : \"Complete\", \"metadata\" : {}, \"childrenrange\" : \"0-0\", \"children\" : [\"cdmi_capabilities/\"]}")
  elsif(uri == "/cdmi_capabilities" || uri == "/cdmi_capabilities/" )
   connection.puts("HTTP/1.1 200 OK\nContent-Type: application/vnd.org.snia.cdmi.capabilities+json\nX-CDMI-Specification-Version: 1.0\n\n{\"objectURI\" : \"/cdmi_capabilities/\", \"objectID\" : \"AABwbQAQnP7GJT2muKDelQ==\", \"parentURI\" : \"/\", \"capabilities\" : {\"cdmi_security_https_transport\" : \"true\", \"cdmi_read_metadata\" : \"true\", \"cdmi_list_children\" : \"true\"}, \"childrenrange\" : \"\", \"children\" : []}")
  else
   connection.puts("404 Not Found\n")
  end
 else
  connection.puts("501 Not Implemented\n")
 end
    
 connection.close
end

* It's a little longer than it could be, because it is written for readability. Removal of comments, code tightening, etc, is left as an exercise for the reader.

Now, since this uses TLS, you'll need a TLS-compatible client. Here's a test client for this purpose:

# Test client for a minimal CDMI implementation
require 'socket'
require 'openssl'

socket = TCPSocket.new('localhost', 2000)

ssl_context = OpenSSL::SSL::SSLContext.new()
ssl_socket = OpenSSL::SSL::SSLSocket.new(socket, ssl_context)
ssl_socket.sync_close = true
ssl_socket.connect

ssl_socket.puts("GET #{ARGV[0]} HTTP/1.0")
ssl_socket.puts("accept: application/vnd.org.snia.cdmi.object+json")
ssl_socket.puts("X-CDMI-Specification-Version: 1.0")
ssl_socket.puts("")

print "-- SERVER RESPONSE ------------------------------------------------------------\n"
while line = ssl_socket.gets
 print line
end
print "-------------------------------------------------------------------------------\n"

In order to run these, you need a x.509 certificate and key. You can generate these using OpenSSL. You can follow the below instructions for apache, then instead of step 5, copy the certificate to "cdmi_server.cert", and copy the key to "cdmi_server.key".

http://www.akadia.com/services/ssh_test_certificate.html

A CDMI Tutorial - Data Management, Part 1

2010-02-23T12:42:00.000-08:00

As we covered in first part of this tutorial, the SNIA Cloud Data Management Interface provides a RESTful mechanism for the basic storage and retrieval of data. However, the core of the standard is focused around data management — how objects are stored, delivered, placed, protected and more.

This post is the second in a series on CDMI. Subsequent posts will cover the following areas:

Basic Input/Output
Data Management, Part 1 (This post)
Data Management, Part 2
Advanced Input/Output
Cloud-to-Cloud Interactions
Queues and Query
Authentication and Access Control
Billing and Accounting

Data Management and Metadata

CDMI storage systems see the world as a tree of objects and containers. Objects store data, and containers contain child objects and containers. Regardless of if the data is stored via HTTP or via traditional protocols such as NFS or CIFS, all data represented through the CDMI protocol is fundamentally seen in terms of objects and containers.

Management of stored data is enabled through metadata. Every container and object can have metadata associated with it. Metadata in CDMI is organized into three general categories; user metadata, storage system metadata, and data system metadata.

User Metadata

User metadata is set directly by CDMI clients, or indirectly through the extended metadata interfaces of other access protocols. For example, in NFS, extended attributes can be mapped to CDMI user metadata items. User metadata items are arbitrary, and are not interpreted by the storage system.

Storage System Metadata

Storage system metadata are generated by the CDMI storage system, and provide read-only access to information about the stored data that is managed by the storage system. The creation time of a object is a good example of a storage system metadata item.

Data System Metadata

Data system metadata are provided by a CDMI client, or specified through an out-of-band management interface, and determine how the stored data should be managed. For example, data system metadata can specify the degree of replication, or an encryption level desired to protect data while stored on disk.

It is through the specification of data system metadata that CDMI enables the management of how data should be stored.

Example Object Metadata

As an example, let's assume that we have a CDMI-enabled storage system that also provides a NFS share, and we've stored some documents onto it.

We can then connect to the system via CDMI, and access the CDMI metadata of the "Documents" container:

GET /Documents/ HTTP/1.1
Host: cloud.example.com
Content-Type: application/vnd.org.snia.cdmi.object+json
X-CDMI-Specification-Version: 1.0

HTTP/1.1 200 OK
Content-Type: application/vnd.org.snia.cdmi.container+json
X-CDMI-Specification-Version: 1.0
{
  "objectURI" : "/Documents/",
  "objectID" : "AABwbQAQ8ypO85j/ml8TZQ==",
  "parentURI" : "/",
  "accountURI" : "/cdmi_accounts/default_account/",
  "capabilitiesURI" : "/cdmi_capabilities/container/",
  "percentageComplete" : "Complete",
  "metadata" : {
    "user.DosAttrib": "0x20",
    "cdmi_ctime" : "2009-12-29T12:43:32.479832Z",
    "cdmi_atime" : "2010-01-02T16:12:53.521983Z",
    "cdmi_mtime" : "2010-01-02T16:12:53.521983Z",
    "cdmi_acount" : "52",
    "cdmi_mcount" : "12",
    "ACL" : {
      "acetype" : "0x00",
      "identifier" : "jdoe",
      "aceflags" : "0x03",
      "acemask" : "0x000F005F",
      "acetime" : "2009-12-29T12:43:32.479832Z" 
    },
    "cdmi_data_redundancy": "2",
    "cdmi_immediate_redundancy": "2",
    "cdmi_infrastructure_redundancy": "2",
    "cdmi_geographic_placement": [
        "US" 
    ],
    "cdmi_encryption": "AES_256_CTR",
    "cdmi_data_redundancy_billed": "2",
    "cdmi_immediate_redundancy_billed": "2",
    "cdmi_infrastructure_redundancy_billed": "1",
    "cdmi_geographic_placement_billed": [
        "US" 
    ],
    "cdmi_encryption_billed": "AES_256_CTR" 
  },
  "childrenrange" : "1-3",
  "children" : [
    "Financials/",
    "CDMI_Spec.pdf",
    "hello.txt" 
  ]
}

There is quite a few metadata items here, and it looks quite complex, but it's not as bad as it first appears. To see what these metadata items mean, and how they are used to manage stored data, let's review them individually:

User Metadata Items

The directory (container in CDMI speak) has a single user metadata item:

      "user.DOSATTRIB": "0x20",

This user metadata item is an extended attribute that indicates the archive bit is set in the DOS mode of a directory. In this case, this user metadata item was created when the directory was created by SAMBA, and the storage server presents extended attributes on the filesystem as user metadata items.

Storage System Metadata Items

The directory has five storage system metadata items, and an ACL, which is an example of a more complex storage system metadata item:

        "cdmi_ctime" : "2009-12-29T12:43:32.479832Z",
        "cdmi_atime" : "2010-01-02T16:12:53.521983Z",
        "cdmi_mtime" : "2010-01-02T16:12:53.521983Z",
        "cdmi_acount" : "52",
        "cdmi_mcount" : "12",
        "ACL" : {
            "acetype" : "0x00",
            "identifier" : "jdoe",
            "aceflags" : "0x03",
            "acemask" : "0x000F005F",
            "acetime" : "2009-12-29T12:43:32.479832Z" 
        },

In this example, the first three metadata items contain the creation, last access and last modify times, respectively, and the remaining two show the number of accesses and the number of modifications since creation. The ACL metadata specifies the access control restrictions for the folder, and is based on NFSv4 ACLs.

Data System Metadata Items

Finally, we have the data system metadata items. These items are specified by a CDMI client, or an out-of-band management application, and specify how the data in the "Documents" folder should be managed:

"cdmi_data_redundancy": "2",
"cdmi_immediate_redundancy": "2",
"cdmi_infrastructure_redundancy": "2",
"cdmi_geographic_placement": [
    "US"
],
"cdmi_encryption": "AES_256_CBC",

The first data system metadata item, "cdmi_data_redundancy", indicates how many indpendent copies of the stored data should be kept. In this case, it has been set to two, which means that two copies of the data should be stored. Likewise, "cdmi_immediate_redundancy" indicates that two copies should be provided synchronously, "cdmi_infrastructure_redundancy" indicates that the two copies should be located in separate failure domains, "cdmi_geographic_placement" indicates that the copies should remain within the United States, and "cdmi_encryption" indicates that AES with a 256 bit key in counter mode should be used to protect the data.

It is important to note that data system metadata items expresses the desired management behaviour. This is separate from the actual management behaviour. In order to indicate to a client what management behaviours are actually being provided, CDMI provides a matching series of data system metadata items, ending with the suffix "_billed":

"cdmi_data_redundancy_billed": "2",
"cdmi_immediate_redundancy_billed": "2",
"cdmi_infrastructure_redundancy_billed": "1",
"cdmi_geographic_placement_billed": [
    "US"
],
"cdmi_encryption_billed": "AES_256_CTR",

In this case, the data system metadata is specifying that the storage system is able to meet the requested level of redundancy, the requested immediate redundancy, but is not able to provide the requested infrastructure redundancy, or the requested encryption method. This allows a client to discover if the requested data system services are being provided.

Data Management Summary

So, putting this all together, data system metadata is specified or inherited from parent containers. The data system metadata specified the desired data system services for stored content. The system tries to accomplish the requested data system services, and indicates what services are actually being provided in the corresponding billed data system metadata items.

All of the data system metadata items specified for the "Documents" container will also be applied to the child "Financials" container, unless overridden by a data system metadata value specified there. For example, if the "Financials" container has the following data system metadata:

"cdmi_data_redundancy": "3",

Only the redundency is changed — all other items are inherited from the parent. Thus, the billed metadata values would be:

"cdmi_data_redundancy_billed": "3",
"cdmi_immediate_redundancy_billed": "2",
"cdmi_infrastructure_redundancy_billed": "1",
"cdmi_geographic_placement_billed": [
    "US"
],
"cdmi_encryption_billed": "AES_256_CTR",

Currently all data system metadata items are inherited, and there is no way to override an inherited value except by specifying a new data system metadata item.

This concludes a quick overview of the basics of data management and metadata in CDMI. In the next part of the tutorial, we will discuss the various data system metadata services defined in CDMI.

CDMI 1.0 Draft Available

2010-02-12T13:36:00.001-08:00

All the hard work from the week long meeting of the members of the SNIA Cloud Storage Technical Work Group two weeks ago in San Jose have paid off, and we're proud to announce a final 1.0 draft of the Cloud Data Management Interface for cloud storage.

The specification can be downloaded from the Draft Technical Work for Public Review section of the SNIA web site, and is labelled CDMI 1.0g.

This release completes the remaining sections of the specification that were incomplete in the 0.9 draft, and clarifies a number of areas where the spec was unclear. Areas of improvement include:

Addition of Retention and Hold for compliance
Clarification of terminology from "Account" to "Domain"
Addition of Hash Data System Metadata
ACL syntax in JSON
Ability to perform cross-domain actions
CDMI Logging Queues
Elimination of JSON ordering constraints
Specification of Encryption Data System Metadata
Additional examples

We're very proud of how the specification has firmed up over the last few months, and it is very impressive that we've been able to go from forming a working group to a 1.0 draft in less than a year.

Opaque Clouds and Transparent Clouds

2010-02-02T16:53:00.001-08:00

There are two emerging usage models for cloud use that I am seeing emerging: Opaque Clouds and Transparent Clouds.

Opaque Clouds are clouds where users store pre-encrypted data to the cloud, such that the cloud operator has no visibility into the users' data. In this model, the encryption keys are owned and managed by the end user, and the cloud operator is not able to provide any value-added services that require access to the plaintext of the user's data.

Transparent Clouds are clouds where users submit data to the cloud (which may be encrypted during transmission and/or when stored), but the cloud operator is capable of having access to the users' data. In this model, the cloud operator either manages the encryption, or has access to the key repositories where the users' keys are stored. While transparent clouds can still be secure, there are additional security risks, as the cloud operator fundamentally must have access to the users' plaintext.

Both of these models have merits and use cases where they make sense. For example, if a first transparent cloud is using a second cloud to provide a second geographic location for data storage, the first cloud may store data into the second cloud in opaque fashion. A second scenario may be where a cloud user stores data to a cloud for the purposes of data sharing with another trusted user, but the cloud itself is not trusted. In this case, the keys would be shared between the two users, but the cloud would be unable to see the data stored.

Ultimately, I believe that opaque cloud storage will co-exist with transparent clouds, as both can operate concurrently with the same infrastructure. (translucent clouds?) As different use cases determine the security sensitivity of the data, and many of the cloud provided value added services (search, indexing, discovery, data mining, format conversion, etc) are quite compelling, it's going to be an interesting set of trade-offs between cloud security and cloud value.

Wrapping up CDMI

2010-01-25T10:28:00.000-08:00

If you're in the San Jose area this week, SNIA is holding their Winter Symposium, and we've got a week full of items we're working on to bring the proposed Cloud Data Management Interface (CDMI) to the point where it can be a finalized standard.

Here's our working agenda:

http://groups.google.com/group/snia-cloud/web/cloud-storage-winter-symposium-2010

Some areas of focus include:

XML support - Should only JSON be supported, or should XML be included in the standard?
Audit Queues - Access audit data via CDMI
Transaction Signing - Should S3-style transaction signing be part of the spec?
Named Forks - How best to handle files in NFS/CIFS with multiple data forks.
Retention - Finalizing what we take from XAM for retention data system metadata.

If anyone has specific thoughts on these topics and anything else related to CDMI, please don't hesitate to reply to this blog entry, or send me an e-mail. (dslik at bycast.com)

I'm also available on twitter as dslik, and I'll be watching for tweets about CDMI and sent to myself.

A CDMI Tutorial - Basic Input/Output

2009-12-22T10:42:00.000-08:00

With the new 0.9 draft release of the SNIA Cloud Data Management Interface specification now released, this is a good time to highlight the aspects of the standard that make it well suited for cloud storage, and to review the changes from the previous 0.8 draft version.

This post is the first in a series on CDMI. Subsequent posts will cover the following areas:

Basic Input/Output (This post)
Data Management, Part 1
Data Management, Part 2
Advanced Input/Output
Cloud-to-Cloud Interactions
Queues and Query
Authentication and Access Control
Billing and Accounting

Basic Input/Output

While supported, CDMI does not assume or require that clients interact with a storage system using the CDMI for storage and retrieval. CDMI can be used in conjunction with existing cloud protocols (such as Amazon's S3 API) and file system protocols (such as NFS, CIFS and WebDAV). The goal of CDMI is not to replace or even supplant these protocols — it is to provide a standardized access and management method that is independent from these protocols. So while storage and retrieval are important parts of the standard, they are only just one part of the standard. It is even possible to implement a fully complaint CDMI system that does not support the ability to store or retrieve data.

By not restricting to how data is stored or accessed, but still enabling a consistent and uniform way to access and manage stored data, CDMI enables a client to discover, access and manage stored content regardless of how it was originally stored, and regardless of the underlying storage implementation.

To get an idea of the basic object model by example, let's review basic storage and retrieval operations using CDMI:

Storing an Object

The most basic operations in CDMI are object storage and retrieval. Assuming that we have an authenticated session established to a CDMI cloud running at cloud.example.com and we want to store some text, here is the HTTP transaction that would be performed, as described in section 8.2 of the specification:

PUT /hello.txt HTTP/1.1
Host: cloud.example.com
Accept: application/vnd.org.snia.cdmi.dataobject+json
Content-Type: application/vnd.org.snia.cdmi.dataobject+json
X-CDMI-Specification-Version: 1.0
{
    "mimetype" : "text/plain",
    "value" : "Hello Cloud"
}

HTTP/1.1 201 Created
Content-Type: application/vnd.org.snia.cdmi.dataobject+json
X-CDMI-Specification-Version: 1.0
{
    "objectURI" : "/hello.txt",
    "objectID" : "AABwbQAQ810Mpei21pxfzA==",
    "parentURI" : "/",
    "accountURI" : "/cdmi_accounts/default_account/",
    "capabilitiesURI" : "/cdmi_capabilities/dataobject/",
    "completionStatus" : "Complete",
    "mimetype" : "text/plain",
    "metadata" : {
        "cdmi_size" : "11"
    }
}

Let's look at this transaction in more detail:

It's standard HTTP, and we specify that we will be submitting a request body in the form of a cdmi.dataobject, and requesting a response body in the form of a cdmi.dataobject. We submit in our request body JSON that includes a mimetype and value field, and receive in response a result (HTTP 201 Created) and a JSON structure that includes information about the newly created cloud object.

The response body JSON contains the following fields:

Field Name	Field Description
objectURI	The URI for the newly created object. This can be used to access the object via CDMI, and reflects the organization of the stored data. For example, if a storage system provides CDMI access to a NAS share, the objectURI reflects the file path.
objectID	Every object within a CDMI system has a globally unique identifier that can be used to access the object. Object IDs remain constant for the life of the object, even if modified, renamed or even moved to a different cloud provided by a different vendor.
parentURI	The URI for the parent of the created object. Objects inherit metadata from their parent.
accountURI	The URI for the account that the object was created under. Accounts determine the billing relationship, reporting visibility and basic access permissions.
capabilitiesURI	Every object within a CDMI system has "capabilities", which describe what operations the system is capable of performing on that URI. This allows clients to discover the capabilities of a CDMI storage provider. This URI allows a client to find out what the capabilities of a given object are.
completionStatus	This field indicates if an object has been fully created. When performing operations that take long periods of time, such as serialization/deserialization, copies and storing large objects, this field may indicate that the object is still in progress, and thus is not ready to be accessed.
mimetype	This field indicates the type of the value of an object, as specified at the time of creation.
metadata	This field includes user-specified metadata (not used in this example) and system generated metadata (one example of object size is shown in this example) related to the object. Metadata is a key part of the CDMI standard, and will be covered in more detail in later tutorials.

While this is the primary method to create objects in CDMI, there is an even simpler way:

PUT /hello2.txt HTTP/1.1
Host: cloud.example.com
Content-Type: text/plain
Content-Length: 11

Hello Cloud
HTTP/1.1 201 Created

This approach, described in section 8.3, uses completely standard HTTP to create a new CDMI object. The mimetype is specified through the Content-Type header, with the tradeoff that no CDMI-specific data is returned in response. The ability to support operations through 100% standard HTTP makes CDMI very easy to use from JavaScript and web environments, as does the use of JSON.

Listing Stored Object

As every data object has a parent, CDMI also allows the objects owned by a parent to be listed. This is performed by the following HTTP transaction, as described in section 9.4 of the specification:

GET / HTTP/1.1
Host: cloud.example.com
Content-Type: application/vnd.org.snia.cdmi.object+json
X-CDMI-Specification-Version: 1.0

HTTP/1.1 200 OK
Content-Type: application/vnd.org.snia.cdmi.container+json
X-CDMI-Specification-Version: 1.0
{
    "objectURI" : "/",
    "objectID" : "AABwbQAQI2MtLCwAVfYSFA==",
    "parentURI" : "/",
    "accountURI" : "/cdmi_accounts/default_account/",
    "capabilitiesURI" : "/cdmi_capabilities/container/",
    "percentageComplete" : "Complete",
    "metadata" : {
        
    },
    "childrenrange" : "1-2",
    "children" : [
        "hello.txt",
        "hello2.txt"
    ]
}

As a client does not know if a given URI is a data object or a container, it asks for a generic object. In this case, the URI, "/" is a container, so the cloud storage system returns with a response body of type cdmi.container. All common fields shared between data objects and containers have consistent meanings, but the two additional fields, "childrenrange" and "children" provide information about the objects contained by the container.

Like with the data object example above, there is a simpler way to get a list of the children of an object:

GET /?children HTTP/1.1
Host: cloud.example.com

HTTP/1.1 200 OK
Content-Type: text/json
{
    "children" : [
        "hello.txt",
        "hello2.txt"
    ]
}

This approach, as described in section 9.5, illustrates the ability to request specific fields to be returned in the JSON response body, and is easily integrated into AJAX-style javascript for web-based applications.

Retrieving Stored Object

Retrieving a stored object is as straightforward as listing the children of a container:

GET /hello.txt HTTP/1.1
Host: cloud.example.com
Content-Type: application/vnd.org.snia.cdmi.object+json
X-CDMI-Specification-Version: 1.0

HTTP/1.1 200 OK
Content-Type: application/vnd.org.snia.cdmi.dataobject+json
X-CDMI-Specification-Version: 1.0
{
    "objectURI" : "/hello.txt",
    "objectID" : "AABwbQAQ810Mpei21pxfzA==",
    "parentURI" : "/",
    "accountURI" : "/cdmi_accounts/default_account/",
    "capabilitiesURI" : "/cdmi_capabilities/dataobject/",
    "completionStatus" : "Complete",
    "mimetype" : "text/plain",
    "metadata" : {
        "cdmi_size" : "11"
    },
    "valuerange" : "0-11",
    "value" : "Hello Cloud"
}

Like the JSON body when originally creating the data object, the CDMI storage system returns the fields associated with the object. But unlike the original create, it also returns the value range and the value of the object. This is described in more detail in section 8.4.

If just the value is desired, it can be requested as either a JSON content-type, or as a standard HTTP (non-CDMI transaction):

GET /hello2.txt HTTP/1.1
Host: cloud.example.com

HTTP/1.1 200 OK
Content-Type: text/plain
Content-Length: 11

Hello Cloud

This allows CDMI clouds to be used to act as a standard web server, and enables the intriguing possibility that standard web servers could use CDMI as an upload, management and publishing protocol.

Summary

This provides a quick overview of the basics of writing, discovering and reading data in a CDMI cloud. In the next tutorial, we will review CDMI's data management functions, which rounds out the core of the proposed protocol.

A Reference Architecture for Cloud Storage

2009-12-09T14:45:00.000-08:00

Over the last twelve months, the SNIA Cloud Storage Technical Working Group has been busily defining an industry-wide standard for cloud data storage and management. This standard, the Cloud Data Management Interface (CDMI), is currently available for public review, with an updated draft scheduled for release in later in December.

The below diagram illustrates how the CDMI standard fits into emerging cloud ecosystems:

One of the most important aspects to understand about the CDMI standard is that it is not intended to replace existing cloud data access standards. This allows CDMI to be used with existing and new clouds that store data via non-CDMI protocols such as Amazon S3's API and traditional file system protocols such as NFS, CIFS, WebDAV.

CDMI builds on top of these existing data access paths to bring rich management semantics for data in the cloud, and facilitates emerging cloud use-cases including cloud peering, federation and differentiated services. In addition, CDMI provides standard mechanisms to enable the integration of clouds with other external systems (or clouds) for notification, workflow, audit, billing and authorization purposes.

A Reference Architecture for a CDMI-Native Cloud Storage

In order to illustrate how CDMI can enable to creation of next-generation cloud architectures, I've put together an example reference architecture for a CDMI-based cloud storage system, illustrating the different components and primary data flows that would be found in most cloud implementations that fully support CDMI.

The below diagram shows a logical representation of the major components of a cloud storage system build around the CDMI standard:

Looking at the diagram, we have the following major logical components:

Filesystem Clients and Protocol Gateways

CIFS, NFS, FTP, WebDAV, iSCSI, Fibre Channel, FCoE and other standard network file and block protocol clients can be attached to cloud storage via protocol gateways. These protocol gateways translate non-cloud storage protocols into CDMI cloud transactions, and are notified about changes to management metadata in the cloud such that it can export portions of the cloud via these non-cloud protocols.

Existing Cloud Clients and API Protocol Gateways

Existing cloud clients programmed to use cloud protocols such as Amazon's S3 HTTP API can communicate with a CDMI storage cloud through an API protocol gateway. An API protocol gateway translates these non-CDMI cloud storage protocols into CDMI cloud transactions. When a given non-CDMI cloud protocol supports features not directly supported by CDMI, the API gateway can either map these operations into CDMI operations, or use vendor extensions to CDMI to implement the required functionality.

CDMI Clients

Clients that implement the CDMI protocol natively can connect directly to the CDMI storage cloud without having to go through any protocol translation layers. These clients can access the full set of CDMI functionality, and can access and manage content stored by all clients, even if the content was not stored via CDMI.

Low Latency Object Stores

In almost all implementations, CDMI transactions will be routed to one or more low-latency object stores that are able to quickly satisfy storage and retrieval requests. Multiple low-latency object stores will often be connected together, peered, or federated, to allow for data dispersion, replication and other data management functions. Once safely stored, the cloud will be ready to interpret the data system metadata that determines the optimal placement characteristics desired by the cloud user.

High Latency Object Stores

Some vendors may choose to implement higher latency object stores, such as power-managed disk or tape storage tiers. Stored objects will typically be migrated to these stores from low-latency object stores based on the CDMI data system metadata.

Object Catalogue

In order to keep track of the locations where cloud objects are stored, and the namespace in which the objects are stored, many CDMI implementations will include an object catalogue. This catalogue keeps track of objects across all object stores (and external peered and federated clouds), and provides CDMI functions such as object notification, query and billing. Notification provides updates to internal cloud components and to external systems to enable additional cloud functions such as indexing, e-discovery, classification, object processing, format conversion, workflow and advanced analytics.

Data System Manager

In a CDMI system, the data system manager is responsible for interpreting the data system metadata specified for objects and containers through the CDMI interface, and performing operations to attempt to satisfy the requests for data dispersion, redundancy, geographic placement, performance, etc. Each time an object is created or modified, the data system manager will be notified, and if the constraints specified are not met, it can further replicate or migrate content across object stores or clouds. The SDSC iRODS (Integrated Rules-Oriented Data System) is an example of an existing Data System Manager that can play this role within or to federate clouds.

Putting it all Together

When you put all of these components together, you get a cloud that fully implements the main components of the CDMI specification, and provides rich cloud functionality. On top of this foundation, a cloud vendor can innovate and create additional value added services, such as integration with computing clouds, cloud virus scanning, QoS monitoring, and much, much more.

This degree of flexibility demonstrates much of the value that CDMI has to offer the industry, both to cloud users and vendors, and it will be exciting to see these sorts of interoperable architectures emerge over time as CDMI becomes adopted and the cloud marketplace matures.

How low can disk go?

2009-11-26T17:41:00.000-08:00

The capital acquisition cost of disk-based archiving solutions (in cost per terabyte) has dramatically fallen over the last five years. Unfortunately, the rate of reduction in cost is slowing as the cost approaches the raw cost of the disks included with the storage system.

The four major factors that have driven the reduction of the cost of disk-based archiving are as follows:

Increasing disk capacity (density) for a hard drive of a given price
Transition from the use of enterprise disks to the use of consumer grade SATA disks
Transition from storage interconnection via Fibre Channel to switched SAS and Ethernet
Transition from customer storage controllers to commodity servers and software

When analysing the costs of an disk archive, much of these savings come from reducing the cost of the non-disk components. In a large disk archival systems, the system price divided by the number of disks is now approaching the over-the-shelf price of a consumer disk (around $100 per TB). This is an important, because it indicates that most of the cost gains (from factors 2 through 4) are unsustainable in the long term. The closer the cost of the system approaches the cost of the raw storage, the less cost reductions can be achieved.

As a consequence, the rapid decrease in the cost of disk-based archiving is not a result of the intrinsic reduction in the cost of disk storage, but rather is a reduction in the cost of the overall system. And now that this reduction in cost has already largely occurred, the rate of cost reduction flattens out to more closely approximate the reduction in the cost resulting from increasing disk density.

This is a critical point to understand when comparing the costs of disk archiving to tape archiving, since many cost projections have made the assumption that this rate of reduction of disk cost will continue into the future.

The Demise of Tape is Overrated

2009-11-23T20:18:00.000-08:00

Christopher Poelker, in a blog post on ComputerWorld titled, Is Tape Really Dead, makes a series of assertions about the superiority of disk over tape. Now to give credit where it is due, he is talking about tape's role in backup, and does conclude that tape still has a role to play. Unfortunately, many of his statements made along the way are simply inaccurate.

Chris states:

Everyone is aware of the limitations of tape solutions.

Tape Disk
Sequential access Random access
Relatively slow Fast
Shipped offsite Electronically vaulted
Once a day process Periodic or continuous
High operational touch Automated
No-dedupe Dedupe
Inexpensive media More expensive

Let's look at each of these in turn:

Sequential Access vs. Random Access
What Chris is getting at here is seek latency. Both tape and disk provide full random access, only disk is faster at it than tape. However, as hard disk capacities have increased while access bandwidth remains largely constant, from a software architecture standpoint, disks look more and more like tape. This is what is leading to the collapse of RAID 5 as a means to protect data, and, in my opinion, is what will ultimately lead to the death of disk. But more on this in a subsequent blog post.

Relatively slow vs. Fast
Tape systems, when properly used, can provide extremely high levels of throughput performance, into the 10 Gigabit/sec ranges.

Individual tape drives can already stream sequentially accessed data faster than most hard disks (120 MBytes/sec), and LTO5 will increase this lead further. When randomly accessing data spread across a tape or disk, the disk will outperform tape due to lower seek latencies. And, of course, if seek latencies are important to you, you should be looking at flash.

Shipped offsite vs. Electronically vaulted
Disk drives are far more vulnerable to damage than tapes, and simply don't have the flexibility to be able to be shipped around the same way. "Electronic vaulting" often equals expensive WAN data transfers and higher costs for power and equipment.

Once a day process vs. Periodic or continuous
This would be true if we're talking about a pure tape solution, but tape-based systems have been deployed along side disk in a storage hierarchy for decades.

High operational touch vs. Automated
Baby-sitting ten thousand disks isn't low operational touch either. Disks fail continuously, and the wrong swaps can destroy an entire RAID set. Many large archives run with tens of thousands to hundreds of thousands of tapes with very little operator intervention, and modern automated libraries are highly reliable and fault tolerant.

No dedupe vs. dedupe
This is false, as dedupe is yet another data compression technique, and applies equally well to tape as it does to disk. Again, the use of dedupe on tape in backup and archiving systems goes back decades.

Inexpensive media vs. More Expensive
We're thrown a bone here in the cost department, but what isn't considered is in addition to the consumables (disks and tapes), the disk subsystems themselves must be replaced on a far more frequent basis than tape libraries. With tape libraries, the drives can be swapped out and the tapes migrated to newer, higher capacity media without having to replace the entire library.

This also does not take into account the far higher opex costs of power and heat required for disk-based solutions.

Conclusions
Despite the near continuous siren call of the "Tape is Dead" crowd, tape provides significant value, often higher value for the dollar than disk, and has a long life before it. And, in many ways, it is spinning disk that should be more worried about its life in the coming decade.

The Roles of CDMI

2009-10-22T10:27:00.000-07:00

The proposed SNIA Cloud Data Management Interface standard (CDMI) is intended to address a wide variety of different use cases, as described in the draft SNIA Cloud Storage Use Cases document.

Specifically, CDMI has the several distinct and overlapping design goals:

1. To act as a cloud management protocol for other non-cloud data access protocols, without providing data access.

The use case for this mode of use is, for example, the management of a SAN or NAS fabric allowing the provisioning and specification of data system metadata for opaque LUNs which can be dynamically provisioned programatically, for example, in conjunction with OCCI in a cloud computing environment. In this case, there is no data access via CDMI, only management and accounting access.

2. To act as an cloud management protocol and as a secondary cloud data access protocol to existing cloud and unified storage systems.

The use case for this mode of use is, for example, is to provide consistent management access to existing unified storage systems that provide block, file and object protocols. For example, a Amazon EC2 instance could be run that exposes an S3 bucket through CDMI, manages Elastic Block Storage LUNs, and implements some of the data system metadata functionality.

3. To act as a primary cloud management and cloud data access protocol for next-generation cloud storage systems

The use case for this mode of use, for example, is to enable a superset of cloud data access, manipulation and management functions, and to enable advanced scenarios such as distributed application systems build around cloud storage, cloud federation, peering and delegation. For example, a cloud could provide CDMI access to objects for cloud applications, all while manage file-system views into the object space from remote edge NAS gateways, all while federating together existing enterprise storage and public and private clouds.

Queues in a Cloud

2009-10-19T19:42:00.000-07:00

Queues are extremely powerful approaches to data storage and data exchange that are commonly used in application programming, especially when components of a program are executed in parallel. By supporting queues, cloud storage systems provide safe and efficient mechanisms by which programs can persist their state, communicate between components, and interact with other programs and systems.

What is a Queue

A Queue is an object with zero or more data "records", such that only the oldest item is accessed at any given time. Once the oldest item is removed, the second-oldest item is accessed, and so forth until the queue is empty. This provides what is known as "first-in, first-out" access to data.

In a standard object, when you update data, you update the only record in the object. But in a queue, this creates a new record. When you read the value of an object, you get the only record in the object, but in a queue, this returns the value of the oldest record. And when you delete the value of an object, you delete the object, but in a queue, you delete only the oldest record. Thus, unlike basic objects, updating and deleting records from a queue are no longer idempotent operations.

How are Queues Typically Used

Queues are typically used in two different situations:

Internal Work List - Inside a program, the program will store a queue of data that needs to be processed in a given order, such that as resources become available, the items from the queue can be accessed in order. In this use, the queue is being used to store state.

Inter-Process Communication - Between two programs, a queue will contain a list of items that need to be communicated from one program to another. As the sender program encounters data that needs to be sent to the receiver program, it enqueues the items into the queue, and as the receiver program is able to process data from the sender program, it dequeues items from the queue. In this use, the queue is being used to exchange state.

As a quick aside, TCP/IP is an example of a queue used for information exchange between two systems. When you write data to a TCP/IP connection, you are enqueueing the data for delivery, and when the destination application reads from the TCP/IP connection, it is dequeuing the data from the network abstraction.

These uses are best illustrated through an example. Let's say that we are designing a book scanning system that runs in the cloud. We have a series of image scanners that digitize pages of books, and we have an OCR program that converts the images into text that can be indexed for search.

A simple implementation would be to scan a page, OCR it, then move on to the next page, but that doesn't meet all of the criteria of a cloud solution, as we can't scale it. A good solution would be able to handle multiple scanners, and have multiple instances of the OCR process running in parallel. And that calls for queues.

Handling Multiple Writers

A queue can be used to aggregate data values from multiple writers into a single ordered set of values. This behaviour is perfect for logging, job aggregation and any other situation where data originating from multiple entities needs to be consumed by a single entity.

For example, in our book scanning example, a typical scanning workflow may have tens to hundreds of scanners running concurrently, where each scanned page needs to be run through an OCR (Optical Character Recognition) process in order to index the contents of each scanned page.

A cloud application built around queues is easily scaled by running multiple parallel instances of the scanning application, with each of the instances using a common cloud for storage. If all of these instances store scanned pages into the same queue, the interface to the OCR process (the queue) is the same regardless of the number of writers.

The logic for the scanning process would look something like this:

1. Scan Page
2. Write image as object into cloud
3. Enqueue image object ID into cloud
4. Repeat

The logic for the OCR process would look something like this:

1. Read from cloud queue to get the object ID of the next image to process
2. Read the image from cloud using the object ID
3. Perform OCR processing
4. Add OCR text to Index
5. Delete item from cloud queue
6. Repeat

Now, we can arbitrarily scale the number of scanning processes. But what if our OCR takes too long to handle the combined workload of all of these scanning processes?

Handling Multiple Readers

If our OCR processing takes far longer than the time required to scan a page, we need to be able to increase the performance of the OCR processing. And when we add multiple parallel page scanners, things get even worse. We could try to make a faster OCR engine, but it's far easier to be able to scale out the OCR processing by running multiple instances of the OCR processing in parallel.

In this case, we need to be able to have multiple readers of the queue. And, we need to ensure the following characteristics of our solution:

1. No two queue readers will get the same item
2. No items will be lost, even in the event of a queue reader failure

If you just run two queue readers in parallel, both of these situations can occur. If the two readers run lock-step, they will get the same item. And if they both delete, the second deleted item will be lost.

Thus, we need to introduce another concept in order to maintain these characteristics — the ability to atomically transfer an item from one queue to another. With this capability, our OCR process can be modified to ensure that even with multiple readers, no data is lost or processed twice:

1. Transfer item from cloud queue to worker queue
2. Read from worker queue to get the object ID of the next image to process
3. Read the image from cloud using the object ID
4. Perform OCR processing
5. Add OCR text to Index
6. Delete item from worker queue
7. Repeat

Using this approach, you can arbitrarily scale the number of reader processes without modification. And these workers can enqueue their results into a common queue, allowing the results to be recombined for further processing.

By leveraging queues, cloud storage allows the creation of complex reliable workflows that can be scaled arbitrarily and dynamically. It also facilitates the creation of loosely coupled reliable systems of interacting programs that work together to solve a given problem in a flexible and scalable manner.

The Worst Case Scenario

2009-09-30T18:05:00.000-07:00

What do you do when the unthinkable happens, and you lose your entire primary site of operations, including your local cloud storage? It could be a flood, a hurricane, or something as commonplace as a building fire, but it's happened — and now all of your infrastructure and storage media associated with your archive is gone.

But life continues on, and so does your business. Assuming your archive is a critical part of your workflow and corporate assets, how can you protect against these major disruptions?

To the right is a simplified flowchart to help determine when you should protect against site-loss scenarios, and what options are available to prevent loss of archived data and ensure continued access.

Click on the image to enlarge.

Single-Site Archives

If all of the equipment and storage associated with your archive is located at a single site, and that site is lost, unless you have some form of off-site storage, all data stored in the archive will be lost.

While a multi-site archive is the best solution to avoid data loss in this scenario, for cost-sensitive data where rapid restoration of data access and operations is not required, a multi-site archive may be more expensive than is warranted. In this case, two common options are to create two tape copies and have them vaulted off-site, or to store a copy of the data into a public cloud provider, such as Amazon, Iron Mountain, or Diomede Storage.

Vaulting to Tape

Storing to tape, and vaulting the tapes off-site is the least expensive option for protecting archived data. In this scenario, a node would be added to the single-site archive that creates two identical tapes containing recently archived data. These tapes are then sent off-site.

The ability for an archive to be restored from the "bare metal", from only the data objects being archived, is a very important feature of an archival system. This ensures that even if the control databases are lost, the archived data can still be accessed, and the archive can be rebuilt.

When planning a tape vaulting approach, the frequency that these tapes are created determines how much data is at risk of loss in the event of the loss of the site. For example, if tapes are generated every week, and take two business days worst case to be taken off-site, then the business can have an exposure window of up to twelve calendar days.

In the event of the catastrophic loss of the primary site, these tapes would have to be recalled from the vaulting provider, which can take some time, and hardware would have to be re-acquired to rebuild the archive. Don't underestimate the amount of time required to re-order hardware. Often the original equipment is no longer available, so a new archive will need to be specified and ordered, and can take weeks for the servers to be assembled and shipped.

Once the tapes have arrived and the hardware has been set up, the archive is rebuilt from the data stored on the tapes, and once the last tape is processed, the archive is ready to be used. This is known as a "bare-metal restore".

Of course, depending on the size of the archive, this could take a very long time. An 1 PB media archive would take 115 days to restore when running at a restore load of 800 Mbits/s, and a 10 billion object e-mail archive would take 115 days to restore when running at a restore load of 1000 objects per second. Rebuild times must be taken into account when planing for archive restoration, and often the cost of downtime associated such a restore is high enough that cloud or multi-site options are considered instead.

Storing to the Cloud (Hybrid Cloud)

Another option for single-site archives is to store a copy of the archived data to a cloud storage provider. This eliminates the headaches associated with tape management, but introduces the requirement for network connectivity to the provider. In this scenario, for each archived object to be protected off-site, the object is also stored to a cloud provider, which retains the data in the event that a restore is needed.

Unlike with tape archiving, data is stored immediately to the cloud, limited only by WAN bandwidth. However, this limitation can be substantial, and when bandwidth is insufficient, data will be at risk until the backlog clears. If data is being stored to the archive at 100 Mbits/s, an OC-3 class Internet connection would be required, which can be far more expensive than sending twenty tapes out each week.

In the event of the catastrophic loss of the primary site, hardware would be re-acquired to rebuild the archive, and network connectivity would need to be acquired to allow connectivity to the cloud. When both of these are operational, the archive would be reconnected to the cloud. This would restore access to the archived data, albeit limited by the network bandwidth. Over time, the on-site archived data can be then restored back over the WAN.

The primary disadvantages of this approach are the time required to get the hardware and network access for restoring the on-site component of the archive, with the second disadvantage being cost. Fears about unauthorized disclosure of data and loss of control over data are also common, though they can be mitigated with the appropriate use of encryption.

And often, for less than the price charged by most public cloud providers, one can afford to create a multi-site archive, either across multiple premises owned by the business, or into a second premise hosted by a third party.

Why not just use the Cloud?

Some public cloud providers encourage an architecture that has a minimal on-site presence, and stores all data off-site in the cloud. For some scenarios, this approach works very well, as it minimizes capital costs and minimizes the time required restore hardware and access in the event of a disaster. However, one must have sufficient WAN bandwidth for the expected store and retrieve loads (as opposed to just store-only traffic loads when using the cloud as a storage target), and in the event of a network connectivity failure, access to most or all of the archive can be disrupted.

This is contrasted with the hybrid cloud model, where the private cloud on-site allows continued access to the data even during WAN failures, and the public cloud is used as a low-cost data storage target.

Multi-Site Archives

When continuance of business operations are important, or archival data must be accessible across multiple sites, the archive can be extended to span multiple sites. This involves several considerations, including:

What data is created in one site and accessed in another?
What data should be replicated to other sites for protection?
What data should be replicated to other sites for performance?
In the event of a site loss scenario, what will be the additional load placed on other sites?

Such multi-site archives are very flexible, and allow seamless continuance of operations even in the event of major and catastrophic failures that affect one or more sites. While this is obviously the best solution from a business continunace standpoint, it is also the most expensive, as you must duplicate your entire archive infrastructure in multiple sites, and provide sufficient WAN bandwidth for cross-site data replication.

Of course, one can also deploy systems like Bycast's StorageGRID to provide a mixture of the above described approaches, using policies to determine which archived content is stored locally, vaulted to tape, stored in a public cloud, and replicated across multiple sites. This flexibility allows the value to the data to be mapped to the cost of the storage, and leverages a common infrastructure for all levels of protection required.

Introducing CDMI

2009-09-14T13:42:00.000-07:00

Today, the Storage Networking Industry Association (SNIA) publicly released the first draft of the Cloud Data Management Interface (CDMI). The draft standard can be downloaded at the below address:

http://www.snia.org/tech_activities/publicreview

I'm very pleased to have been a significant contributor to this standard since the inception of the working group earlier this year. Over the last nine months, we've been able to come a long way towards defining a working standard for cloud storage management, and Bycast is proud to have contributed many best-of-breed capabilities first pioneered in Bycast's StorageGRID HTTP API, used by hundreds of customers worldwide to store and access many dozens of petabytes of data in cloud environments, both public and private.

Why CDMI?

CDMI provides a standardized method by which data objects and metadata can be stored, accessed and managed within a cloud environment. It is intended to provide a consistent method for access by applications and end-users' systems, and provide a consistent interface for providers of cloud storage.

Currently, almost all of the cloud storage providers and vendors use significantly different APIs, which forces cloud application and gateway software vendors to code and test against different APIs, and having to architect their application around the lowest common denominator. CDMI significantly reduces the complexity of development, test and integration for the application vendor, and is specifically designed to be easy to adopt for both cloud providers and application vendors. CDMI can run along side existing cloud protocols, and, as an example, a customer could run a CDMI gateway in an EC2 instance to gain access to their existing Amazon S3 bucket without Amazon having to do any work — a great example of the power of cloud!

Much like SCSI, FiberChannel and TCP/IP, such industry-wide standards provide many advantages. These range from simple but essential efficiencies, such as standardized interface documentation, conformance and performance testing tools, the creation of a market for value-added tools such as protocol analyzers and developer awareness, libraries and code examples.

Industry standards also jump-start the network effect, where more applications encourage providers to support the standard, and more providers supporting the standard encourage application vendors to support the standard. Finally, and most excitingly, CDMI increases inter-cloud interoperability, and is a fundamental enabler for advanced emerging cloud models such as federation, peering and delegation, and the emergence of specialized clouds for content delivery, processing and preservation.

A Whirlwind Tour of CDMI

CDMI stores objects (data and metadata) in named containers that group the objects together. All stored objects are accessed by a web addresses that either contain a path (eg: http://cloud.example.com/myfiles/cdmi.txt) or an object identifier (eg: http://cloud.example.com/objectid/AABwbQAQvmAJSJWUHU3awAAA==).

CDMI provides a series of RESTful HTTP operations that can be used to access and manipulate a cloud storage system. PUT is used to create and update objects, GET is used to retrieve objects, HEAD is used to retrieve metadata about objects, and DELETE is used to remove objects.

Data stored in CDMI can be referenced between clouds (where one cloud points to another), copied and moved between clouds, and can be serialized into an export format that can be used to facilitate cloud-to-cloud transfers and customer bulk data transfers. All data-metadata relationships are preserved, and standard metadata is defined to allow a client to specify how the cloud storage system should manage the data. Examples of this "Data System Metadata" include the acceptable levels of latency and the degree of protection through replication.

In addition to basic objects and containers (similar to file and folder from a file system), CDMI also supports the concept of capabilities, which allow a client to discover what a cloud storage system is capable of doing. CDMI also supports accounts, which provide control and statistics over account security, usage and billing. Finally, CDMI supports queue data storage objects, which enable many exciting new possibilities for cloud storage.

In fact, queues are important and significant enough that I'll be writing more about them and what they enable in a subsequent blog entry.

The Next Steps

With CDMI now "out in the wild", this is the point where the standards effort starts to get really interesting. Up to this point, it has been a relatively small group that has been working on the standard, and we've had to make some controversial decisions (such as eliminating locking and versioning from the first release). There's still a lot of work to be done, and as CDMI gets more visibility, we look forward to increased involvement from other players in the industry. Together, we can make this standard even better, and help shape the future of cloud storage.

So, if you are interested in cloud storage and cloud storage APIs, I would strongly encourage you to take the time to read the CDMI draft documentation, and contribute your thoughts and suggestions.

We're proud of what we've achieved, and together, we can make it even better.

Cloud Computing and Cloud Storage Standards

2009-09-10T09:22:00.000-07:00

All of our work at the Cloud Storage technical working group in the Storage Networking Industry Association (SNIA) has been coming together, and we are nearing a public release of the Cloud Data Management Interface (CDMI).

We've also been working with the Open Grid Focum (OGF) on making it such that CDMI can be used in conjunction with the OCCI standard to manage data storage in cloud computing environments, and there are some exciting possibilities of combining CDMI with the VMWare vCloud standard that was recently donated as a potential industry standard to the DMTF.

You can read our joint whitepaper on CDMI and OCCI at the below URI:

http://ogf.org/Resources/documents/CloudStorageForCloudComputing.pdf

If any readers have any questions about CDMI and how it facilitates cloud computing, please don't hesitate to comment!

Cloud Computing needs Cloud Storage

2009-08-28T09:09:00.000-07:00

I recently spoke on a panel at the Cloud User 2009, held in San Diego, on the subject of "Navigating the Cloud Vendor Community to Achieve Sped of Migration and Ease of Use". We were originally scheduled for an hour of discussion, but due to a subsequent presenter not showing up, and great interest from the audience, we went a half an hour over schedule.

My presentation was focused on the need for all IT infrastructure to be capable of supporting cloud computing. Often, people organizations focus on one or two aspects of cloud computing (often virtualization and elastic computing), while not always taking into account the impact, requirements and dependencies on other areas of IT.

I started out by reviewing a mapping of the U.S. Government's Service Component Reference Model into the Cloud model. This diagram provides a good example of all the different areas that IT and business organizations need to consider when embarking on a cloud project. This diagram is quite effective to discuss interdependencies between different technologies and IT areas of expertise.

Cloud Storage as a co-requisite for Cloud Computing

After introducing the concept of cloud as a holistic IT practice, I spent some time focusing on the specific dependencies that cloud computing has on storage, and how cloud computing drives the need for cloud storage.

	Cloud Computing Drivers	Emerging Storage Needs
Self-contained "Packages"	Virtual Appliances, vApps	VM Image Mgnt & Object Stores
Location Independence	Hybrid Clouds, vMotion	Distributed & Multi-site Storage
Loosely Coupled	Dynamic Provisioning	Simpler & Less Fragile Interfaces
Scale-Free	Elastic Scaling, Billing on Usage	Tiering & Dynamic Placement
Shared	Co-Hosting	Multi-Tenancy & Storage Security

The above table maps many of the business advantages and approaches for cloud computing to new requirements for storage that emerge as a result. Let's review these in detail:

Self-contained Packages

As the environment and resources for a given application or computing problem is packaged up and managed at the VM and application level, new requirements are created around managing these images and associating data with VM sessions and applications to allow them to migrate together, be snapshotted together, managed together, etc.

Thus, when you move to cloud computing, you need a new way to package up the data along with the applications, and ensure that they are self-contained.

Location Independence

These packages and data then must be able to migrate, both within the enterprise (such in DR and Business Continuity applications), and between organizations, when utilizing hybrid and public clouds. Once the stored data is packaged, this becomes easier, but often data movement must be more granular, as some data may need to remain within the organization, or may need to be spread across multiple clouds.

Thus, when you move to cloud computing, you need to make your data accessible from multiple locations, and ensure that it is consistent, complete and correct.

Loosely Coupled

One of the benefits of cloud computing is loose coupling between systems. This allows simple reconfiguration, enables the mixing of applications and infrastructure to quickly create new applications and update existing applications. The resulting collections of services are dynamically provisioned, and often do not involve people. In order to accomplish this, you need to ensure that all of the parts fit together, and can be controlled to dynamically assemble systems programatically.

Thus, when you move to cloud computing, you need to have simpler and less fragile interfaces to allow storage to be dynamically connected up to storage, as needed, when needed.

Scale-Free

In order to quickly scale up and down cloud computing environments, one needs to be able to deploy applications and storage in a scale-free manner. Being able to dynamically create a thousand-node compute cluster is not of much use if there is not also storage infrastructure that can scale to support this cluster.

Thus, when you move to cloud computing, you need to ensure that your storage is also capable of scaling elastically, and is capable of tiering data so that it is available at the right cost and performance. Often, one of the first problems one runs into when deploying an elastic computing infrastructure is mismatches between computing and storage, and the cost of keeping all the data resulting from the cloud computing activities.

This is important: Cloud computing results in an explosion of data, and this data has to be tiered in order to stay economical.

Shared

While less of an issue with private clouds, multi-tenancy and support for multiple users on shared infrastructure is critical for leveraging many of the economic advantages resulting from resource pooling. However, this sharing brings requirements for partitioning and security to prevent unauthorized disclosure of data. When storage is shared, there must be strong assurances that information will continue to be protected.

Thus, when you move to cloud computing, especially in public and hybrid clouds, the placement and security of stored information must be carefully assessed to ensure that additional risks are not introduced.

More Industry Alignment on Object Storage

2009-08-28T08:40:00.000-07:00

In a discussion on one of EMC's blog entries, The Future Doesn't have a File System, Paul Carpentier, of Centera fame, reiterated the need for an industry-wide, lightweight web-based standard for object-based storage access.

His initial thinking was as follows:

1. Unique identifiers; 128 bit, hex representation proposed
2. Object = immutable [Content + Metadata]; content is free format, metadata is free format, XML recommended
3. Simple access protocol; HTTP proposed; non-normative client libraries optional
4. READ and READ METADATA operation; (READ gets metadata and content)
5. WRITE and DELETE operation
6. Small set of standardized XML policy metadata constructs re service level, compliance, life cycle; TBD
7. Persisted Distributed Hash Table to allow variable identifier mapping; 128 bit to 128 bit; HTTP accessed

What is interesting is the degree to which this proposal is aligned with the work being done by the Storage Networking Industry Association in it's Cloud Storage Technical Working Group. This working group is creating a new standard call the Cloud Data Management Interface, which is intended to provide a standardized method for access and management of cloud data using a light-weight RESTful access method.

While the draft standard is not quite released to the public, let's take a quick peek at how it compares to Mr. Carpentier thoughts:

1. Unique identifiers; 128 bit, hex representation proposed

SNIA is proposing to use XAM XUIDs for identifiers, which allows vendors to innovate and define how their identifiers are comprised, while still ensuring global uniqueness and the ability for any object ID to be managed by any vendor's system.

While a basic 128-bit identifier, such as a UUID, is simpler, it does not provide strong guarantees that it will be unique across cloud vendors, and this is critical for emerging cloud models such as cloud migration, federation, peering and interchange.

2. Object = immutable [Content + Metadata]; content is free format, metadata is free format, XML recommended

While some vendors (such as Bycast) will implement the proposed standard by using immutable objects, the standard includes the optional ability to modify both object content and metadata for existing objects, without changing the object identifier.

Metadata will include both user-generated items and system-generated items, and will be represented using XML or JSON.

3. Simple access protocol; HTTP proposed; non-normative client libraries optional

SNIA is using RESTful principles and the HTTP protocol as a foundation for the standard, and simplicity is a key design goal. Almost every part of the standard is optional, and the client can discover what parts of the standard are supported by any given implementation.

Client libraries to provide simplified language mapping is anticipated, but the goal is to enable full use using standard HTTP libraries.

4. READ and READ METADATA operation; (READ gets metadata and content)

The HTTP GET and HEAD operation map to these functions.

5. WRITE and DELETE operation

The HTTP PUT and DELETE operations map to these functions.

If a cloud does not support mutable objects, then the cloud storage provider can indicate this to a client via the capabilities discovery interface, and any attempts to modify an existing object would fail.

6. Small set of standardized XML policy metadata constructs re service level, compliance, life cycle; TBD

SNIA is actively working on standardizing a set of "Data System Metadata", which allows a client to specify what level of service that it desires from a cloud. Examples include maximum latency, degree of replication, etc.

7. Persisted Distributed Hash Table to allow variable identifier mapping; 128 bit to 128 bit; HTTP accessed

This is outside of what the standard is proposing, but by using the included queue data object functionality, vendors can add functionality such as lookups and transformations. This allows extension by vendors in a standardized way, and allows them to take advantage of much of the common infrastructure provided by the standard.

In summary, I would encourage everyone who is interested in cloud storage or in the industry to take a look at the work that the SNIA is doing, and to get involved!

Dictionary of a Cloud Standard

2009-08-04T09:42:00.000-07:00

Any cloud storage standard requires the consideration of many interrelated functional areas. Below is a comprehensive list of all the areas that must be considered in the standardization process, and any standard must choose the scope and degree to which these items are addressed.

As all of these subjects are inter-related, they are listed below in alphabetical order:

Accounts – A grouping of stored objects for the purposes of administrative control and billing purposes. Each object is owned by one or more account, and is billed to those accounts. Accounts may have sub-accounts, where content owned by a sub-account is rolled up to a higher-level account.

Accounts (Provisioning) – The interface by which administrative clients can create new accounts, modify account characteristics or remove accounts.

Audit – Detailed records of accesses and state changes to the storage system used for troubleshooting, forensic analysis and security analysis. Audit information should be accessible based on partition, account, client, audit types and other filters, ideally through a cloud API.

Client Authentication – The method by which the credentials presented by a client are verified and mapped to local identities used to determine permissions.

Client Authentication Delegation – The method of verifying and mapping credentials, where the processing is delegated to an external system or alternate cloud. (eg, AD, LDAP)

Client Interface Protocols – The method by which clients are able to access data stored in the cloud. Interface protocols are typically specific to how the client interacts with the data. For example, a file-system client would expect an interface protocol such as NFS or CIFS, where a database client would expect a SQL interface. Clients that interact with objects directly may use protocols such as XAM or RESTful HTTP object access.

Cloud Partitioning – The ability to take a single cloud and create multiple partitions that act as completely independent clouds while still using the same common cloud infrastructure.

Cloud Peering – The ability for one cloud to transparently reference and store objects in another cloud such that a client can transparently access content directly from either cloud.

Cloud Federation – The ability for one cloud to transparently reference and store objects in another cloud such that a client can transparently access content from the primary cloud.

Event Feeds – A client-accessible queue of events that occur against a subset of content (as defined by a query) that can be used for state synchronization between clouds, billing and other inter-system communication.

Introspection – The ability for a client to discover what services a given cloud is capable of performing, and what subset of these capabilities the client is allowed to use.

Metadata (User) – Arbitrary client-named key/value pairs and tag lists specified by a client that are stored along with an object.

Metadata (Data System) – Standardized named key/value pairs and tag lists specified by a client that indicate to the cloud how the object or objects should be managed. Examples include ACLs, encryption levels, degrees of replication or dispersion, and QoS goals.

Metadata (System) – Standardized named key/value pairs and tag lists generated by the cloud that indicate to the client properties of the object. examples include last access time, actual degree of replication (as opposed to requested degree of replication), and lock status.

Namespace (Local) – Each client or set of clients may see a different subset of the global namespace, or a client-specific namespace. Objects within the cloud may reside in one or more namespaces. By sharing objects across namespaces, different faceted views into the cloud can be created, and use cases such as sharing can be enabled.

Namespace (Global) – An administrator or suitably configured client may see all objects in the cloud within one namespace.

Object (Composite) – The ability for an object to contain other objects such that the collection of objects can be managed and stored together as a single object.

Object Identifiers – Each object stored within the cloud needs a globally unique identifier that stays with the object across its entire lifecycle. Ideally, these identifiers are unique across clouds, and are preserved across clouds, which enables federation, transparent migration and peering.

Object Locking – Clients may wish to be able to lock an object to prevent modification or access by other clients.

Object Reference Counting – Clients may wish to gain a form of a lock on an object that ensures that the object will remain in the cloud. Only when all of these references are released can the object be considered for removal.

Object Versioning – Clients may wish to have historical versions of objects be retained when an object is modified.

Object Permissions – Each object stored has a list of which entity is permitted to perform which action. This is typically called an Access Control List (ACL). These access controls specify basic and administrative operations can be performed.

Object Referencing – The ability to specify that a given entity within a namespace is an object in an alternate location within the namespace, in another namespace, or in another cloud, while allowing transparent client access (see cloud peering and federation).

Object Serialization – The ability for a client to take one or more objects and transform them into a single bitstream that can be used for inter-system interchange.

Object Snapshots – Clients may wish to be able to create snapshots of an object or set of objects, such that the state of the objects at that point in time can be accessed via an alternate location within a namespace.

Query – The ability to submit a series of criteria (for object content and/or metadata) and have returned (possibly as another static object) the list of objects that match the specified criteria.

Query (Persistent) – The ability to create queries that run continuously and dynamically update their results over time as the state of the cloud changes.

Usage Statistics (Client) – The ability to obtain information about how many operations have been performed by a given client over a given timeframe is required for accounting, billing and reporting purposes.

Usage Statistics (Object) – The ability to obtain information about how many operations have been performed against a given object or set of objects, and the operations that have been required to manage a given object or set of objects over a given timeframe for accounting, billing and reporting purposes.

These are all aspects that we have considered here at Bycast, and that we are contributing as part of our involvement with the SNIA Cloud Storage Technical Working Group.

Standardizing Cloud Storage

2009-06-30T16:11:00.000-07:00

As the cloud storage concept matures and an increasing number of service and technology providers emerge in the market, there is a growing recognition of the need to standardize protocols for data access and storage management functions in a cloud storage environment.

The advantages of an open standard for cloud storage include:

Allowing a cloud storage client to interoperate with multiple providers
Enabling data portability between cloud providers
Facilitating common documentation, sample code and educational material
Allowing common test infrastructure and conformance testing
Reducing development work for cloud clients and providers
Reducing the complexity of standardized access libraries
Encouraging the creation of debugging tools for diagnostics, profiling and interaction analysis

With cloud computing and cloud storage being such a hot topic at this time, there are multiple initiatives underway attempting to standardize various components and interfaces within the cloud storage stack. And one of the leading initiatives is the cloud storage technical working group within the Storage Networking Industry Association.

A Cloud Storage Reference Model

In the last six months, we've been working on several deliverables, with the main work product being the creation of a standard reference model for the management and access to cloud storage resources.

In summary, here are the highlights of the working group's vision of cloud storage:

An HTTP API

Management functions and data access are provided via a light-weight HTTP RESTful API. Object, block and database storage APIs co-exist with this core API, facilitating emerging cloud use cases and allowing continued innovation as new applications are moved into the cloud.

The HTTP API also facilitates discovery and introspection of provided API capabilities, allowing providers to support as little or as much of the API as they wish, and allowing clients to discover which capabilities are provided. This approach allows cloud vendors to provide additional capabilities (such as Nirvanix's media transcoding capabilities), and still being compatible with and leveraging the common functions of the API.

Containers and Data

Any data item stored in the cloud, including simple data streams to XAM objects, iSCSI LUNs, database tables and other data objects, can be accessed directly in a form that facilitates peering and transfer from cloud to cloud. As data items can be grouped together into named "containers", and containers can be nested, transferring aggregations of data items is as easy as transferring a container.

Likewise, management operations can be performed on containers, reducing the management complexity when compared to managing individual objects, and allowing changes to be performed atomically on sets of objects. Management properties (metadata) of data items can either be explicitly specified for a given data item, or can be inherited from the parent container.

A Vision for the Future

This simple set of principles allows for a powerful, extensible API that spans all classes of storage in the cloud. Bycast is proud to be participating as a primary contributor to this initiative, and I would encourage anyone with interest in this area to take the time to read the currently released documentation and to get involved at the SNIA cloud Google group.

SNIA is also holding a summer technical symposium in July, held in San Jose. At this event, one entire track is dedicated to cloud storage. If you are in the area, don't hesitate to get in touch with the SNIA to find out what it takes to get involved, and join us in this exciting project.

In the mean time, I'd encourage reading the current draft documentation that can be downloaded from the SNIA, as we're proud of what we've accomplished so far, and are excited about where we're going.

Where does Encryption Fit in the Cloud

2009-06-29T21:48:00.000-07:00

Any analysis of the use of encryption in the cloud always needs to start with a discussion of the threats that the use of encryption technology is designed to reduce. In addition, often encryption alone is not the most important or tricky part of protecting against these threats. A commonly misunderstood aspect of encryption is that it somehow eliminates these risks — encryption just concentrates and separates these risks from the data itself, by moving security to the encryption keys. These keys must then be securely managed and protected against the original threats that encryption was deployed to reduce.

The appropriate use of encryption in the storage stack depends on the business requirements, risks and storage technologies used. For example, while entire drive or tape encryption may be useful in shipping a disk or tape from one branch office to another, it is not appropriate for multi-user document sharing on a NAS, or in any other scenario where the access granularity is at a different level from that of the physical media.

The Top Three Threats

For the sake of this discussion, let us assume that for cloud storage, the most significant three threats are as follows. While this list is not complete, it does include the most common threats considered in the cloud storage space:

Unauthorized disclosure due to cloud customer operations
Unauthorized disclosure due to cloud provider operations
Unauthorized disclosure due to transport eavesdropping

1. The Insider

Despite the focus on exotic and headline-grabbing threats to computer security, the most common form of unauthorized data disclosure is from employees or other authorized individuals within the companies or organizations that generate and use the data. Often these employees even have legitimate access to the data, which is then used in an unauthorized manner, or weak access controls are bypassed to gain access to the data.

In this threat model, encryption within the storage system does not and cannot protect the data. The only approach to protect the data is to store the data in an encrypted format before it is made available to internal end-users. Examples of this include encrypted password-protected PDFs and various digital rights management schemes for media.

Such systems often require positive verification back to a network server before access is permitted, and thus have the trade-off of being complex, costly, and ultimately easily bypassed by taking screen shots or re-recording the protected content. One can only look at the lack of success of digital rights management systems to stem the piracy of digital media to understand that a determined attacker with legitimate access to the content to be protected is an almost intractable problem.

Once again, it is worth emphasizing — Encryption will not protect your corporate data against an insider, and from a security risks standpoint, this is the most probable means of loss.

2. The Provider

Let us assume that a cloud storage system has been selected, and corporate data is being sent outside of the organization's security perimeter (these same risks are present with an internal, or private cloud, as a result of the above risk category). Once data has been stored in the cloud, there are many opportunities for a cloud provider to inadvertently or deliberately cause unauthorized disclosure of a customer's data. These can range from poorly configured firewalls, unauthorized or compromised devices on internal networks, disgruntled employees, or even bankruptcy of the provider where their assets are sold off, along with customer's data, to the highest bidder.

With this threat model, the encryption of the customer's data is a good technological countermeasure that can ensure that while sitting at rest within the cloud provider storage equipment, the data cannot be accessed except by the customer.

Now, an important wrinkle to be aware of is that cloud storage has two different ways which encryption can be architected: Blind, or Transparent.

In blind cloud storage, data is encrypted at the cloud customer's premises, and the cloud provider has no visibility into the data. They can claim no knowledge of the data being stored, and have no way to access it, since the keys are held only by the customer. While this can be a significant advantage to the cloud provider for liability reasons, it also prevents them from building any value-added services into their cloud that require access to the customer's data, and forces all data accesses to go through customer equipment before the data can be accessed.

In transparent cloud storage, data can still be encrypted, but the cloud provider must also have access to either the customer's keys or a second set of keys that allow the provider to access the customer's data. This can be used to provide value-added services such as full-content search, retention management, format conversion, and other capabilities that require the ability to read the customer's data.

From a strict security standpoint, blind cloud storage is more secure. However, with the judicious management of encryption keys, many of the threats mentioned above can be avoided, even if the provider's systems can access the plaintext.

Ultimately, if an organization is giving anyone their plaintext, or the ability to access their plaintext, they need to be sure that the organization has sufficient operational safeguards to protect against common threats to data security and disclosure.

3. The Network

Finally, disclosure during transport is an example of where encryption is virtually mandatory. Any time data is transported across an untrusted or uncontrolled network, such as the Internet, it must be encrypted, and fortunately, there are widely deployed standards, such as TLS, that are commonly used to perform this function. Cloud storage services that uses raw HTTP should only be used if the data being sent is not of concern if disclosed, if the network is completely secured, or if the data is already encrypted (blind cloud storage).

This touches on some of the issues related to the use of encryption in cloud storage, and as can be imagined, there are significant complexities related to key management that make implementations that balance usability and security quite challenging.

Object Security, Continued

2009-05-29T09:35:00.001-07:00

Reader yossib left a comment to the previous blog entry, Cloud Storage - Part 5, Security, that warranted a more detailed response and discussion:

I enjoyed reading your article, your focus on the issue of user authentication and access control is important as it surely does not get the attention it deserves.

Do you see the security model and user access management for object storage evolving from current ACLs, Active Directory/LDAP or taking a different direction

How do you see the concepts of users and groups evolving?

Authentication

Authentication of identity is so critical, because it is the foundation of access control, and as you alluded, deserves far more attention than it gets. Fortunately, the rise of a plethora of services on the Internet is forcing the issue of federated identity management, and while systems are not yet mature, there is a strong trend towards common mechanisms by which a user or computer program can have a universal identity that can cross systems.

Examples of emerging standards include OpenID, and Sun's IDM.

On Active Directory

Active Directory, while hugely successful and very valuable in a corporate setting, simply was not designed to accommodate the scale that is needed, nor the timeframes over which identities need to persist. As digital data and archives become core to our civilization, we need ways to ensure that the security of digital data can survive hundreds of years, and things that were often disregarded as "edge cases" must come to the front and centre.

Examples include:

What happens when someone dies?
What happens when someone gets subpoenaed?
What about the expiration of statutory rights?
What if the law determining the length of statutory rights is changed?

These and so many more issues make the protection of digital assets a double-edged sword — If we enshrine given restrictions in code, can we change them? And if we can change them, how can we prevent this from being defeating the original point of the protections.

And this ignores many of the challenges that are emerging from the loss of centralized control of systems. In emerging federated cloud worlds, objects may pass from system to system, both trusted and untrusted, and security must be preserved. Much of the challenges associated with the work done to try to build DRM systems is directly applicable to trusted repositories and archives, and the research tells us that this is a really hard problem.

For example, it is still an open debate if it is actually possible to have one user grant a second user access without this enabling that user to grant access to further users. And revoking access can be even more thorny.

Ultimately, we need to move away from the centrally enforced security models to a more distributed security model where objects can float around in systems that do not need to be trusted, and access is granted based on trust relationships. (An example of this is that you may grant an online search and indexing company the privileges to read your data, based on your trust that they will not disclose your data).

ACLs

While ACLs have developed a reputation of being far too complex to be manageable, I believe that when tempered with methodologies such as Role Based Access Control, it can be made far simpler for the end user and application developer than it is right now.

However, ACLs fundamentally are merely advisory guidelines for a "trusted" system that interprets them to restrict access. ACLs need to evolve to the point where you have "grants" for each privilage, that enable you to perform that action. So if I wanted to share an object with you, I would give you a "grant" that gives you the ability to read a given object or set of objects. This grant could be revoked, and I could engineer it in such a way that you couldn't delegate the grant without revealing your own credentials.

Ultimately, this involves a much more complex multi-actor interaction, and my gut feel is that we can't do this with static objects. This, of course, would mean that revocation of grants could never really be absolute, (unless they expire, but who enforces that, then?) since you can't always ensure that all replicas of a given object are always kept in sync.

Finally, if these systems grow too complex, they won't work. There is much to be said for simplicity, especially in global-scale systems.

Users and Groups

This is always an interesting discussion — Groups provide such a valuable level of abstraction, but introduce so much complexity. I tend to lean towards abandoning the concept of groups as first class entities. If we just have users, we can create a user that is trusted to act as a delegate on behalf of other users. As long as one user can be granted the authority to delegate privileges to other users, we get the same functionality, and distributed group membership can be re-cast as a trust relationship between the owner and the delegator.

My feeling is that this is the only model that will scale.

Much to Consider

This is just the tip of the proverbial iceberg, and there are so many additional complexities and challenges associated with security. I'd love to continue this discussion, so if you have any questions, comments or ideas, please don't hesitate to comment.

Also, as I mentioned on my last twitter, there are many other security-related items that I plan to discuss further in a follow-up blog post, covering user identify federation, trust domains, "blind storage", peering, object destruction and more.

Object Storage, Part 5 - Security

2009-05-26T17:32:00.000-07:00

In the days when storage was directly connected to computers and there was only one user per computer, security was simple — Just physically secure the computer and attached storage. But fast-forwarding to the Internet age, not only is storage networked, but potentially accessible to every user in the enterprise or Internet. As storage migrates from silos hidden behind computing servers to being a first-class peer on computer networks, security rapidly becomes front and centre as a key requirement, not just to protect (deny), but also to facilitate multi-application and user collaboration and sharing (allow).

As part five of the object storage series of posts, this entry covers the issues related with security in an object storage system. This extends far beyond just simple access controls, such as security issues related to the search functionality discussed in the last entry, Object Storage - Query.

Who's that Looking at my Data?

Fundamentally, security is about controlling the flow of information. Like with any storage system, there is information flow out from the system, and information flow into the system (read and write, in the case of a block device). But unlike a block device, which can only restrict read or write operations on a device or block level, an object storage system has a much richer set of information flows that need to be regulated.

Take query, for example. Information leakage through a query result set or index would be a significant problem, and access controls on each object must extend to the query results. In some environments, even timing matters, as variations in the time required to return a query may allow a user to determine if an object with a given metadata value exists or not, even if they do not have privileges to see that object.

Like all forms of access control, one first needs to authenticate the entity that is requesting access. Once you have determined who is talking with you, then the system can proceed to the question of what they are allowed to do. Then and only then can you proceed to perform the operation.

Expressed in English, this takes the following form:

Entity "X" is requesting to perform operation "Y" against object "Z".

Some examples include,

Entity "XYZCorp\Archiver" is requesting to perform operation "Modify" against object "8D73F687BA26C1A03F9D8E796A497338/com.bycast.metadata.lastmodifiedtime"

Entity "XYZCorp\Admin" is requesting to perform operation "Delete" against object "Financial Storage Archive Container"

Like what we saw with Implicit and Explicit Policies for managing storage placement, retention and others, this same model extends to security. A list of who is allowed to perform what operations on a given object can either be explicitly specified for a given object (this is called an ACL, or Access Control List, in the file world), can be implicitly specified for a given group of objects (inherited ACLs), or can be implicitly specified for a given user.

As one can imagine, these security models can become quite complex, especially when you start combining all three together. For example, if objects are stored into a container that specifies that only one user can access the objects, if the application explicitly specifies that another user can access the object, what is the correct behaviour? Should the explicit specification can override the implicit specification, or should the implicit specification can override the explicit. And both are valid, depending on the use case.

For example, in the Windows world, the "Backup Operator" must be capable of accessing all objects, regardless of their ACLs. This is an example of an implicit security policy overriding explicit security policies. In other cases, if an application wishes to explicitly share some objects with a second application, but by default, all objects should be managed by an implicit security policy, we have an example of an explicit security policy overriding an implicit security policy.

Policy Contexts

As most object storage systems are built around a flat namespace, implicit security policies apply against objects that match a set of metadata criteria. For example, the policy may say "For all objects where "com.bycast.metadata.creator" equals "XYZCorp\dslik" ...".

When logical containers are supported, they can be implemented as a special metadata item, such as where com.bycast.metadata.container having the value of "/corporate/financial" would express a subcontainer "financial" in a container named "corporate". As implicit policies can include which container they are applying to, this allows security policies to be restricted to a given container, or set of containers.

Thus, we end up with three different contexts:

Explicit security policies, included with an object, that indicate who can do what.
Implicit security policies, specifying for which objects they apply to, that indicate who can do what.
Implicit security policies, specifying for which users they apply to, that indicate what the users can do what.

And this leads us to the "What":

Oh, The Things We Can Do

How does the operations that can be performed for object storage compare to file and block storage approaches? Well, in a summary, there's a lot more you can do — Below is a partial list of the actions that one can perform against a stored object:

Create an object
Destroy an object
Discover an object's existence
List an object's contents
Add a new metadata item to an object
Remove an existing metadata item from an object
Read the value of a object metadata item
Write a value to an object metadata item
Add a new data stream to an object
Remove an existing data stream from an object
Read the value of a object data stream
Write a value to an object data stream
Query for the existence of a named metadata item
Query for the existence of a named data stream
Query for the contents of a named metadata item
Query for the contents of a named data stream

These actions can becomes even more complex when you consider that some object storage systems allow read-only metadata items and streams, immutable metadata items and streams, or increment-only metadata items.

Unlike with file systems, which have evolved to a relatively standardized set of operations that are specified in ACLs, the security models for object storage operations is not yet well understood and standardized. Open questions include how privileges are overlaid onto the contents of objects, and how special behaviours, such as increment-only for a retention metadata item are handled.

This is one of the areas where the SNIA XAM working group has done excellent work, and I would encourage anyone interested the details of how security models map onto object storage to read the XAM Architectural Specification.

Cloud Storage Standardization, Part 1 - Why a Standard?

2009-04-29T11:20:00.000-07:00

As cloud storage matures and use increases, there is a strong need for standardized interfaces for performing basic storage operations. Just as the SCSI standard enabled interoperability and facilitated innovation within the directly-attached storage market, the presence of a standard interface for cloud storage will provide many advantages. These include:

Improving quality by allowing standardized conformance testing
Creating a market for test, validation, profiling, debugging and analysis tools
Allowing the creation of standardized documentation
Encouraging the publication of articles and books discussing the standard
Reducing development work required to use cloud storage and to support multiple providers
Enabling the creation of standardized access libraries
Reducing customer lock-in and enabling multi-vendor selection

At Bycast, we provide a RESTful HTTP API for object storage access, and next month, I will be presenting our HTTP API at the SNIA cloud storage summit. As part of my preparations, I've been reviewing many of the other HTTP storage APIs used by other industry players, such as Amazon, Microsoft, and Nirvanix. There is significant overlap and commonalities between all of these APIs, and I believe that there is a good chance, at least from a technical standpoint, to create a common lightweight HTTP storage access protocol.

Just what is "Cloud Computing"?

2009-04-07T23:23:00.000-07:00

As cloud computing increases in prominence, the argument over exactly what cloud computing actually is has also grown in proportions. Never before has a buzzword been so truly nebulous.

So, with the definition in such dispute, this is the perfect time to throw out yet another definition.

With that said, my definition of cloud computing is: *drum roll*

"Distributed location-independent scale-free cooperative agents"

"What?", one might say... "That's nothing like the others I've seen." And they would be right. That's exactly the point. This is my technical definition of cloud computing, and let's pick it apart, piece by piece:

Distributed

Distributed is part of almost every cloud definition, but even the word "distributed" has different meanings to different people. For me, distributed signifies the presence of a substantial separation between entities. This definition implies several things:

That there are well-defined entities that are distinct from each other
That there is a method of communication between entities that is separate from the entities themselves
That there is some degree of isolation between entities

This is broad definition, but one that I would argue that most technical people would agree with. This would mean that a client-server system is a distributed system, as is a telephone exchange or an interacting web server and client. However, a web server alone would not be considered distributed, nor would the Linux kernel.

This definition of distributed is about logical entities and their interactions, so it does not matter if the entities are all co-resident on a single computing system, or scattered across multiple computing systems. That, in my books, is more related to the the property of location independence.

Location-Independence

What separates your typical computer operating system from what is typically known in the academic world as a distributed operating system is the property of location independence. This is what allows the physical location of a given entity (be it a program or state) to not matter to its correct operation. More specifically, location independent entities have the ability to transparently move between locations without affecting system operation.

Location independence is a key enabling technology. For example, in the Internet, DNS and IP routing provides location independence between Internet clients and servers. This allows the servers to be moved around without the clients having to know their physical location, or that there even was a change.

When systems are built out of location-independent entities, they can be run and stored on different physical configurations. This lays the foundation for designing scale-free systems.

Scale-Free

In most system implementations, the chosen algorithms and architectures only work efficiently at a given scale. As the size and scope of a system is increased or decreased, bottlenecks, inefficiencies, and waste result in diminishing returns, and put limits on how big and how small a system can be. Systems designed for portable embedded devices must be designed very differently than systems designed for the worlds largest supercomputers, and rightly so.

However, a new class of computing systems has emerged that has the property of being scale-free. This means that regardless of the scale of their deployment, they are able to continue to perform their task efficiently. From an algorithms standpoint, this means that the computation required scales linearly as a function of the load on the system.

Thus, when you have scale-free location-independent systems, you can scale just by adding more servers, networks and storage devices. This is what the all of the largest (at least in terms of computational load) web companies, such as facebook, flickr, twitter and friends, all aspire to.

Cooperative Agents

Web services counts. SoA counts. In fact, most message-passing distributed systems are based around the concept of cooperating agents working together to solve a problem.

If the agents aren't cooperating, or they aren't agents, then they're not a cloud to me.

What isn't in my Definition of Cloud

It is also interesting to look at what is not in my definition of cloud. The first thing that one may notice when comparing it to most other definitions is the complete lack of business terms. That's because Cloud Computing is a technology, not a business model. There's a reason why previous attempts at cloud-based business models were called "Software as a Service", and "Application Service Providers". That's because these terms describe business models, as opposed to technological models.

Much of my annoyance with the "Cloud" as a buzzword would go away if marketing and businesses just said that they have a "Cloud business model", which provides Internet-based delivery of their application or service and leveraged cloud computing techniques.

In fact, one could arguably develop and deliver such a business model without using any cloud technology. A good example of this is the credit card processing networks, which is mainframe based, largely centralized, and very specifically designed around a given scale, with an implementation that requires very specific location bindings for entities. Yet, it provides what is arguably from a business standpoint, a cloud service.

The Fight over Cloud

2009-03-27T17:50:00.000-07:00

As far as buzzwords goes, "Cloud" is a pretty good one. I've complained about this one in the past, and cloud is a very annoying term to me precisely because of its lack of firm definition, vapidness, and that it means different things to different people.

Now that the much debated "Cloud Manifesto" has been leaked to the web, there's a little more to chew on. The war over the definition of what cloud means has begun.

Below is my analysis of what annoys me about this document, ignoring for now the politics associated with its creation and distribution.

The author(s?) of this "manifesto" say that the document "does not intend to define a final taxonomy of cloud computing", yet that is exactly what they have ended up doing. And given that their definition reads more like a advertisement for outsourcing VMs, this is at odds of my personal view of cloud as a general architecture for building distributed systems.

To me, none of their listed criteria, either independently or in combination, make something "cloud", nor does being "cloud" imply the existence of any of these proposed criteria.

So, with that said, let's take a more detailed look at these "key characteristics of the cloud":

Scalability On Demand

While this is value that can be offered by a cloud, there are lots of non-cloud systems that provide exactly this, and you can have a cloud that does not provide scalability on demand.

For example, IBM has been offering mainframe systems with extra processors that you can pay to use, "on demand". I wouldn't classify a zSeries a cloud.

Streamlining the Data Centre

As "streamlining" is an ambiguous word, we'll assume that the authors mean outsourcing or cost reductions. However, not all uses of cloud will result in the reduction of cost (capital or infrastructure) or moving work outside the data centre.

Improving Business Processes

Technological systems are more often than not orthogonal to business process improvement. One can deploy cloud systems and end up with a worse business process, and one can improve business processes without deploying cloud technology.

Minimizing Startup Costs

Fractional allocation to reduce the minimal quanta that must be purchased to do useful work is the closest to a acceptable criteria for cloud, but is a VM server a cloud, then? Also, one can deploy a cloud system that does not support fractional allocation.

All Together Now?

Depending on your definition of what cloud is, you could create a cloud system that provides fixed capacity processing, increases data centre costs and brings additional work into the data centre, makes no changes to business processes, and requires large startup costs.

Conversely, you could create a system that has variable on-demand capacity, reduces data centre costs, outsources data centre work, improves business processes, and minimizes startup costs, all without it being a cloud.

Of course, the linchpin of this entire argument is just what a cloud is, and this is why this manifesto matters. It is the first major attempt to put a stick in the sand and say that Cloud is X, Y and Z.

And that is why is has generated such a storm of controversy. At stake is who gets the first mover advantage in the struggle to define what exactly a cloud is.

Object Storage, Part 4 - Query

2009-03-25T17:45:00.000-07:00

Being able to store data is of limited use unless there are efficient mechanisms by which data can be located and retrieved. In fact, one can argue that file systems are just special purpose data allocation and query systems. Most users and applications locate and access files through a directory of one sort or another, be it a file system, relational database or a custom index within a proprietary file structure, highlighting the importance of query in data storage.

As part four of the object storage series of posts, this entry covers the use of metadata to provide rich query capabilities, and how these capabilities enable implicit policies as discussed in the last entry, Object Storage - Explicit and Implicit Policies.

Where Has That File Gone?

Most of us have experienced the frustration of searching for a file and not being able to find it. But we have it easy compared to how data was stored before the widespread adoption of the file system.

In block-based storage systems, data is accessed by an address that defines where the data starts, and a length, which determines how much data needs to be read (or written). While this approach is simple and efficient, it is difficult to manage, as you need to keep an external catalogue of where each item is stored.

File systems simplified the problem for the user and the developer by creating a standard directory that could be used to organize files into hierarchical structures and associate metadata, such as file names and creation dates with each file. With the introduction of the file system, search systems were able to look at the directory and build an index of information such as file names, dates and other metadata.

So, we progressed from walking the disk to walking the file system directories. We then skipped to reading an index, then reduced the index results by filtering out all entries except the ones that matched our query. WIth full-text indexing of the contents of the files, in addition to the file metadata, now integrated into the operating system, a user can even filter their results to just files that contain specific words and phrases.

Of course, these impressive improvements in technology have been largely offset by an explosion of files, as hundreds of thousands of files to millions of files are now quite common in home and small office settings, and in large enterprises, tens to hundreds of billions of files can reside on enterprise storage systems.

But What About the Developer?

Despite these impressive achievements, for software developers, the facilities offered by a file system have not changed significantly since early file systems were created in the 60's and 70's. While search has improved the life for users, developers often are forced to create their own application-specific index of files for searching purposes.

A good example is to look at two popular applications offered by Apple on the Macintosh platform: iTunes and iPhoto. Both of these applications store each song and photo, respectively, as a file. But as a user, you never see these files — you see a custom user interface that is designed for tasks associated with managing and playing music, and managing and organizing photos.

When you open these applications, they do not access every single photo or song. They access indexes, which allow queries to be performed quickly to get results to the user interface. Thus, if you want to hear those hits of the 1600's, or view a slideshow of photos tagged "Tafoni", iTunes and iPhoto is actually doing a query, much like the following SQL:

SELECT * WHERE CENTURY == "1600"

SELECT * WHERE TAG == "tafoni"

Specifically, the applications are performing a query for metadata, and the century that a song was composed in, and the tags of a photo are all examples of metadata.

Unfortunately, a file system only has limited fixed metadata, and doesn't understand or have a way to be extended to include application-specific metadata. But an object storage system does understand metadata, and thus offers developers powerful query features that can dramatically reduce the complexity of development while also increasing the value of metadata interoperability across applications.

What can Object Query Do?

Simply put, an object storage system can do everything that a basic relational database can do, but without needing a schema. Every piece of metadata associated with objects can be queried, and the metadata is arbitrary, defined by applications and end users.

Storing an object with metadata is analogous to a INSERT
Changing metadata in an object is analogous to an UPDATE
Deleting metadata or an object is analogous to a DELETE
And object storage query is analogous to a SELECT

Because the metadata is an intrinsic part of each stored object, you never have to worry about transactional consistency, or inconsistencies between an index and the actual metadata of the object.

And, one can visualize implicit policies as just policies performed against a query result. Specify the metadata constraints, perform a query, and apply the policy to the results. (In reality, it is a little more complex, but we'll get to notifications later in the series)

If iTunes used Object Storage...

To illustrate how query in the storage system is of significant value to developers, let's imagine that Apple included an object storage system as part of the Mac OS, and had written iTunes to use object storage instead of using SQLite.

When you first start up iTunes, it remembers the last view you were looking at. A view is the results of a query, so it would issue that query to the storage system for the metadata of all song objects that match the query parameters. This would return the list of metadata that is used to display the list of songs. When a user double-clicked to play a song, iTunes would open the song data from the corresponding object, and start playing.

There is no need to incorporate a database, no need to create a fixed schema (just add metadata and go!), no need to worry about consistency or corruption, and way less code to manage.

Most importantly, because the query is done by object storage system, your query performance scales as your storage infrastructure scales. So a query from a million objects on a home PC runs just as fast as a query from a billion objects within an enterprise. And as the object storage system is improved over time, all the applications get faster, for free.

But easing the load on the software developer isn't the only value. Because the objects are stored in a common storage system, if iTunes desires, it can allow any application to query for the music it manages. So if another application developer wants to each for music, it can construct a query that will return results from iTunes' repository. Controlled access across applications opens up all sorts of opportunities to create systems built around loosely coupling agents accessing and manipulating a shared repository. For example, a format conversion tool could convert MP3 files into AAC files in the background, transparently, and fully interoperate with iTunes.

Of course, if an application wanted to keep its objects private, that's just a permissions setting. Which is an excellent segue into the next entry — Security in a object storage system.

On Ten Year Trends - Parallel Goes Mainstream

2009-03-12T11:03:00.000-07:00

Several bloggers have been discussing the current "ten year trends" that are transforming the computing environment, including Cloud Computing, Virtualization, Flash, and Mobility.

However, there is a deeper trend that while well underway, is so significant that it is often overlooked. That is the forced transition to loosely coupled distributed computing models caused by the knee in increasing compute performance by making individual processors faster.

Over the next five years, I believe we are going to see a full-fledged transition in hardware systems from tightly coupled single and multi-processor shared memory systems to loosely coupled many-processor systems that communicate via message passing. This trend is already evident in the high-performance computing space, where having hit the limits of individual processor performance, then having hit the limits of shared memory, virtually all new architectures have adopted this approach.

One of the fundamental drivers of this transition is a new economic model that measures computing power per dollar, or computing power per watt. This model favours many smaller less powerful processors over fewer high-power processors. After all, with Atom and ARM processors costing only a few dollars per core in volume, and performing around the same as a 1 GHz Pentium did just a few years ago, with a fraction of the power and heat dissipation, I predict we will soon see a transition to inexpensive servers where one or two rack units are crammed full of hundreds to thousands of these class of processors, all connected together by what looks like a Ethernet fabric.

The timing is right — Microsoft's talking about it, IBM's already done it, and the mobile market has brought the component volumes up to the point where the economics are right.

But five years does not make a ten-year trend: The larger change is that to programming models, languages and mindsets. Very few people in the industry can work comfortably and productively in massively parallel environments, as we have seen in the difficulty in building scalable web-based systems like Twitter. Computer Science still teaches programming and computer architecture around the old model of the "one true core", with parallel and distributed programming taught as a speciality, often at the graduate studies level.

It is going to take five or more years for these new massively parallel hardware and ways of thinking to filter down to the next generation of programmers who can think parallel, create new tools, libraries and stacks, and then play tens to hundreds of thousands of nodes like an instrument.

Imagine having a hundred thousand cores on your desktop. Imagine having ten million of them in your data centre. The problems are significant, and exciting.

Object Storage, Part 3 - Explicit and Implicit Policies

2009-03-09T17:41:00.000-07:00

Once metadata becomes an intrinsic part of each stored object, application-specified metadata provides a rich vocabulary by which to enable applications to communicate with the underlying storage system and for administrators to manage them.

As part three of the object storage series of posts, this entry covers the ability to specify explicit and implicit policies that gives both the application and the administrator control over how data is managed, and drives additional value in the storage subsystem. This entry builds on top of the last entry, Object Storage - Metadata, which introduced the importance of metadata and why it applies to storage and to object storage in particular.

More Than Just a Bit Bucket

When people first think about storage, they think about bits. And that's fundamentally what storage systems do. They take your bits, keep them, and give them back to you. But if that's all a storage system does, it's pretty dumb, as there is a lot more to what applications need then just storing bits.

Storage is also more than the storage system and applications — The storage administrator is also an important player in enterprise storage, and is often charged with goals that may or may not agree with the desires of the application.

So, if storing bits is "Dumb Storage", what is "Intelligent Storage"? Well, an easy example is the below list of many of the things that applications and administrators want to have their storage system do for them:

Index
Protect
Share
Compress
Replicate
Distribute
Archive
Cache
Tier
Version

And this list is just the beginning.

Explicit Policies

In order for these higher-level behaviours that are desired by administrators and applications to be fulfilled, they first need to be communicated to a storage system. And metadata fulfils this role perfectly.

If an application wishes for a given stored data to be protected such that only that application can access it, it needs only to attach metadata to the object that indicates this intent, and trust that the storage system will honour its request. This agreement between the application and the storage system is the contact of functionality.

Want multiple copies? Add metadata. Want it shredded on delete? Add metadata. Want index keywords? Add metadata. Etc.

Thus, through a vocabulary of well-defined metadata that it will honour, the capabilities of a storage system can be advertised to an application. And the sum of this metadata forms an explicit policy, specified by the application to the storage system, as an atomic part of the stored object.

Implicit Policies

But this isn't the only way that policies can work. While the application knows best from its perspective, it is only one small and limited part of an enterprise. There are larger forces at work — desires to ensure that data is not lost in a disaster, desires to reduce costs, desires to meet legal obligations, and to manage information over time and space.

Enter the storage administrator.

Like with explicit policies, implicit policies are also built around metadata. But instead of having an intent being explicitly stated as metadata directives, implicit policies map an intent to a collection of objects with common characteristics.

Let's imagine that the storage administrator wishes to ensure that critical financial documents are protected against a site disaster, and are retained for a minimum of ten years. The administrator can create an implicit policy that says:

For all objects with metadata that indicates it is a financial document, make two copies, one remote, and keep them for ten years.

Once again, the metadata is key. The metadata might be a path in a file system, or a document type, or the division within an organization. Regardless, the administrator now has a tool to take subsets of stored data, limited only by their imagination and the avaialble metadata, and make things happen.

And unlike explicit policies, implicit policies can be changed without having to change the metadata of the stored objects. By combining both types of policies, an application can specify metadata (an explicit policy) that selects which implicit policy is to be used. And all of these approaches can be combined: An application may say that this object "Must be high performance", that the retention is "governed by implicit policy named Finance", and say nothing about replication.

As can be imagined, this approach to storage management is very powerful, and allows the value of storage to be expressed in terms of business values to an organization. In our next entry, we will look at some concrete examples of explicit and implicit policies, and see how these are often implemented in object storage systems.

Object Storage, Part 2 - Metadata

2009-02-24T17:12:00.000-08:00

A key aspect of object-based storage is the storage of metadata as an intrinsic part of each stored object. Allowing applications to define and include arbitrary metadata as part of a stored object provides the foundation for enabling many enhanced capabilities for both the application and the storage system.

As part two of the object storage series of posts, this entry covers the importance of metadata to object storage and builds on top of last week's entry, On Object-Based Storage, which introduces Object-based Storage.

What is Metadata, Anyways?

Metadata is one of those unfortunate terms that the computer industry has abused to the point where it now has a wide range of meanings. From a strict definitions standpoint, metadata is "data about data", but in more general use, it refers to any descriptive data.

For example, a file name is considered to be metadata about the file, as is file creation date, access lists of who is allowed to access the file, and a thumbnail view of how the file would look when printed.

Figure 1 - Objects Including Data and Metadata

Metadata is often blurred together with data, and can be considered to be data depending on the context. This can be illustrated in the below example of a compound object document. For example, an indication of which typeface to use for a paragraph of text is often referred to as metadata associated with the text, as is the location offset and scaling factor describing how an image should be rendered. While this information is indeed metadata to the paragraph and image, respectively, from the context of the document, this information is an intrinsic part of the document that must be present for it to be rendered as the user intended.

Figure 2 - Composite Objects Including Metadata as Data

Thus, while in principle, data can stand alone without the associated metadata, in reality, the metadata often provides the context that makes the data usable. (After all, think about how hard it would be to find a file without file names or directories)

Metadata for Applications

For most applications, being able to attach metadata to stored data is a fundamental requirement for structured storage. By storing metadata along side, or intermixed with the data, applications are able to ensure that the data is sufficiently described for manipulation or display to the end user.

Since there is no universal file format, each application vendor has had to choose between living with the limits of standard file formats, or creating their own proprietary file format. While standardized file formats have emerged over the years, and often include the ability to be extended to include application-defined tags or properties, there is no "universal file format", and more often than not, application developers resort to creating their own format. And even with general formats such as XML, without additional descriptive information, such as a schema (more metadata), the files are not self-describing.

While object storage does not create a universal file format, it does provide a consistent and standard way for applications to store metadata along with data, in a format that is independent from any specific application. Providing applications with a consistent way package up all of the data and metadata into a single storage object, then commit it atomically to storage provides many advantages over ad-hoc solutions.

For example, let us use the example of someone who is writing a web-based blogging system. Each blog entry has the body text in HTML (the data), and a series of metadata items associated with the post, such as a title, creation date, posting date, posting status (draft, posted), and an author. A typical design would be to use a database to store the metadata, and store the HTML posts as files. In this implementation, even if the data is stored in the database (always a temptation, but rarely a good idea), databases are intrinsically loosely coupled data stores, and much complexity and fragility ensues.

Contrast this to an application designed around an object store. For each post, a "Blog Post Object" is created, which includes the post data, and the metadata. The object is committed to storage as an atomic element, where it persists. Each committed object is self-describing, no schema is needed, and the amount of complexity to the application writer is vastly reduced.

Now one might question how an object store is different from a database, and that is a good question. In fact, one could consider a database a specialized form of an object store, and an object store to be a specialized form of a database. And ultimately, both of these perspectives are correct. The key aspect to keep in mind is where services are being provided — In a database, services are provided to the application by a middle layer that runs on top of a non-intelligent storage system, where with object-based storage, services are provided by the storage system itself. This is a key difference, which will be the basis of much of the remainder of this series of articles.

Metadata for Storage Systems

Today's storage systems is like having an illiterate librarians managing a building full of pages of paper. People given a page, they put them in a location, people ask for a page at a given location, and they give them back the page. While this works, it's not very intelligent.

But what if storage could be more? What if you could ask for a book? What if you could ask for all the books about a given topic? Or by a given author? Object storage is our literate librarian, who understands the metadata associated with stored objects.

When the storage system understands what is being stored, this enables all sorts of capabilities and optimizations that would otherwise not be possible. Now the storage system can provide the ability to search content. Now the storage system can intelligently optimize storage and retrieval performance and latency. And most importantly, now there is a way for the application and the storage system to communicate with each other.

There are many exciting capabilities that emerge from having richer communication between the application and the storage system, and these are worth describing in more depth. The subsequent entries in this series will discuss these emergent capabilities, including query, placement, protection, permissions, representation, policies, compression and versioning. These will be the subject of the entries to follow.

Watch for Goats in the Cloud

2009-02-17T14:39:00.001-08:00

George Crump, of Storage Switzerland posted an article titled, Cloud Storage Reality, where he talked about the emerging class of "Cloud Storage" solutions. His conclusion: That Cloud Storage is a reality and ready for prime time.

But is it? Or more specifically, is all that is called "Cloud Storage" ready for prime time.

George's listing of the key advantages of cloud storage when compared with traditional enterprise storage systems, in dispersion, nodes, scale, granular, ease and self-upgrading are dead-on.

"Say, What's that mountain goat doing clear up here in
this cloud bank?"

Similarly, we agree with his taxonomy of the three different deployment models, Service Only, Software Only and Pre-packaged Cloud.

But there is a key distinction between different ways that clouds can be deployed that can make the difference between a high-risk failure and a low-risk success:

Storage Just in the Cloud?

While cloud storage is a proven architecture, pure Internet-based storage remains risky. Before enterprises will be willing to trust their data and their business to a provider, they first look for industry maturity, stability and reliability. After all, the pure internet-based storage industry is still in early stages of adoption, and one can argue that it already failed once, during the "Storage Utility Provider" craze at the beginning of the decade. Heck, even Enron was getting into that business.

And enterprise uptime is only half the QoS battle — Even if the remote storage service provider has 100% uptime, access to the provider is limited by the reliability of the Internet networks, and access is restricted by the bandwidth to the Internet.

After all, despite all the talk of bandwidth being free, the costs of an OC-3 to the Internet still makes most CFO's reach for their chests.

Then, if you really want to kill pure Internet-based storage, get the lawyers involved...

What Really Works

Cloud Storage is production ready and widely deployed, but only in configurations that extend into the customer's data centre. I would wager that virtually all enterprise-class cloud storage deployments include data being stored in the customer's data centre. You see this with profiles of Amazon's S3 customers, and we see this with our customers. This is to be expected, of course, since all private cloud deployments exist primarily within the customer's data centre.

So, to summarize, where does internet-resident cloud storage work?

Cloud Storage providing off-site protection copies for data that is also held on-site.

Cloud Storage providing lower-cost storage for data where high levels of QoS are not required.

Cloud Storage facilitating data sharing across sites.

Object Storage, Part 1 - Introduction

2009-02-15T23:11:00.000-08:00

Object-Based Storage is an alternate approach to specifying an interface between higher level application programs and storage devices for the purposes of storing digital data. While not commonly used, the many advantages of object-based storage are resulting in increasing adoption, and over the next decade it is expected to become widespread.

As the first part of a series of blog posts talking about object-based storage, this post introduces object-based storage in the context of other widely used storage interface technologies, and briefly covers the advantages inherent in object-based storage.

Stream-Based Storage

In early computing systems, data was stored as series of bits or bytes that could be written or read over time. Examples included ticker tape for bit-streamed data, and punch cards for byte-streamed data. Within a computing system, some of the first persistent storage systems were based around writing data as sequences of magnetic signals on tape or on a rotating drum. These stored values could then be read back in the same order they were written.

While this approach is still used in storage devices such as tape, for low-latency storage purposes, the main disadvantage of stream-based storage was the time required to access a given piece of information. In order to access a given piece of information, a program had to specify the location within the stream that the information was contained, which required keeping track of many bits of addressing information. Thus, in order to save bits, the locations in the stream were divided into equally sized "blocks" that could be used to refer to locations within the stream.

Block-Based Storage

From an interface standpoint, today's hard disks are still conceptually modelled as a long stream of bytes divided into equally sized blocks. All accesses to the storage devices are performed by reading or writing blocks over industry standard protocols such as SCSI and Fibre Channel, which typically specify that each block contains 512 bytes of user-accessible information.

Under the covers, the hard disk controllers understand that the data isn't actually stored in one long sequential stream of bits, and is stored across multiple platters of spinning discs, and the physical location of a given block is specified by translating the block address into a distance and angle on the surface of the disc. For solid-state storage devices, block addresses are mapped to physical groups of semiconductor devices arranged in two or three-dimensional structures.

File-based Storage

Given that application data, be it documents or images or sound data, are often larger than a block and may not fill up blocks completely, a higher-level logical structure maps these documents, typically called files, onto the blocks on disk. This software is typically called a file system, and provides a directory of files, metadata about the files, and information known as "extents", which points to the list of blocks that contain the contents of the file.

When a file system is layered on top of a block storage device, the user is already being presented with a form of object storage. In this case, there are two types of objects, files and directories. However, this is where the similarities end.

Object-Based Storage

In an object-based storage interface, instead of writing a stream of data, or writing blocks of data, or writing a file, an application program writes an object. In interfaces such as XAM (eXtensible Access Method) and OSD (Object Storage Device), instead of manipulating files, applications manipulate objects, and can perform a much wider variety of operations on these objects.

There are several aspects of objects that differentiate them from files:

Objects are compound. While files can also contain multiple different types of data, as is commonly found in an .zip or .tar archive file, objects intrinsically contain multiple different pieces of information together into one package. Metadata and data are all stored with an internal directory that allows each sub-component of the object to be accessed.
Objects are self-describing. Instead of just having the metadata that is associated with a file, such as creation time and file name, objects have arbitrary metadata specified by an application that is specific to the application and data's nature. For example, a sound object can have sampling frequency, gain levels and other sound-specific metadata, while a document may have metadata describing the application that created it, and may even contain a low-resolution preview image of the document.
Objects are self-contained. Unlike a file, the metadata associated with an object is an intrinsic part of the object. When a file is stored on disk or even copied from one location to another, the metadata is not stored as part of the file contents, and may not be transferred along with the file data. With objects, the metadata is an intrinsic part of the object, and cannot be separated without creating a different object.

How is This Different?

This isn't new, and these three concepts have been widely used for organizing the contents of files stored on block devices in most software systems. Every Microsoft Word document, Macintosh Resource Fork and Windows Executable is already an object based on this definition. So, why is object storage needed when these concepts are already widely implemented?

The key to answering this question is looking at the entire storage stack, instead of looking at each part in isolation. In a file-based storage system, if an application wants to load a preview image from a file, the application must first open the file, know how to understand the file (which is typically application-specific), and read the data. At the next layer down, the file system must translate these read requests (which view the file as a stream of data) into block addresses using the file system extents metadata, issue block requests to the storage device, which in turn need to look up the location of the data in the blocks to return it back up to the application.

With object-based storage, the application tells the storage system direction that it wants data from a specific object, and the request can be sent all the way down to the storage device, which because it knows about the object structure itself, is able to better respond to the request, and process the request more efficiently.

And because the storage device can understand the structure of objects, it is able to more intelligently and more efficiently handle these application requests.

This may seem like a subtle distinction, but there are numerous advantages resulting from standardizing object structures and pushing down awareness of object contents to the storage layer, and this additional awareness enables many new capabilities and increased efficiencies. These will be the subject of the entries to follow.

Software Development and Evolutionary Biology

2009-01-29T20:52:00.000-08:00

I've finally figured out why, over time, open source will become the dominant form of software development:

Open source is, well, open. They can freely share knowledge, experiences, mistakes and achievements without restrictions.
Open source projects cross-pollinate, exchange code and rapidly change over time.
Open source projects tend to have significant diversity, with multiple competing projects that do the same or similar things.

From an evolutionary biology standpoint, we have more frequent sharing of genetic code, higher rates of mutations, and a broader diversity. Thus, these projects have a better chance of survival.

Contrast this to closed source development projects:

Closed source is still open, in that people take their experiences, skills and knowledge with them when they move from company to company. But instead of information being shared on a hourly basis, the sharing of information tends to be on the time scale of years, and the degree of sharing is far lower.
Closed source projects resist "contamination" of their code base, and try to prevent any leakage of their code to the outside world. They also try to do the minimal amount of changes necessary in order to maximize return on investment.
Closed source projects tend to avoid markets where there is significant diversity. Interestingly, when there is competition, the products involved tend to be far better than in areas that are dominated and controlled by one or two vendors.

As a consequence, closed source projects tend to evolve slowly, become rigid and inflexible, and are unable to adapt to rapidly changing environments.

When you mix open source and closed source together in an ecosystem, interesting things happen. The first is that the competitive pressure resulting from open source pushes closed source projects to act more like open source projects. Closed source projects also tend to initially be parasitic, using open source without contributing much back.

There are exceptions to all of these generalizations, but this feels about right to me. It would be fascinating to see the results of research that looks at software development from the viewpoint of evolutionary biology.

Jump Starting Off-Site Storage

2009-01-29T18:25:00.001-08:00

During a discussion about Cloud Storage on the StorageMojo blog, Pete Steege asked about the challenges of the initial load of data into a cloud storage provider.

How are storage cloud companies handling the “first backup” issue? Multiple terabytes or petabytes that need to be migrated to the cloud initially?

The incremental part of the process is a no-brainer.

One solution that we use at Bycast is to deploy two or more edge servers with attached storage at the customer's premise, and allow them to perform bulk ingest over the REST API (or via CIFS/NFS). When the ingest is complete or nearly complete, the object storage repository on disk can be physically shipped to be integrated into the "cloud", and subsequent transactions can be performed over the network.

Another approach taken by EMC's Mozy backup service is what they call "data seeding". Here, you purchase a 2 TB USB drive from EMC, store your data onto the drive, then ship it back to EMC in order to get the process going. I couldn't find any current references of this capability on their web site, so this may no longer be a supported feature.

With such hybrid models, you need the software intelligence to ensure that the data is always accessible via the cloud API, always protected from loss, corruption and unauthorized disclosure, and to ensure that the data is audited to ensure that you know that all data that you ingested locally made it into the cloud.

Despite the associated complexity, this is a very powerful approach, as one should never underestimate the bandwidth of a 747 full of disks or tapes.

When to use S3

2009-01-23T16:08:00.000-08:00

In response to my previous post, jeredfloyd of Permabit asked about when S3 would be useful use as storage for our customers.

Do you feel S3 has the reliability and availability for your customers today? I love the concept, but I've so far been scared off by horror stories of downtime. Also, what about security concerns?

These are good questions, and I'm going to elaborate on these concerns and where we see S3 as providing value to our customer.

The Bottom Line

I wouldn't use or recommend S3 for anything other than a low-grade secondary replica location for redundancy purposes. Having said that, the levels of reliability and accessibility that I've seen are already higher than what my experiences have been with tape libraries.

Bring Your Own Security

From a security standpoint, I wouldn't put anything on S3 that hasn't been encrypted and wrapped with an integrity verification layer, as we do in StorageGRID. And if the data is encrypted, there is less of a concern about deleting it if you can't get to it any more. Just throw away the keys.

As you can't implement secure wipe using their API, even if you overwrite the data, so you would also want to be sure that you're not storing really sensitive information there, even with today's standard encryption algorithms and key strengths.

S3 Isn't Inexpensive

One of the things that I want to emphasize is that based on our analysis of their economics, if you are storing data for long periods of time, it's far cheaper to just add storage nodes with SATA shelves.

Tape isn't cheaper until you're looking at 50+ TB libraries. For infrequently accessed data and redundancy copies (you need to make more when putting them on tape, since it's not as reliable as disk), it quickly becomes very economical for large capacity deployments.

Despite this, S3 Still Has Value

Having said this, even with these concerns, I see several situations where S3 support brings real value for our customers:

If you're really small (less than 50 TB), adding storage capacity is still pretty expensive as a percentage of your yearly budget because our customers typically add in 10TB or larger increments. Using S3 as an overflow pool (keeping one or two copies locally on disk, and using S3 as your second or third copy) lets you defer that purchase for a little while, and when you do make that purchase, you can automatically migrate all the data on S3 off onto your new storage resource.
If it takes you too long to purchase hardware, or your budgetary cycle for capital purchases takes too long, or even just an unexpected load where there just isn't time to provision more storage, you can shift second or third copies off onto S3 to free up space, and expense it to the business as a opex or project cost.
If you have a short-term storage need, and don't want to invest in hardware yet, just put it off onto S3. It will cost a little more per TB, but since wouldn't be able to amortize the storage costs of in house hardware across the typical three-year lifespan of that hardware, it works out to be cheaper in the end.
If you're almost full, you've ignored the alarms telling you that you don't have enough space on other nodes to repair your storage redundancy if you loose a node, and you don't have any storage ready to replace a failed storage, S3 would be a good "last resort" option for creating new replicas to restore your desired level of redundancy.

So, to summarize, I'd use it for a sort-term storage resource to defer capital costs, a short-term emergency storage resource to keep you going, and for storage of short-term data. And in all cases, I wouldn't have the only copy in the grid on S3.

Based on these use-cases, it would be of most value to smaller IT shops with smaller systems. As you get into larger archives and storage systems (200+ TB), many of these situations will never come up.

Regardless of your size, having S3 as a choice as a storage tier gives administrators another tool to handle different situations, and that flexibility can be quite useful. Ultimately, it's up to them to decide if the costs (and bandwidth usage) makes sense for them.

Some Notes on Amazon S3

2009-01-23T01:07:00.001-08:00

During our recent meetings, there were a fair number of questions and discussions about the economics of public cloud storage providers, such as Amazon's S3 service.

This YCombinator discussion has lots of good information about pricing, usage and experiences of some of S3's supporters and detractors. It's well worth reading.

Interestingly, thanks to a new S3 user space file-system FUSE module, Bycast has pretty much everything we need to provide a S3 tier of storage to our StorageGRID customers. Of course, such a capability would need to be productized, which would allow an administrator to have a place to configure the tier and securely enter and store their S3 credentials through our administrative interface, but thanks to the filesystem virtualizing the S3 API, all the hard work is already done.

Cloud Storage Protocol Standardization

2009-01-23T00:34:00.000-08:00

During one of the panel discussions at the SNIA Cloud Storage Summit, the topic of why standards for data exchange and system management would be beneficial for cloud storage. While there are many different advantages, one of the areas that I spent a few minutes talking about was the development efficiency that is realized as a result of standards.

When a protocol is standardized and adopted multiple vendors as a way to connect systems or subsystems, the following things start to emerge:

A formal protocol specification
Web pages describing the protocol
Books about or with chapters about the protocol
Example open-source implementations
Standard interface libraries
Conformance test suites
Benchmarking suites
Protocol analysers and recorders

In essence, an ecosystem starts to emerge around the protocol, and many small companies and individuals build expertise and tools that enable the rapid uptake of the protocol.

It's been my observation that the software developers and architects tend to have a major say in the selection of protocols, especially for subsystem interconnects, and that they tend to choose the protocol that makes their life the easiest. Thus, protocols that have all of these resources widely and inexpensively available quickly become the protocol of choice, resulting in a continued upward spiral of adoption, experience, tools and systems.

We're starting to see this with XAM, with #1, #4 and #5 already available, and #2, #6 in progress. And I'm sure that somewhere out there, someone's writing a book about XAM, or at least a chapter about it.

In the cloud storage protocol arena, Amazon's S3 service has such a strong lead in this area with their S3 HTTP protocol that many of these resources have already been built, despite it being a proprietary protocol. While most other cloud storage service providers have built similar HTTP protocols, with the IP ownership restrictions around Amazon's protocol still up in the air, there is a fair bit of uncertainty if their protocol will ever be able to be used with more than S3.

Which leads us back to the need for standardized protocols.

At the SNIA Winter Symposium

2009-01-23T00:29:00.000-08:00

I've been busy attending the SNIA Winter Symposium this week. In addition to the usual XAM workgroup meetings, I've also been participating in the Cloud Storage Summit.

On Wednesday, the organizers were gracious enough to let me present a quick talk on Private Storage Clouds, where I covered the economic drivers behind cloud storage, the differences between public and private clouds, talked about how Bycast's StorageGRID software allows the creation of private clouds, and discussed some examples of customers where we are deployed and in production.

While I didn't have as much time as I would have liked, I was able to cover a majority of the points that I had planned to discuss, and the session was well received.

It's interesting to see how much surprise there was regarding how our system is being used by customers. Robin Harris of the Data Mobility Group called us the "most surprising company" based on what we have been doing. In contrast to most of the cloud storage deployments, many of our customers are placing mission critical data on our storage system, either for archive or primary storage, and they have no backups outside of the redundancy provided as an intrinsic part of StorageGRID.

During some of the other sessions, the data being stored on many other cloud deployments was described as being "data that could be lost", or "the garbage dumpster".

Thanks to an innovative use of WebEx, you can view all of the talks online at the cloud storage sessions page. I'd encourage you to have a look at these, as there is quite a lot of interesting material here, especially the views of the analysts from the Thursday sessions.

Concurrency and Complexity

2008-12-23T05:26:00.000-08:00

HP's Russ Daniels understands a critical issue facing the next generation of software development:

"If you think about whatever those skills are that are necessary to be able to program a computer, we don't understand what they are very well, but we know that they're not very widely distributed. So you end up with a relatively small portion of the population who can write a program successfully. Then you start introducing certain kinds of abstractions: if you introduce recursion, you lose some people; if you introduce pointers, you lose some people; if you introduce event-driven models, you lose some people; when you introduce concurrency, you lose virtually everybody.

So we create these programming languages that support concurrency and then you find that there's a lock and an unlock around every method, because otherwise people hurt themselves. It presents an interesting dilemma—we need to be able to get more stuff done, we need to figure out how to do stuff in parallel, but we're pretty bad at doing that."

One of the keys to solving this complexity problem is to make the mechanics of parallelism safe, automatic, and transparent to the programmer. Ultimately, this approach needs to extend far beyond just parallelism, as we see the same failings in the areas of security, fault tolerance, fault recovery, dynamic scaling, and resource economics.

An ADE Example

In order to see how this could work, let's take a quick look at an example of how ADE transparently provides concurrency and protection of resources that must be accessed serially:

Let's imagine that we're tasked with building a simple LAN switch, and we want to provide an IP address resolution for connected Ethernet hosts. If we want to send an IP packet to a given computer, we first need to find what the MAC address of the computer with a given IP address is. This is performed using the Address Resolution Protocol (ARP), which was originally documented in IETF RFC 826.

At a simplified level, the protocol works as follows:

A host which wants to talk with a specific IP address sends an ARP broadcast request asking for the MAC address that traffic should be addressed.
The switch intercepts the broadcast, and looks up the IP address in its local cache. If the IP to MAC address mapping is found, it responds to the originally requesting host with the ARP response.
If the mapping is not found in the local cache, the switch broadcasts the ARP request to all other ports.
If the switch receives a ARP response, it adds the IP to MAC mapping to its local cache, and forwards the response to the originally requesting host.

This is equivalent to a simple database insert/lookup problem, and in traditional concurrent programming models, errors can easily be introduced if care is not taken to lock the local cache while performing modifications.

In contrast, because ADE handles concurrency by forcing all component interactions to be atomic transactions, no locks are required and the system can be proven to be correct and deadlock free.

In this example, we are going to consider three actors: The ARP Lookup Process, the ARP Cache Process and the ARP Discovery Process. These actors have the following roles:

The ARP Lookup Process receives ARP lookup requests from the network hosts, asks the ARP Cache Process if the mapping is known, asks the ARP Discovery Process for the mapping if not known, and responds to the requesting host.
The ARP Cache Process looks up mappings in the local cache, and adds mappings when asked.
The ARP Discovery Process sends ARP lookup requests to the network, and when a mapping is discovered, it asks the ARP Cache Process to remember them.

In this model, there would be one instance of the ARP Lookup Process per ARP lookup packet received on the network, one instance of the ARP Cache Process, and one instance ARP Discovery Process per cache miss.

In a busy network, there could be dozens of ARP requests being processed, and many concurrent requests that may require changes to be made to the ARP cache. Even if ARP discovery is adding a new mappings at the same time that a new request is looking up a mapping, there is no possibility that the cache can become corrupt, because all access to the cache is automatically serialized by virtue of the entities and their messages that are exchanged.

This is the key aspect of ADE that simplifies the creation of distributed systems. Because concurrency is automatic, when problems are decomposed into collections of interacting actors, any parallelism inherent in the problem will be automatically utilized.

XAM Active Objects

2008-12-15T19:28:00.000-08:00

The SNIA XAM standard provides a comprehensive object model and interface for the storage of dynamic data objects. The standard provides methods by which collections of data (known as XSets) can be created and deleted, data can be stored, retrieved and manipulated inside these XSets, and queries can be performed to locate XSets that match specified charactertistics.

While the first version of the standard was under development, I worked with Jared Floyd of Permabit to ensure that the XAM Job model was architected in such a way that it would include all of the components required to support active objects, like those supported in ADE.

What are XAM Active Objects?

This is best illustrated by an example:

Let us create a hypothetical XSet that includes an XStream (binary data) that contains the bytecodes defining a Java program. When this XSet is committed, it can automatically start executing. The entity that executes these active objects can either be performed by a sidecar system that attaches to the XAM Storage System (XSS), and becomes aware of new active objects via the standard XAM query facilities, or the execution of these active objects can be an intrinsic part of the XSS.

The Java program would be executed in the security context of the XSET it is contained within, and thus it can access local data stored within it's own XSET, and optionally perform XAM operations based on its security credentials. This allows it to read other XSets, create new XSets and perform queries to discover XSets.

For example, an active XAM object could remain resident within the storage system, performing queries for specific types of XSETs (for example, PDF objects), and convert them into a newer version of the PDF format. Such an model would allow dynamic format conversion of archived content, just by loading in a new XSET. This model is also be useful for analysis, where a large number of XSETs need to be analysed or datamined in the background.

Because XAM Storage Systems will often be distributed, multiple active objects can easily be parallelized across the compute infrastructure the makes up the system, and parallel computing patterns are easy to implement, as one active object can create XSETs that act as child active objects (the equivalent of performing a fork in UNIX). As code can be bundled with data, this model will also enable the creation of data-driven MIMD and SIMD parallel data processing systems.

XSETs become process contexts, complete with inputs, code, state and outputs, with the full ability to discover and access data within the storage system, create new data, spawn child processes and report status information to external systems. Because XSETs belong to user contexts, adding processing quotas, limits on computing resources, and other policies can use the same methods as used for other XAM policies.

Part of the Standard?

All this could be implemented today as a vendor-specific extension to the standard. However, adding this capability to the XAM standard would require work to be done in the following areas:

Active object language profiles would have to be defined, since multiple languages including Java, .net and interpreted languages such as ruby could all be supported.
Language bindings would need to be standardized to allow these programs embedded in the active objects to be perform XAM operations on the XSS they are resident in.
Standard job-style status reporting would be beneficial to allow standardized active object execution status monitoring.
Policies for computing-related aspects, such as resource usage and quotas would need to be defined.
Requirements for security isolation would need to be defined.

Given that none of these would require changes to the core XAM standard, this could easily be added as an optional part of the standard, much like Level 2 query, which provides full-text search within XStreams.

In summary, the XAM standard provides a foundation upon which a rich data-driven distributed computing system can easily be created. This opens up many intriguing possibilities, and would be relatively easy to formally add to the standard.

The ADE Process Model

2008-11-23T15:18:00.000-08:00

To illustrate how ADE-based systems operate, it is best to use an example. In this entry, we will examine at a high level how a simple ADE program is run, from bootstrapping to shutdown.

ADE Versions

Over the last ten years, there have been three major revisions of ADE:

ADE V1 was developed in 1998, and was a minimal system designed to facilitate experiments with message-based computing. In this version, ADE was hosted as a library that ran on Mac OS.
ADE V2 was a result of continued improvements over the next two years, and included a pure message-based execution model and the ability to run directly on a processor without requiring a host operating system.
ADE V3 was created specifically for the Bycast StorageGRID platform during 2000 through 2004, and has been tuned to provide high levels of performance while retaining most of the advantages of the ADE approach to distributed computing. In this version, ADE was updated to use an Operating System abstraction layer, allowing it to run on a wide variety of UNIX-like and embedded operating systems.

In order to demonstrate the pure message-based execution model, this example is based on ADE V2.

Bootstrapping

When ADE first starts, there are no processes present and no messages being sent. When in this state, there are no processes to send a message to, and there are no processes that could send a message. Thus, in order to transition the system into an executing state, we must "bootstrap" the system to the point where there is at least one process executing.

Given that ADE processes are just messages, when I refer to a message as a process, I'm referring to a message that contains executable code as part of the message contents. And since messages are just self-describing data structures (known as "containers"), a message can be serialized to disk. As messages are created, manipulated and exchanged in serialized form, one of the key design criteria of the container data format was highly efficient serialization.

We use this property to bootstrap the system. Once the ADE kernel is running, we load an (initially hand-created) message into memory. This message contains executable code for our bootstrap process, as well as the state for the process.

Figure 1: An executable message (Capability ID 100)

Loading this message into memory involves the following steps:

An entry is added to an in-memory database, keyed by the message capability. The message capability is a randomly assigned 128 bit identifier assigned by the system that uniquely identifies the message.
The kernel creates a protected memory space for the message. This memory space provides the execution context.
The serialized message is loaded into this memory space.

At this point, we have a process in memory, ready to do work, but no messages to trigger any work to be done. In order to kick off execution, we need to load a second message into the system. This message is far simpler than the first message, as it just triggers the executable code in the first message. This message consists of a message capability, a source CID (set to zero for bootstrapping), a destination CID, which is set to the capability identifier for the first message, and a message type, which is a string that identifies which executable code in the destination that should be invoked.

Figure 2: A message destined to CID 100

This is read as, "Message CID 200, from CID 0, to CID 100, invokes /create".

Message Processing

When this second message is loaded into memory, the presence of the destination capability identifier field triggers the message to be added to a scheduler queue for delivery to the specified destination capability. The ADE kernel sees this scheduler entry, and moves the triggering message into the memory space of the destination message, where it becomes part of the destination message. The scheduler then switches to the context of the destination message, and executes the code indicated by the triggering message. As this executable code has access to the message contents, it can now inspect the contents of the triggering message, and manipulate its own message state.

Figure 3: Message CID 100, after receiving message CID 200.

In our example, message CID 100 then executes the "create" method, and the executable code in the create method creates and sends a new message to itself (which is the only possible destination, and the only destination known to the process), invoking the "destroy" method. The final step in the "create" method is to clean up the contents of triggering message from the process message state.

Figure 4: Message CID 100, with message CID 300 about to be sent.

This newly sent message in turn gets enqueued by the scheduler, then processed, causing the "destroy" method to be invoked by the destination process. This executable code for the "destroy" method removes the entire state of the message, which then results in the message itself being removed (and the entry being removed from the in-memory database), returning us to our original pre-bootstrap state.

In Summary

This process model, of sending messages to other messages, executing code in destination messages, manipulating the state of the destination messages, and creating new messages, allows full computational capabilities, and provides a rich foundation for the creation of many advanced distributed computing models.

In our next ADE post, we'll examine some of the design patterns and capabilities that emerge as a result of this process model.

SAS for Blades

2008-11-19T11:09:00.000-08:00

Blade-based computing form factors offer many potential advantages for constructing storage subsystems using standard hardware, but until recently, the lack of cost-effective storage connectivity has hampered blades when compared to 1 and 2U servers with directly connected SCSI or SATA/SAS shelves.

As I mentioned in an earlier post about IBM's BladeCenter S storage attachment capabilities, this is starting to change as low-cost SAS storage connectivity switching is integrated into blade solutions.

And now I am pleased to see that HP has introduced a new product, the HP StorageWorks 3Gb SAS BL Switch. This product provides eight external facing SAS 3GB ports, and if you install two of these switches in a c3000 chassis, you can attach up to 16 external SAS/SATA shelves. Assuming 12 TB of SATA storage per shelf, that allows you to have 192 TB of raw addressable storage per c3000.

This is perfect for high density storage systems, as you end up with 6U for the c3000, 32U for the storage shelves, and 4U remaining for a KVM and networking equipment. As the c3000 series blades also allows you to install "Storage Blades", you can host databases locally to the blade. This makes an excellent platform for a scalable high-density StorageGRID deployment, as each site can start with two blades providing basic services, and as storage capacity is expanded, additional shelves and blades can be added as needed. And building out to 200TB per rack is a pretty respectable density. One could easily use this hardware to build a 1 PB storage system based around five racks running StorageGRID, connected together using 10 GB network uplinks.

When comparing this to the IBM offering, there are both advantages and disadvantages. In order to allow an HP blade to access the SAS switch, you have to install a Smart Array P700m controller card, which represents an additional cost. With the IBM offering, the RAID controller in part of the SAS switch module itself. But given that the HP switch allows twice the SAS attachment density, and that having the controller card as part of the blade provides a greater degree of failure isolation, I'm inclined to prefer the HP solution.

But regardless of the minor differences, the bottom line is that both systems are now far more viable for use as lower-cost storage infrastructure due to the elimination of the need for fibre-channel based connectivity to the storage shelves. This is a huge improvement in terms of costs, and one that we won't see again until FCOE (or SAS over Ethernet) becomes widely deployed.

ADE: A Decade of Distributed Computing

2008-11-18T00:54:00.000-08:00

The Asynchronous Distributed Environment (ADE) is a message-based framework for the creation of distributed grid computing systems, including the Bycast StorageGRID. With over 40,000 node-years of production runtime, this framework represents one of the more mature distributed computing environments.

I originally developed ADE as a testbed for distributed computing concepts, and published a paper describing the system, titled "The ADE Environment: A Macroscopic Object Architecture for Software Componentization and Distributed Computing" at the MacHack conference back in 1998. As an environment for experimentation, it was very successful, allowing the rapidly exploration of many patterns for creating massively concurrent distributed systems, and for refining the environment to improve the programming model and associated infrastructure.

The ADE Process Model

ADE takes a rather unique approach to the traditional CSP process model: processes are messages. Thus, instantiating a process is as easy as sending a message that includes executable code. Each message has a unique "capability" identifier that allows messages to be sent to itself. When a message is received by a process, the executable code corresponding to the received message is run, which allows the process to manipulate its state (the process message) and to optionally generate additional messages.

An example of a more complex process trace can be found as part of this Graphviz .dot file.

This approach has several major advantages:

Parallelism is implicit and automatic, as dependencies are expressed as message exchange relationships. All forms of serial and parallel computing can thus expressed as directed acyclic graphs.
Processes can easily be migrated from one system to another, as they are just messages. Lightweight and automated migration permits experimentation with alternate approaches to distributed processing, such as migrating processes to a data source or resource.
As processes are just messages, the entire execution state of a system can be captured by checkpointing all message across a given cut line, and by retaining historical message states, execution can be run backwards, and "anti-messages" can undo processing.
Execution state can easily be logged and inspected for debugging and visualization. By inspecting the real-time message graph, deadlock and livelock can be automatically detected, as can the critical path for performance optimization.

And as one can imagine, this is just the tip of the iceberg.

A Maturing Environment

Much has changed over the last decade, and ADE has transformed from a research system into an industry proven technology. As ADE originally was based on ATM networking, we re-wrote the networking and node-to-node messaging to use TCP/IP, and the process model was changed to allow statically linked code to be associated with processes, instead of including it with the message. As storage systems require extremely high execution performance, we optimized for message processing speed and efficiency, and some of the less-used features, such as process migration, were retired from our production builds.

The result has been a high-performance, yet flexible system that facilitates the rapid development and testing of distributed systems. By focusing on allowing distributed software to be easily developed, visualized and tested, we've been able to rapidly innovate and add functionality to our system. And ultimately, this has been the most significant advantage of ADE.

Welcome, EMC

2008-11-10T14:59:00.000-08:00

Today, EMC announced their Atmos product, perviously code-named "Maui". It is an interesting offering, and one that we are quite familiar with... given that object-based storage is what Bycast has been developing since 2000, and has had deployed and operational in customer sites since 2002.

Let's do a quick rundown of the core features that Bycast StorageGRID offers that are also offered by Atmos:

Object based storage? Check
Metadata with objects? Check
Multi-site? Check
Multi-tenancy? Check
File-system interface? Check
Web-services API? Check
Policy-driven replication? Check
Compression? Check
Object-based De-dupe? Check
Object versioning? Check
MAID-style spin-down? Check
Web-based admin? Check

There are also a couple of significant features found in StorageGRID that are not present in Atmos:

Encryption? Unknown
Object integrity protection? Unknown
Clustered gateways? Unknown
Storage hardware independent? Nope
Storage vendor independent? Nope
Support for tape and other high-latency media? Nope
Six years of operation in the field? Nope
Mature, customer-proven product? Nope

Over the last five years of deploying large-scale distributed object-based storage systems, we've learned a lot about what things work well, and where things go wrong. Based on the initial feature set, we know that Atmos will be a successful offering for EMC. There will, of course, be the usual teething pains, not unlike those that happened with Centera, but that is the nature of all 1.0 software.

So, to our friends at EMC, I say, Welcome. We're glad to see object-based storage becoming mainstream.

On Graphing Data

2008-11-03T22:44:00.000-08:00

Over the years, I've had to do a fair bit of data analysis for network monitoring and software instrumentation. Computers, especially when monitoring themselves, can easily generate vast amounts of data, and when you are looking for the one anomaly or correlation, the only way to quickly find them is to represent the information visually. And for data that changes over time, that means generating graphs of the data.

A Little History

Back in 2000, when we starting building our web-based management and monitoring tool at Bycast, we spent a lot of time looking at different third-party graphing toolkits. This was long before AJAX and mature client-side JavaScript engines, and simply weren't able to find any that even met the following basic requirements:

Low-latency chart generation
Anti-aliased rendering
Data binning
Display of minimum, maximum and average values

Having looked at many poorly rendered charts, I set very high visual standards for our chart rendering classes. And since many of our graphs would be displaying data collected over weeks, months and years, a single graph could often require the processing of hundreds of thousands to millions of data points. As one might imagine, this was a non-trivial problem.

Min-Max-Average

Attempting to plot these values directly would never be efficient or responsive for the end user, so we grouped data points into time-based bins, calculated the minimum, maximum and average value of each bin, and rendered the bin values. Since each chart had the same number of bins, this allowed us to optimize our data processing and storage, while still allowing high resolution display of data, and allowing users to easily zoom into areas of interest.

This is best illustrated with an example:

This graph, which shows system memory usage over time, has a dark green line, which indicates the average memory usage, and the light shaded region shows the minimum and maximum range within each bin.

This allows deviations from the average to be easily identified visually. For example, one can see that memory usage dropped significantly on the evening of September 28th, and at one point, fell to almost 210 MBytes. This would not be visible with a chart that only displayed the average value, as most charts tend to do.

We've found that displaying this information is very important. Often it is the deviations from the average that are the most important to focus on, and this method of displaying them allows one to quickly identify and zoom in on the areas of interest.

Graphing Guidelines

I have long been looking for a well-documented set of guidelines for how to render graphs, but until now, I had not found anything that met the bill.

In August 2008, Microsoft's Microsoft Health Common User Interface Group released a new guidance document, titled Displaying Graphs and Tables. This document is excellent, and I would encourage everyone involved with information visualization related to graphing to read this document. It fits exactly what I had been looking for, and is very well written.

Some Additional Graphing References

Here are some references that I have read and found useful when designing graph-based information visualizations:

Beautiful Evidence, by Edward R. Tufte
The Elements of Graphing Data, by William S. Cleveland
The Grammar of Graphics by Leland Wilkinson

Microsoft Azure is Cloud Storage

2008-10-27T22:21:00.000-07:00

Microsoft today announced their Cloud Computing initiative, the Azure Services Platform. While many of the commentators have been comparing it with Amazon's EC2 offerings, and with VMWare based hosting, Azure is as much about cloud storage as it is about computing.

Quoting from Microsoft's Port 25 Blog:

Both Windows Azure applications and on-premises applications can access the Windows Azure storage service, and both do it in the same way: using a RESTful approach. The underlying data store is not Microsoft SQL Server, however. In fact, Windows Azure storage isn't a relational system, and its query language isn't SQL. Because it's primarily designed to support applications built on Windows Azure, it provides simpler, more scalable kinds of storage. Accordingly, it allows storing binary large objects (blobs), provides queues for communication between components of Windows Azure applications, and even offers a form of tables with a straightforward query language, Chappell says.

This is worth emphasizing. Azure provides global cloud-based object storage. And it takes it one step further, by providing active objects such as queues and tables. The presence of queues, message busses and other persistent data structures are a real game-changer, as they form the location-independent "glue" by which to hold together large-scale loosely-coupled applications that are best suited for cloud-based hosting. This directly competes with Amazon's S3, and by creating a platform that runs .Net and other interpreted languages directly without the weight of a full OS running in a VM, it should be able to scale much more elegantly.

For storage object access, the API Microsoft has adopted is very similar to the HTTP API used by Bycast for the StorageGRID platform, so this continues the trend of standardization around HTTP for object storage and access.

Their API documentation can be found at the below link:

http://go.microsoft.com/fwlink/?LinkId=131258

Also of interest from the announcement are Microsoft's .Net Services for Ruby, which provide first class access to the Azure services for ruby applications.

Now all we need is a XAM VIM that talks to Azure. Their containers map closely to XSets, blobs to XStreams, and stype data can be placed in tables and thus be queried. They've even provided methods to handle cache coherency for multiple simultaneous writers. Very interesting...

Discussions on the Web

2008-10-23T14:03:00.000-07:00

XAM and content-based search

A post about the advantages of including support for full-text search in XAM implementations, and my response regarding the pros and cons of providing this functionality at the storage layer instead of the application layer.

The Universal Design Pattern

A discussion of a commonly used design pattern where objects have properties that are both inherited and overridden, and my response regarding the similarities of the XAM objects storage model, and it's use for persistent storage of serialized objects created when using this design pattern.

Web 2.0 and Archiving

2008-10-19T15:37:00.001-07:00

A few days ago, Facebook announced that they had reached the milestone of having 10 billion images stored in their servers. This translates to over 40 billion files under management, as each image is stored at four different resolutions. In total, all of these files requires 1 PB of storage space, and most likely consumes 2 PB of physical storage, assuming their data is mirrored at a minimum of two sites for availability purposes.

If we take those 40 billion files and 1 PB of storage consumed, we can figure out that the average file size is around 25 Kbytes. All of these files need to be stored, managed, distributed and replicated — exactly the solution space for an archive. And this is why I believe that the Web 2.0 problem space has been often viewed as an archiving problem.

But the Web 2.0 problem is larger. It is not the number of objects stored that makes this an engineering challenge, but the number of objects accessed: Facebook serves 15 billion images per day, thus, serving more files each day then the number of unique images they have stored! At peak load, they are serving over 300,000 images per second. Even though a majority of the images accessed each day fall in a small (<0.01%) subset of the total image set, the images that are popular change daily, and the storage system must handle these "hot files" efficiently in order for the system not to collapse under the load.

Thus, while it is true to say that Web 2.0 sites require an archive (for all those infrequently or never again accessed objects), due to the retrieval requirements, an archive is just one part of the infrastructure required to serve a Web 2.0 workload.

Extraordinary Concurrency as the Only Game in Town

2008-10-07T16:58:00.000-07:00

DARPA released a new report today providing a detailed analysis of the challenges involved in scaling high-performance computing up to the exascale levels, three orders of magnitude higher then today's petascale supercomputers.

The report is available for download at the below URL:

TR2008-13: Exascale Computing Study: Technology Challenges in Achieving Exascale Systems

This report covers many fascinating issues, ranging from advancements in semiconductor technologies to interconnects to packaging and all the way up the stack to the software that would run on such a computing system.

What jumped out to me is the degree to which concurrent programming models have become so critical to scaling computing, and the degree to which this is still an open problem. Single threaded programming has reached the end of the performance road, and the only way to improve is to parallelize. This is directly in agreement with the research in concurrent programming and runtime systems that I have been running for the past decade, and I'll be sharing more thoughts related to my findings and future research over the coming months.

Resilient Exascale Systems

A computing system with millions of loosely coupled processing elements running billions of concurrent processes will be continuously encountering failures at the hardware, interconnect and software levels. A successful software environment must automatically and transparently recover from such faults without passing the burden of complexity on to the programmer.

Ultimately, the only successful approach will be a dispersal-driven replicated checkpoint-and-vote architecture which allows a trade-off to be made between reliability and efficiency. This approach offloads the complexity of error detection and recovery from the programmer, and allows the system to automatically optimize system efficiency (a higher voting degree trades compute efficiency for time efficiency, and a lower voting degree trades time efficiency for compute efficiency).

This presence of such a resiliency layer also allows the use of fabrication techniques that have far higher defect densities and error rates then traditional system fabrication, as near-perfection is no longer required. This is a key to enable feature size and voltage to continue to be reduced, to provide high-density arrays of compute elements, and to support larger dies and stacked wafers.

Storage for Exascale Systems

One of the most fundamental changes we need to make is to stop separating storage resources from computing resources. CPU power is continuing to increase, and storage density is continuing to increase, but the bandwidth and latency between the two is not.

Instead of moving the data to the compute element, we need to move the computation to the data. The first step is to replace the traditional demand-paged memory hierarchy with generic compute elements that can act as programmable data movers, create vastly larger register stores, and consider such data movement as a subset of a more general message passing approach. Instead of just sending data to fixed computational processes, we need to turn processes into messages themselves, thus allowing processes to also be sent to the location(s) where data is stored.

Ultimately, persistent storage (for checkpoints/state) and archival storage needs to be automated and under the control of dedicated system software, not specified as part of the software. Checkpoint information needs to be streamed to higher-density storage, as does parallel I/O used to read and write datasets being processed. This allows us to move to a model where storage is just process state (and thus, ultimately, just messages), and the files as we know today are just views into the computing system, much like reports are views into a database.

Blades for Storage

2008-10-02T15:47:00.000-07:00

Over the last decade, I've had a significant interest in the benefits of blade server architectures. In early 2000, I worked on a development project that used CompactPCI cards from ZiaTech (later acquired by Intel) where we used the CompactPCI bus as a high-performance low-latency system-to-system interconnect. Several years later, we also evaluated blade systems from IBM and HP to see if we could leverage some of the benefits of blade systems for StorageGRID deployments. But, at every point, there were major downsides, namely, cost, bandwidth and storage interconnectivity.

Thus, until now, I've never found a product that provides the storage connectivity that allows a blade system to complete economically with DAS attached shelves linked to 1U servers. That changed this week, with IBM announcing the new BladeCenter S Chassis.

Enabling Features of the BladeCenter S

There are three features in this product that allow it to be a perfect foundation for software-based storage systems:

Feature #1 - SAS/SATA RAID modules, which allow you to hang up to eight SAS/SATA shelves off each BladeCenter chassis. With 12 TB of SATA storage per shelf, that allows you to have 96 TB of raw addressable storage per BladeCenter. This feature is unique to the IBM offering at this time, and is the key enabling feature of the product for storage software solutions.

Feature #2 - Internal SAS/SATA storage, with two bays, each holding 6 3.5" drives (up to 6 TB SATA, 3 TB SAS). This is perfect for system and database disks, and allows all of the control storage to be co-located with the compute blades.

Feature #3 - 10GigE switch modules and blade NICs give you the bandwidth you need to provide high-performance storage services to external clients without bottlenecks.

Scalable Hardware for Storage Software

With the BladeCenter chassis starting at less than $2,500, this allows a common architecture and hardware platform that extends from entry-level systems consisting of two HA blades with internal storage all the way up to multi-chassis systems managing hundreds of terabytes of storage. This also reduces support costs, as you can leverage a common pool of replacement parts.

The blade architecture allows the compute hardware to be field replaceable, which reduces MTTR. And as the attached storage is assignable across blades, there is the possibility to design a system that has a hot standby blade that dynamically reassigns the storage associated with the failed blade, and resumes providing services.

If I were to design a new software storage product, this would be the foundation hardware I would choose. The software would run in VMs, granted, but this hardware finally provides the cost/performance/supportability tradeoffs that I view are critical for such a product to succeed, and provides sufficient built-in storage to allow a common hardware platform to extend down to entry-level systems.

Google Trust

2008-09-23T17:08:00.000-07:00

Search providers such as Google, Yahoo and Microsoft are in a unique position to provide indicators of trustworthiness of a given web site, as they act as a trusted intermediary between end-users and their desired destinations. Currently, there are currently no clear methods by which an end-user can assess if a given site in a search result is indeed controlled by the organization that it claims to represent. If there were a way that a user could identify which sites had verified identities, this would result in vast improvements in the trust relationships between end-users and the organizations behind any given web site.

The User Experience

Let us consider a hypothetical user expierence: A user navigates to Google and enters a company name. They are presented with a listing of search results, often including the official company web site, other web sites for companies with a similar name, sites reviewing products by the company and sometimes sites maskarading as the official site.

In the search results listing, each site where the identity of the organization that controls the site has been verified includes a special icon known as a "trust mark". This icon indicates that Google has established a chain of trust that allows the identity of the organization responsible for the content on that site to be verified.

Figure 1: An example UI from Safari indicating the validity of a certificate.

The green check icon is a good example of a visual representation of a trust mark.

The presence of the trust mark may be sufficient for the user to navigate to the site in confidence, or they may click on the trust mark, showing a page containing legal information about the entity, including their location (using Google Maps, of course). This information would especially be useful for disambiguating different companies with similar names.

The Technology

Standard web certificates are already used for secure transactions and providing information about the authenticity of a secured web site. But these are limited to the secure sections of web sites, such as pages for authentication and payment processing. Most web sites do not use SSL/TLS for the bulk of their web site due to the computational cost of processing HTTPS transactions when compared to standard HTTP.

However, the same certificates used to provide HTTPS could also be used for indicating a degree of trust. By placing the certificate as a file in the root path of the web site, the Google crawler could retrieve a "certificates.txt" file, much like the current "robots.txt" file. As most certificates contain the top level domain name, Google would be able to verify the chain of trust of the certificate, check to make sure that the URL it was crawling matched the URL in the certificate, and then display the trust mark and associated information.

As this approach leverages existing infrastructure, does not require any new protocols, and allows web sites operators with existing certificates to immediately use them for this purpose, this would facilitate rapid adoption of this technique.

Lightning Between the Clouds

2008-09-15T15:38:00.000-07:00

Today's VMWare announcements about vCloud are the first concrete announcement of a product that enables migrating VM sessions across corporate boundaries. With the success of VMotion at a technical level, the concept of outsourcing your DR centre was an obvious next steps, and vCloud formalizes the ability to define these relationships in code.

However, there are many significant technical and non-technical risks associated with such a service offering, including:

Storage Synchronization - In order to migrate VM sessions, the storage backing the VM session must also be synchronized across the two organizational entities. This plays well into EMC's hands to bundle location-independent storage along with VM functionality.

Security - VM sessions contain sensitive in-memory data, such as encryption keys, that (in a well-designed system) never makes it to disk. This is on top of the security issues associated with the storage associated with the sessions.

Multi-tenancy - An outsourcing provider will most likely be running VM sessions for multiple customers on a common infrastructure. Thus, network isolation, in additional to storage isolation and partitioning, become major issues that are not present when all resources are within a single enterprise.

Management - Resources need to be billed, QoS monitored, SLA's tracked and enforced, and loads predicted and managed. All this will have to work seamlessly across both the customer's and outsourcer's infrastructure. The back-office part of such a service is always under-estimated, and is critical to get right in order for a service to succeed.

Solving these problems is a pretty tall order, and I have to commend VMWare for their vision. This is version 2.0 of the VM revolution, and things are starting to get really interesting.

Time for Compliance

2008-09-12T13:23:00.000-07:00

Many aspects of compliance storage rely on trusted time. These include timestamps indicating when an object or file was stored, retention durations that indicate when files must not be deleted or modified, and audit records indicating when operations were performed against the storage system. All of these timestamps must be accurate, and, more importantly, must be resistant against attack in order to satisfy the multitude of compliance regulations, such as Sarbanes-Oxley and HIPAA.

A Question of Time

When evaluating such a storage system, here are ten good questions to ask your vendor:

How and when is the clock set?
Who can set or adjust the clock?
Are changes to the clock audited?
How much can the clock drift over time?
If the clock is synchronized, is the synchronization chain trustworthy?
Is clock synchronization traceable to the NIST?
If clock synchronization is no longer possible, how does the system react?
When clock synchronization is regained, how does the system react?
What protections are present to prevent tampering with the clock at the system level?
What protections are present to prevent tampering with the clock at the network level?

Two Architectures

Generally, two architectures have emerged, one that involves a completely sealed system that is capable of maintaining accurate time with drift less than one minute per year for the life of the system, and one that involves network-based transactions that cryptographically prove that a given event happened at a given time.

The advantages of the first architecture include strong resistance against tampering, and low maintenance requirements. However, the downside to such an architecture is the requirement for custom hardware, both to keep accurate time (the clocks in typical servers range from largely inaccurate to downright embarrassing), and to provides the means to physically secure the hardware from prying eyes and screwdrivers. Because this requires custom enclosures and maintenance contracts (who do you trust to have the keys to the rack?) this typically lends itself to solutions from larger storage hardware vendors. And, after all, if you are spending hundreds of thousands to millions of dollars on something, it better well be able to keep accurate time.

The second architecture is unfortunately far more complex and difficult to design and implement correctly. In a software-only solution, very little can be relied upon to be trusted. After all, a standard x86 server is only one boot-disk away from unfettered tampering, and it's difficult to detect if you are running under a hypervisor. Thus, such systems must rely on complex network transactions to determine accurate times of events, often resulting in increased transactional latency. Unless these time transactions are designed and tested to ensure that a malicious time source or compromised node is unable to alter the timestamps and compliance durations, this can be a significant point of weakness.

Beware NTP

One protocol to keep a watch out for is NTP. A malicious NTP server, combined with a poisoned DNS cache and the quick throw of a circuit breaker might result in all your compliance data being unprotected ahead of schedule, or even worse, automatically erased from the system. And given that NTP security is rarely used and not well regarded, it is almost a certainty that it forms a weak link in the chain of trust.

Many systems that use NTP just use it to set the server and operating system clock, which they then trust blindly. For a given server, this clock can be easily altered, and in order to obtain trusted timestamps, information from multiple sources that can not all be easily compromised must be used.

Time is of the Essence

Time is often overlooked when evaluating compliance storage, but is a fundamental aspect of the compliance process. After all, in a court of law, if the timestamps of events cannot be proven to be accurate, and retention durations cannot be shown to be enforced, that expensive compliance system may end up being even more expensive.

Raining on the Cloud

2008-09-09T18:19:00.000-07:00

A thorn in my side, as of late, has been the Wikipedia article on Cloud Computing. Describing yet another newly coined buzzword for distributed computing, this article contains many examples of of the worst of Wikipedia, and reminds me of some of the articles I have been subjected to by SoA fundamentalists (and and before them, that of the CORBA-cultists, etc).

Cloud, as in Network Cloud

The concept of Cloud Computing originated as a analogy to the network cloud, a mainstay of whiteboard and Visio diagrams everywhere. Thus, in order to understand what it means, one must consider what a network cloud means. Fortunately, this is simple to answer and relatively un-contentious: all a network cloud means is "the stuff we don't have to worry about". It's infrastructure. It's the stuff that we can let the network and/or the networking people figure out how to make work, and by ignoring those details, allows us focus on the problem at hand.

If we take this concept of the network cloud and apply it to computing, we end up with "The practice of using known resources to provide computational services as a component of solving a larger problem." Just like that network cloud in the diagram indicates that we don't care how the packets get from site A to B, cloud computing allows us to not worry about how and where computation is performed.

Distributed Computing, Renamed, Yet Again

When viewed from this perspective, cloud computing is just yet another flavour of distributed computing, one where computational services are provided over a network, typically the Internet. The fact that the users of these services does not have to own, control, manage or even be aware of how the service is provided, is important, but not ground-breaking. The only key difference is that the service contract and information hiding resulting from a well-defined and managed service allows application complexity to be built on top of the services without having to worry about their implementation or operation.

When it all works, that is...

Vanishing into the Infrastructure

2008-09-03T16:48:00.000-07:00

Throughout our history, it is our infrastructure that we build our civilization upon. From roads to electrical power grids, from the telephone network to the soon to be ubiquitous Internet, it is the technologies that we no longer think about that enable our way of life. For it is when technology fades into the infrastructure that things become interesting:

This is when the next generation of technology emerges.
This is where the majority of the dollars under the adoption curve are found.
And most importantly, this is when the benefits to human existence become widespread and far-reaching.

Fading into the infrastructure is not easy, nor predictable. After all, it involves technical and business challenges that far exceed the complexities of any original innovation. But for those who thrive on these challenges, it offers opportunities to make a far-reaching difference.

Welcome.

Tape	Disk
Sequential access	Random access
Relatively slow	Fast
Shipped offsite	Electronically vaulted
Once a day process	Periodic or continuous
High operational touch	Automated
No-dedupe	Dedupe
Inexpensive media	More expensive