Cloud Storage Concepts and Challenges

This week’s article should have been part 2 of our look into iSCSI but I wanted to take a break while the SC09 conference is happening. So the next parts of the iSCSI article will come later. This week I want to discuss a new but potentially very important trend in storage – “Cloud Storage.”

This type of storage has a number of names. For example it is sometimes called “on-line archiving” or “RAIN – Redundant Arrray of Inexpensive Nodes”, but “Cloud Storage,” for better or for worse, is the name that is the most popular. The overall concept is fairly simple – Cloud Storage takes distributed storage nodes and combines them using a file system of some type, to create a single storage system but with fairly low performance (hence the phrase “on-line archive”). It relies on replication rather than RAID for data resiliency by having multiple copies of the data on various nodes.

It’s a fairly simple concept but like anything it has its attractive features and its challenges and the devil is always lurking in the details. This article will explore some of the basic concepts and implementations of Cloud Storage.

Object Storage Introduction

Debating the fact that storage is growing at an alarming rate is at best a futile discussion. Certain sectors are growing at a rate that is truly unprecedented and possibly frightening. For example, gene sequencing applications generate massive amounts of data to the point where various groups have several Petabytes (PB’s) on-line and regularly talk about hundreds of Petabytes in the next 2-3 years. Users of the data want all of it on-line, all of the time. They understand that the performance doesn’t have to be the top-notch since they may not access the data for long periods of time(hence the phrase “on-line archive”). However, they want to have the data on-line all of the time and many times in a single namespace.

Traditional storage concepts can start to break down in the Petabyte (PB) range. For example, block based storage in the PB range has a massive number of blocks. If we assume a 4 KB block size, than a 1 PB file system has the following number of blocks:

1 PB = 1015 bytes = 1013 KB

Number of Blocks in 1 PB = 1013 KB / 4 KB/block = 2.5 x 1012 blocks

That’s 2.5 trillion blocks that a file system has to address. Just imagine an fsck of this file system. Now multiple the number of blocks by 100 or even 1,000 (Exabytes) and you understand why people are searching for alternative storage solutions to traditional block based schemes..

The current concept that has the greatest possibility of succeeding is object-based storage. The “object” term has been somewhat over-hyped in general, but it actually works well in the context of storage. Rather than have the file system track each block of the file system in a data structure, an object based file system breaks the data into pieces. Each piece is an object that contains not only the data but also some additional information such as an object ID. The file system then just has to track the object ID’s which is much less metadata than tracking all of the blocks associated with the data. This approach gives you more freedom in designing the file system and its features and allows you to scale to larger file system sizes.

Jan Jitze Krol of Panasas has a great analogy for object storage. Let’s assume you are going to a mall to go shopping. You can drive your car into the parking garage and park on the third level, zone 5, seventh spot from the end and then go in and shop. When you finish you come out and you have to remember where you parked and then you go directly to your car. This is an analogy to the way a typical file system works. For the object storage analogy, let’s assume you arrive at the parking garage and the valet parks your car for you. You go in and shop and when you finish you just present the valet with your parking ticket and the valet determines where your car is located and gets it for you. The advantage of the valet is that they can move the car around as they need to (perhaps they want to work on part of the garage). They can even tear down and rebuild the garage and you won’t even know it happened. This analogy of moving your car around without your knowledge gives tremendous flexibility to the file system. The file system can optimize data layout (where your car is located), add more storage (add to the parking garage), check the data to make sure it’s still correct (check your car to make sure it’s not damaged and if it is, fix it for you or inform you), and a whole host of other tasks. Notice that these tasks can be done with minimal involvement of the metadata manager.

RAIN instead of RAID

Regardless of whether the file system is object based or not, there is always the issue of storage resiliency. In this context resiliency means the ability to tolerate failures while maintaining access to the data. Data resiliency in the Petabyte range is very important for a simple reason – it is virtually impossible to provide backups for storage in that range. Consequently, focusing on the resiliency of the storage is extremely important.

Everyone is probably aware of RAID which is a technique for data resiliency. It is focused on data resiliency within the node itself, allowing failures to happen within the storage node without lose of data resiliency. Some types of RAID allow you to lose one or disks without losing any data or access to the data. But RAID is only one technique for improving data resiliency.

Another technique is called RAIN. With this concept rather than tolerate the lose of a drive or two in a RAID group without losing data, the storage can tolerate the lose of an entire storage node(s) without losing any data. This is accomplished by replicating data on distributed nodes — keeping multiple copies of data in the file system. However, RAIN usually involves the coupling of the file system with the hardware because the file system has to know where copies of the data are located and the status of the nodes in the file system.

Generically, the coupling of object based storage or object-like storage and RAIN is what is termed Cloud Storage.

Cloud Storage – Ideals and Challenges

With everything, as has been said before, the devil is in the details. To make Cloud Storage function well, some ideals have been incorporated. One of the best articles that discuss cloud storage and the ideals and challenges is here. The four fundamental aspects of Cloud Storage are:

• Ease of Management
• Self-replicating
• Self-healing
• Self-balancing

The most fundamental ideal is that the storage is simple to manage (who wants to manage a PB size storage system with the tools and concepts today?). With Cloud Storage the administrator details policies on how the storage functions. Fundamentally, the admin determines how many copies of each data file or type of data file is in the storage. Then the file system tracks the data to make sure that the policies are fulfilled.

Probably the most fundamental ideal, technically, is that the file system is self-replicating. If a node goes down then the file system has to react by taking that node off-line and then checking what data was located on the node and ensuring that the data is replicated on other nodes based on the policies. This is the idea of using RAIN as previously discussed and goes to the core of how Cloud Storage functions.

The technique of self-healing can include the file system performing checksums of the data checking for data corruption and correcting any corrupt data from the other copies. This can include checking data transfers to ensure that the data tranferred matches the data in the storage.

Another aspect of the storage that is important is self-balancing. Ideally the file system should move data around the storage to minimize hot spots – to balance the storage. This balance can be for performance, although cloud storage is usually thought of as low performing storage, or for capacity. There can also be some heuristics in the file system based on a failure model so that copies are balanced as to their location with the storage (i.e. which nodes they are stored on).

However, with every type of storage, there are challenges in the implementation (i.e. the devil is in the details). These challenges include:

• Security (always an issue and not necessarily a cloud storage specific issue)
• Data integrity (making sure the stored data is “correct”)
• Power (since you have copies you will have extra storage which adds power)
• Replication time and costs (how fast can you replicate data since this can be important to data resiliency)
• Cost (how much extra money do you have to pay to buy the extra storage for copies)
• Reliability

This last issue was discussed in some detail in one of Henry Newman’s articles. While RAID can tolerate the lose of a disk or two, RAIN relies on replication to maintain data resiliency. So how quickly it can replicate data and how the integrity of the data is maintained is key to the usefulness of Cloud Storage.

Two Examples of Cloud Storage

There are several (many?) examples of Cloud Storage but two examples are discussed here as examples of what is in the market. These two examples are Caringo and Parascale.

The concepts for both are remarkably similar. You have a node with some storage, either internal or external, that can be utilized as part of the Cloud Storage as long as they can communicate with one another typically using TCP/IP (while not required many people recommend using a dedicated network for the Cloud Storage system). You either install the software or pop in a USB key and the new node adds itself to the storage pool. You don’t necessarily need RAID – just lots of storage (the so-called “cheap and deep” strategy). The metadata is distributed so that if a node fails there is always another way to get metadata (and real data). Then the admin creates the policies and you can start moving data into/out-of the storage pool.

Accessing the storage is where the two examples is slightly different than other systems. They each have about the same access protocols:

• HTTP
• WebDAV
• FTP
• NFS (in some form)

Notice that neither has a custom client so that various systems can access the file system directly. Rather you have to use one of the previously mentioned protocols to access the data.

Summary

Cloud Storage systems have a great deal of promise. They use a different approach to data resiliency, RAIN – Redundant array of inexpensive nodes, coupled with object based or object-like file systems and data replication (multiple copies of the data), to create a very scalable storage system. They aren’t designed to be high performing file systems but rather extremely scalable, easy to manage storage systems.

As with everything they have their attractive features and their no-so attractive features. This article presents a high-level overview of the main characteristics of the approach. Cloud storage can meet the needs of extreme scalable storage that has fairly low performance requirements (i.e. on-line archiving) by coupling object-based storage concepts with RAIN concepts. But at the same time Cloud Storage faces some challenges including the need to constantly check the data for corruption and repair it (self-healing) and reliability (how many copies of the data provides the uptime that is required?).