Intro to Nested-RAID: RAID-01 and RAID-10

Comparison of RAID-01 and RAID-10

Nested RAID configurations are all about combining standard RAID levels to produce something better in terms of performance or data redundancy (access to data). How you mix the standard RAID levels is the key to producing what you want (just like cooking), but you have to examine the results from all angles to judge which one is better (if, in fact, one is better). Again – just like cooking (it may smell and look good but taste like dirt). So let’s examine RAID-01 and RAID-10 from different angles.

The capacity and storage efficiency are two common aspects to be compared. Both RAID-01 and RAID-10 only use half of the disks in terms of capacity. This is due to the RAID-1 (mirroring) in the RAID configuration. The result is that the capacity and storage efficiency for RAID-01 and RAID-10 are the same.

Another aspect that can be examined is the number of drives that can be lost without losing access to data. In this case, both RAID-01 and RAID-10 are the same. Losing one drive is the worst case for both configurations (i.e. fewest drives lost before losing access to data). But, both configurations can also lose up to n/2 drives before data access is lost, although the pattern of failed drives is unique to each configuration and unlikely to happen in real-life

It is difficult to compare performance for RAID-01 and RAID-10 since in many cases the performance is heavily dependent upon the performance of the RAID controller. Consequently, we can really only objectively say that the performance of the two configuration is about equal. But we can also say the performance of both is far better than the standard RAID levels except for RAID-0 (pure performance), which is one of the goals of Nested RAID configurations.

The last category in comparing RAID-01 and RAID-10 is RAID rebuild. This category covers the number of drives used in the RAID rebuild which affects the amount of time needed to rebuild the array. When a drive fails you want the RAID array to rebuild as quickly as possible because you may be able to tolerate the loss of another drive, but it is more likely that you cannot. So the faster you get the RAID array rebuilt so that it has no failed drives, the more reliable your data access will be.

In the case of RAID-01, you need to access all of the remaining drives in the array to rebuild the loss of a single drive. The reason is that you have a RAID-1 across two sets of drives that are RAID-0. The loss of a single drive means the entire RAID-0 group has failed. The only way to restore the lost data is to copy all of the blocks from the “healthy” part of the RAID-1 array to the “old” drives in the failed RAID-0 as well as the new drive in the failed RAID-0 group. So in the case of the configuration in Figure 2 assuming that drive 2 has failed, you have to read all of the blocks from disks 4, 5, and 6, and write those blocks to drives 1 and 3 which were there before, as well as the new disk 2. This means that every drive inthe RAID group is touched requiring more time for a rebuild. The result is an increased data access vulnerability while the rebuild is happening.

In the case of RAID-10, if a drive fails, the only it’s pair drive is accessed. Using the example in Figure 4, if drive 2, fails, then only drive 1 is accessed to restore the data on the new drive 2. Since you are only having to read the data on a single drive the rebuild time goes much faster.

In addition, since only one drive has to be touched we stay away from the magic number of blocks were we can tickle a URE (Unrecoverable Read Error). So in the case of RAID-01 we run a much greater risk of hitting a URE during the rebuild as compared to RAID-10.

Using Figures 2 and 4 we can do the math on how much data has to be read and how much as to be written during a rebuild.

RAID-01: Amount of data read = 360GB
RAID-10: Amount of data read = 120GB

RAID-01: Amount of data written = 360GB
RAID-10: Amount of data written = 120GB

So for this particular example, RAID-01 has to read and write three times as much data as RAID-10 (although it can read the data a bit faster because of the data striping). If you like, we can make this into simple formulas:

RAID-01: Amount of data read = (n/2) * single drive capacity
RAID-10: Amount of data read = single drive capacity
RAID-01: Amount of data written = (n/2) * single drive capacity
RAID-10: Amount of data written = single drive capacity

So as the number of drives increases and the capacity of a single drive increases, the longer it takes RAID-01 to rebuild. However,
the RAID-10 rebuild time is only affected by the capacity of a single disk not the number of them.

Table 3 below summarizes RAID-01 and RAID-10 from the above discussion.

Table 3 – RAID-01 and RAID-10 Comparison

Linux is Special

While we may be jumping the gun a little by talking about specific implementations of RAID, Linux has some special RAID modes that impact RAID-10 and provide unique configurations.

Remember that Linux has something called software RAID (the md driver which can be managed using mdadm). This is RAID as implemented in software in the Linux kernel using the CPU’s for RAID processing (as opposed to a dedicated RAID card). A few years ago, the “md” and mdadm maintainer, Neil Brown, created some rather unique RAID-10 options for Linux, that, in some cases, can use fewer than 4 drives (the “normal” minimum for RAID-10) and use an odd number of drives. There are two concepts used in RAID-10 for md: “near” and “far”.

In the “near” concepts each data chunk is repeated m times in a k-way stripe. However, it is not required that m evenly divide k as in the normal RAID-10. With two drives and assuming that m is two, the “special” near RAID-10 configuration just boils down to RAID-1.

Disk 1    Disk 2
------    ------
  A1        A1
  A2        A2
  A3        A3
  ..        ..

Notice that we can lose one drive and still retain access to the data just like RAID-1. But the Linux md drive thinks this is RAID-10 (allows for easier upgrading and expansion).

But, one of the cool things about RAID-10 using the md driver in Linux is that we can use an odd number of drives. In the case of three drives the data layout looks like the following.

Disk 1    Disk 2    Disk 3
------    ------    ------
  A1        A1        A2
  A2        A3        A3
  A4        A4        A5
  A5        A6        A6
  ..        ..        ..

Notice that you have 2 copies of each data chunk across the drives just like RAID-10 but the chunks are added in a round-robin fashion. As with RAID-10 we can also tolerate the loss of a single drive without losing access to the data. However, this configuration allows us to use an odd number of drives and get increased capacity. The capacity of this configuration is the following:

Capacity = 2/3 * capacity of single disk

For four disks, the data layout looks just like RAID-10.

Disk 1    Disk 2    Disk 3    Disk 4
------    ------    ------    ------
  A1        A1        A2        A2
  A3        A3        A4        A4
  A5        A5        A6        A6
  A7        A7        A8        A8
  ..        ..        ..        ..

As with RAID-10 we can tolerate the loss of one drive in general, and two drives if they don’t result in the loss of certain data chunks.

The phrase “near” comes from the fact that the mirrored data blocks are “near” to each other. According to Neil Brown, this means that you can get read and write performance that is similar to RAID-0 but with half the number of drives.

The “far” concept arranges the data more like RAID-01 than RAID-10. For a “far” data layout, you split the drives into f sections. All of the data chunks are repeated in each section but are offset by one device putting the mirrored data chunks “further” apart on the drives. Let’s assume that f is 2 meaning that a drive is broken into two parts. A two-drive “far” layout with f=2 would look like the following.

Disk 1    Disk 2
------    ------
  A1        A2
  A3        A4
  A5        A6
  ..        ..
  A2        A1
  A4        A3
  A6        A5
  ..        ..

Notice how the same data chunks reside on different drives so we can lose one drive and still have access to all of the data (just like RAID-1). However, the overall capacity with two drives is still the following.

Capacity = (n/2) * capacity of single disk

where n is the number of drives.

A three-drive configuration with f=2 looks like the following:

Disk 1    Disk 2    Disk 3
------    ------    ------
  A1        A2        A3
  A4        A5        A6
  A7        A8        A9
  ..        ..        ..
  A3        A1        A2
  A6        A4        A5
  A9        A7        A8
  ..        ..        ..

Again, we can lose up to one drive and retain access to all of the data, but we have increased the capacity of RAID array to the following:

Capacity = 2/3 * capacity of single disk

which is the same as in the “near” configuration.

Notice that for the “far” configuration the emphasis is on striping. This means that reads can be striped making them very fast, but writes are somewhat limited because of the head movement on the two drives (mirroring the data). According to this article the performance is,

• Sequential read performance is excellent since it is RAID-0 striped
• Random reads are maybe 10-20% better than conventional RAID-10 due to using the faster outer disk sectors (short stroking) and smaller average seek times
• Sequential and Random writes are about the same performance as RAID-1 (i.e. not that good)

The conclusion is that “far” configurations are good for reading but not so good for writing. However, both “near” and “far” configurations allow you to use fewer than four disks and use an odd number of drives.

Just as Nested RAID allowed you to combine RAID levels, Linux md allows you to combine “near” and “far” RAID configurations. For example, in a m=2 and f=2 configuration with four drives (the minimum allowed) the data layout looks the following:

Disk 1    Disk 2    Disk 3    Disk 4
------    ------    ------    ------
  A1        A1        A2        A2
  A3        A3        A4        A4
  A5        A5        A6        A6
  A7        A7        A8        A8
  ..        ..        ..        ..
  A2        A2        A1        A1
  A4        A4        A3        A3
  A6        A6        A5        A5
  A8        A8        A7        A7
  ..        ..        ..        ..

Notice that there are four copies of each data chunk! Also notice that we can lose two drives without losing access to the data. However, in contrast to the normal RAID-10 we can lose any two drives and still retain access to all of the data. This is much better than RAID-10 where we can lose any single drive, but only specific combinations of two drives. The capacity of this particular four-drive “near” and “far” RAID configuration is the following.

Capacity = (n/2) * capacity of single disk

So we don’t gain any extra capacity relative to RAID-10 but we do gain the ability to lose any drives (a strong plus in my book).

Remember that with RAID-10 you have to have an even number of drives. The RAID-10 md drive allows you to use an odd number of drives. For the same “near” and “far” combination with m=2 and f=2, a five-drive configuration would look like the following:

Disk 1    Disk 2    Disk 3    Disk 4    Disk 5
------    ------    ------    ------    ------
  A1        A1        A2        A2        A3
  A3        A4        A4        A5        A5
  A6        A6        A7        A7        A8
  A8        A9        A9        A10       A10
  ..        ..        ..        ..        ..
  A2        A3        A1        A1        A2
  A5        A5        A3        A4        A4
  A7        A8        A6        A6        A7
  A10       A10       A8        A9        A9
  ..        ..        ..        ..        ..

Again, notice that there are four copies of the data on the arrays but we’ve increased the capacity of the RAID array because we’ve a fifth drive. So we have retained the ability to lose any two drives without losing data (a HUGE plus in my opinion), and we’ve gained capacity which is,

Capacity = 3/5 * capacity of single disk

Summary

Nested RAID was created to combine the performance and/or data redundancy features of the standard RAID levels to give us a wider range of RAID behavior. Moreover, it allows us to improve the data redundancy of standard RAID levels that are focused on performance or to improve the performance of standard RAID levels that are focused on data redundancy.

In this article, we reviewed the basics of Nested RAID and the typical naming convention for them. Don’t forget that the first number to the left is the lowest RAID configuration and then as you move to the right in the RAID name, more RAID levels are added. For example, RAID-01 starts with two groups of RAID-0 and then combines them with RAID-1 (mirroring).

How you combine the standard RAID levels can impact the overall behavior. This means that RAID-01 is not the same as RAID-10. While they are very similar, RAID-10 has better recovery behavior if you lose a drive and should have a faster rebuild time if a disk is lost. So the moral of the story is that RAID X+Y is not the same as RAID Y+X.

Finally we examined the Linux md RAID-10 driver and it’s unique capabilities. It has uses two concepts: “near” and “far” when deciding where to place data chunks on the drives. For both concepts, it allows you to use as few as two drives (normally RAID-10 requires at least four drives) and also allows you to use an odd number of drives (again, something that a normal RAID-10 doesn’t allow) which increases capacity. You can also combine the “near” and “far” concepts using a minimum of four drives, allowing you to lose any two drives in the array without losing data. In contrast, RAID-10 allows you to lose one drive and then certain other drives without losing data (an unlikely scenario).

In the next article in our RAID series, we’ll examine using RAID-5 and RAID-6 in Nested RAID to discover what kind of RAID combinations we can create and how they behave.