Tuning CFQ – What Station is That?

This point of this article is not to perform a detailed tuning, but rather to show an example of how you can go about tuning. In particular, this article will change the value of the parameter, quantum which is the number of dispatched requests in the device queue. Increasing the value of this parameter means that the IO scheduler increases the possible number of IO operations executed in the alloted time slice. For this simple tuning example, the default value of quantum is used (quantum=4). It was also increased to 32 so that the impact of increased dispatched requests can be understood.

The goal from tuning (increasing the number of dispatched requests) is to improve throughput. If the application is doing synchronous IO (reads and/or write), then it is presumed that increasing the number of dispatches could improve the throughput.

One of the most important aspects of tuning is to define how you are going to measure performance before and after the tuning. This allows you to understand the impact of varying parameters. In the interest of time, only two benchmarks will be used to measure performance changes – fdtree and IOzone. In addition, in the interest of time, only the ext4 file system will be tested.

As with tests from previous articles each test for fdtree and iozone was run 10 times each for ext4. The test system used a stock CentOS 5.3 distribution but with a 2.6.30 kernel (from kernel.org) and e3fsprogs was upgraded to the latest version as of the writing of this article, 1.41.9. The tests were run on the following system:

• GigaByte MAA78GM-US2H motherboard
  
• An AMD Phenom II X4 920 CPU
  
• 8GB of memory
  
• Linux 2.6.30 kernel
  
• The OS and boot drive are on an IBM DTLA-307020 (20GB drive at Ulta ATA/100)
  
• /home is on a Seagate ST1360827AS
  
• There are two drives for testing. They are Seagate ST3500641AS-RK with 16 MB cache each. These are /dev/sdb and /dev/sdc.

Only the first Seagate drive was used, /dev/sdb, for all of the tests.

Fdtree Results

There was a previous article that used a simple bash script, called fdtree, for measuring some basic metadata performance metrics. fdtree is a simple bash script that performs four different metadata tests:

• Directory creation
  
• File creation
  
• File removal
  
• Directory Removal
  

It creates a specified number of files of a given size (in blocks) in a top-level directory. Then it creates a specified number of subdirectories and then in turn subdirectories are recursively created up to a specified number of levels and are populated with files.

As in the previous article there are four tests for stressing the metadata capability:

• Small files (4 KiB)




• Shallow directory structure


• Deep directory structure




• Larger files (4 MiB)




• Shallow directory structure


• Deep directory structure

The two file sizes, 4 KiB (1 block) and 4 MiB (1,000 blocks) were used to get some feel for a range of performance as a function of the amount of data. The two directory structures were used to stress the metadata in different ways to discover if there is any impact on the metadata performance. The shallow directory structure means that there are many directories but not very many levels down. The deep directory structure means that there are not many directories at a particular level but that there are many levels.

The command lines for the four tests are:
Small Files – Shallow Directory Structure

./fdtree.bash -d 20 -f 40 -s 1 -l 3

This command creates 20 sub-directories from each upper level directory at each level (“-d 20″) and there are 3 levels (“-l 3″). It’s a basic tree structure. This is a total of 8,421 directories. In each directory there are 40 files (“-f 40″) each sized at 1 block (4 KiB) denoted by “-s 1″. This is a total of 336,840 files and 1,347,360 KiB total data.

Small Files – Deep Directory Structure

./fdtree.bash -d 3 -f 4 -s 1 -l 10

This command creates 3 sub-directories from each upper level directory at each level (“-d 3″) and there are 10 levels (“-l 10″). This is a total of 88,573 directories. In each directory there are 4 files each sized at 1 block (4 KiB). This is a total of 354,292 files and 1,417,168 KiB total data.

Medium Files – Shallow Directory Structure

./fdtree.bash -d 17 -f 10 -s 1000 -l 2

This command creates 17 sub-directories from each upper level directory at each level (“-d 17″) and there are 2 levels (“-l 2″). This is a total of 307 directories. In each directory there are 10 files each sized at 1,000 blocks (4 MiB). This is a total of 3,070 files and 12,280,000 KiB total data.

Medium Files – Deep Directory Structure

./fdtree.bash -d 2 -f 2 -s 1000 -l 10

This command creates 2 sub-directories from each upper level directory at each level (“-d 2″) and there are 10 levels (“-l 10″). This is a total of 2,047 directories. In each directory there are 2 files each sized at 1,000 blocks (4 MiB). This is a total of 4,094 files and 16,376,000 KiB total data.

The first combination tested was for small files (4 KiB) with a shallow directory structure. Table 1 below lists the wall clock times for the functions with an average value above and just below it, in red, is the standard deviation.

Table 1 – Benchmark Times Small Files (4 KiB) – Shallow Directory Structure

Our good benchmarking habits tell us that all but the File Create and the File Remove tests don’t really run long enough to be considered in the performance evaluation.

Table 2 below lists the actual performance results with an average value above and just below it, in red, the standard deviation. Recall that we’re only considering the File Create and the File Remove tests.

Table 2 – Performance Results of Small Files (4 KiB) – Shallow Directory Structure

The results are about the same for the two values of quantum (4 and 32). Actually, the increased value of quantum decreases the performance slightly but the values are within the standard deviation so it’s very difficult to say one value of quantum is better than the other.

The second combination tested was for small files (4 KiB) with a deep directory structure. Table 3 below lists the benchmark times with an average value above and just below it, in red, is the standard deviation.

Table 3 – Benchmark Times Small Files (4 KiB) – Deep Directory Structure

All four tests can really be considered for comparison since the average time is fairly large.Table 4 below lists the performance results with an average value above and just below it, in red, the standard deviation.

Table 4 – Performance Results of Small Files (4 KiB) – Deep Directory Structure

The performance result for the two values of quantum are almost the same. But the differences between the two are within the standard deviation of the measures making it difficult to say that one is better than the other.

The third combination tested was for medium files (4 MiB) with a shallow directory structure. Table 5 below lists the benchmark times with an average value above and just below it, in red, is the standard deviation.

Table 5 – Benchmark Times Medium Files (4 MiB) – Shallow Directory Structure

The only measure that can be considered for comparison is File Create because it takes over 60 seconds (our minimum amount of time).
Table 6 below lists the performance results with an average value above and just below it, in red, is the standard deviation.

Table 6 – Performance Results of Medium Files (4 MiB) – Shallow Directory Structure

The differences between the two values of quantum are fairly close but the case of quantum=32 is a little bit faster. The difference between the two is just outside the standard deviation of the two, so we can say that the case of quantum=32 is a bit faster.

The fourth and final combination tested was for medium files (4 MiB) with a deep directory structure. Table 7 below lists the benchmark times with an average value above and just below it, in red, is the standard deviation.

Table 7 – Benchmark Times Medium Files (4 MiB) – Deep Directory Structure

As with the previous case the only measure that has a reasonable amount of testing time is File Create.Table 8 below lists the performance results with an average value above and just below it, in red, is the standard deviation.

Table 8 – Results of Medium Files (4 MiB) – Deep Directory Structure

The values for File Create are about the same for the two values of quantum. However, as with the previous case, the differences are just outside the standard deviation of the two measures. So that means that the case of quantum=32 is a bit faster.

IOzone Results

Fdtree tested metadata performance and IOzone is used to test throughput. As with the previous article the tests that IOzone run are:

• Write
  
• Re-write
  
• Read
  
• Re-read
  
• Random Read
  
• Random Write
  
• Backwards Read
  
• Record Rewrite
  
• Strided Read
  
• Fwrite
  
• Frewrite
  
• Fread
  
• Freread
  

For this article, as with the last, only four record sizes are tested: (1) 1MB, (2) 4MB, (3) 8MB, and (4) 16MB. For a file size of 16GB that results in (1) 16,000 records, (2) 4,000 records, (3) 2,000 records, (4) 1,000 records.

The command line for the first record size (1MB) is,

./IOzone -Rb spreadsheet_output_1M.wks -s 16G -r 1M > output_1M.txt

The command line for the second record size (4MB) is,

./IOzone -Rb spreadsheet_output_4M.wks -s 16G -r 4M > output_4M.txt

The command line for the third record size (8MB) is,

./IOzone -Rb spreadsheet_output_8M.wks -s 16G -r 8M > output_8M.txt

The command line for the fourth record size (16MB) is,

./IOzone -Rb spreadsheet_output_16M.wks -s 16G -r 16M > output_16M.txt

As mentioned previously, there are 13 tests that are each run 10 times and there are 4 record sizes. This makes a total of 520 tests that were run per file system.

Because of the large number of tests that are run, the results are split into two tables. The first table is for the write tests: write, re-write, random write, record re-write, fwrite, frewrite. The second table is for the read tests: read, re-read, random reads, backwards read, strided read, fread, and freread. Each group below has two tables for a specific record size group (1MB, 4MB, 8MB, 16MB). So that means there are 8 tables of results.

The first two tables of IOzone results are for the 1MB record size. Table 9 below presents the throughput in KB/s for the file systems for the 6 write tests.

Table 9 – IOzone Write Performance Results with a Record Length of 1MB and a File Size of 16GB

In comparing the two rows of data we see the following:

• Random write is a little faster for quantum=32 (the difference is outside the standard deviation)
  
• fwrite for quantum=32 is a little slower (the difference is outside the standard deviation)
  
• The standard deviation of re-write is much smaller for quantum=32 (2 orders of magnitude)
  

Table 10 below presents the throughput in KB/s for the file systems for the 7 read tests for a record length of 1MB.

Table 10 – IOzone read Performance Results with a Record Length of 1MB and a File Size of 16GB

In comparing the the two results, the only difference is that fread for quantum=32 is a little slower (the difference is outside the standard deviation).

The next two tables of results are for the 4MB record size. Table 11 below presents the throughput in KB/s for the file systems for the 6 write tests.

Table 11 – IOzone Write Performance Results with a Record Length of 4MB and a File Size of 16GB

In comparing the two results, the following observations are made:

• Re-write for quantum=32 is faster (the difference is outside the standard deviation)
  
• Random write for quantum=32 is much faster – about 12% (this is larger than the standard deviation so it’s a valid comparison)
  
• The standard deviation for re-write is two orders of magnitude smaller for quantum=32
  

Table 12 below presents the throughput in KB/s for the file systems for the 7 read tests for a record length of 4MB.

Table 12 – IOzone Read Performance Results with a Record Length of 4MB and a File Size of 16GB

In comparison the results seem to be about the same (within the standard deviations) for both quantum=4 and quantum=32.

The next two tables of results are for the 8MB record size. Table 13 below presents the throughput in KB/s for the file systems for the 6 write tests.

Table 13 – IOzone Write Performance Results with a Record Length of 8MB and a File Size of 16GB

• Re-write is faster for quantum=32 but the difference is very close to the standard deviation.
  
• The standard deviation for rewrite for quantum=32 is two orders of magnitude smaller for quantum=32 than quantum=4
  
• The standard deviation for frewrite is much smaller for quantum=32 than quantum=4
  

Table 14 below presents the throughput in KB/s for the file systems for the 7 read tests for a record length of 8MB.

Table 14 – IOzone Read Performance Results with a Record Length of 8MB and a File Size of 16GB

The results for quantum=32 and quantum=4 are about the same (i.e. the differences are smaller than the standard deviations).

The final two tables of results are for the 16MB record size. Table 15 below presents the throughput in KB/s for the file systems for the 6 write tests.

Table 15 – IOzone Write Performance Results with a Record Length of 16MB and a File Size of 16GB

• Re-write for quantum=32 is faster than quantum=4 but the differences are somewhat close to the standard deviation of quantum=4
  
• Random write is faster for quantum=32 than than the default
  
• The frewrite performance is faster for quantum=32 than quantum=4 but the differences are somewhat close to the standard deviation of quantum=4
  
• The standard deviation of re-write is two orders of magnitude smaller for quantum=32
  
• The standard deviation of frewrite is two orders of magnitude smaller for quantum=32
  

Table 16 below presents the throughput in KB/s for the file systems for the 7 read tests for a record length of 16MB.

Table 16 – IOzone Read Performance Results with a Record Length of 16MB and a File Size of 16GB

The performance for quantum=32 and quantum=4 are about the same.

Summary of Results

The metadata performance using fdtree didn’t change too much when the CFQ parameter, quantum, was changed from 4 to 32 (maximum number of dispatches). For the larger file case (4 MiB), quantum=32 was slightly faster, but not appreciably.

In the case of throughput performance as measured by the specific IOzone tests, the read performance between quantum=4 (default) and quantum=32 is about the same (i.e. within the standard deviation). However there is a bit more of an impact on write performance.

• Re-write is a bit faster for quantum=32 than the default of quantum=4
• Random write is faster for quantum=32 than the default. At a record size of 4MB, it is 12% faster.
• In some cases, frewrite is a little bit faster for quantum=32 but it’s not consistent across the record sizes tested.
• One of the most pronounced changes is in the standard deviation. For the re-write test, the case of quantum=32 the standard deviation in performance is two orders of magnitude smaller than the default case.
• While not as pronounced, the standard deviation for frewrite decreases as the record size increases for quantum=32 to the point where at a record size of 16MB it is two orders of magnitude smaller than the default case.

Overall, there was not as much change in the performance for an increase in quantum from 4 to 32. However, this is just a simple experiment to show what you can do in tuning CFQ and to illustrate that it’s not always easy because you have to be sure to run your performance tests before and after the changes in the IO scheduler. In addition, there are a number of parameters in CFQ that can be changed to affect performance. So changing one parameter at a time may not produce the performance change you want.