Previously, we examined the impact of journal size using a separate disk on metadata performance as measured by fdtree. In this follow-up we repeat the same test but use a ramdisk for the journal, thereby boosting the best performance. Or does it?
Appendix – Table of Results
The first combination tested was for small files (4 KiB) with a shallow directory structure. Table 1 below lists the results with an average value and just below it, in red, is the standard deviation.
Table 1 – Benchmark Times Small Files (4 KiB) – Shallow Directory Structure
| Journal Size |
Directory Create
(secs.) |
File Create
(secs.) |
File Remove
(secs.) |
Directory Remove
(secs.) |
| 16MB |
26.30
0.46 |
351.10
3.33 |
70.50
1.02 |
15.70
0.64 |
| 64MB |
15.10
0.94 |
339.40
2.29 |
67.70
0.46 |
3.90
0.3 |
| 256MB |
9.60
0.66 |
330.20
2.40 |
52.70
1.00 |
2.00
1.34 |
| 1GB |
9.30
0.46 |
328.50
1.20 |
53.30
1.19 |
2.70
2.93 |
Table 2 below lists the performance results with an average value and just below it, in red, is the standard deviation.
Table 2 – Performance Results of Small Files (4 KiB) – Shallow Directory Structure
| Journal Size |
Directory Create
(Dirs/sec) |
File Create
(Files/sec) |
File Create
(KiB/sec) |
File Remove
(Files/sec) |
Directory Remove
(Dirs/sec) |
| 16MB |
319.40
5.50 |
959.00
9.25 |
3,837.20
36.44 |
4,778.60
69.39 |
536.90
21.62 |
| 64MB |
559.60
37.25 |
991.80
6.87 |
3,969.70
27.03 |
4,975.20
33.91 |
2,175.20
210.60 |
| 256MB |
880.80
58.41 |
1,019.50
7.34 |
4,080.10
29.51 |
6,093.30
3,026.97 |
5,852.40
2,692.57 |
| 1GB |
907.10
42.62 |
1,024.70
3.87 |
4,101.00
15.03 |
6,322.20
140.45 |
6,093.30
3,026.97 |
The next test was for small files (4 KiB) with a deep directory structure. Table 3 below lists the benchmark times with an average value and just below it, in red, is the standard deviation.
Table 3 – Benchmark Times Small Files (4 KiB) – Deep Directory Structure
| Journal Size |
Directory Create
(secs.) |
File Create
(secs.) |
File Remove
(secs.) |
Directory Remove
(secs.) |
| 16MB |
312.40
3.56 |
624.80
4.66 |
333.00
3.07 |
253.00
25.78 |
| 64MB |
230.90
2.81 |
518.10
7.67 |
243.40
5.00 |
82.40
3.38 |
| 256MB |
145.30
5.02 |
413.00
16.25 |
169.00
9.69 |
52.30
3.55 |
| 1GB |
120.10
7.23 |
400.60
13.35 |
136.60
6.81 |
52.50
17.20 |
Table 4 below lists the performance results with an average value and just below it, in red, is the standard deviation.
Table 4 – Performance Results of Small Files (4 KiB) – Deep Directory Structure
| Journal Size |
Directory Create
(Dirs/sec) |
File Create
(Files/sec) |
File Create
(KiB/sec) |
File Remove
(Files/sec) |
Directory Remove
(Dirs/sec) |
| 16MB |
282.80
2.99 |
566.60
4.25 |
2,267.80
17.08 |
1,063.60
9.62 |
362.60
3.95 |
| 64MB |
383.20
4.77 |
683.40
10.03 |
2,735.40
40.14 |
1,455.70
30.04 |
1,076.50
45.00 |
| 256MB |
597.90
14.59 |
832.10
30.30 |
3,330.60
121.08 |
2,091.90
110.24 |
1,639.70
102.91 |
| 1GB |
609.80
21.67 |
858.60
32.71 |
3,435.90
130.96 |
2,102.80
119.26 |
1,701.20
119.50 |
The third combination tested was for medium files (4 MiB) with a shallow directory structure. Table 5 below lists the benchmark times with an average value and just below it, in red, is the standard deviation.
Table 5 – Benchmark Times Medium Files (4 MiB) – Shallow Directory Structure
| Journal Size |
Directory Create
(secs.) |
File Create
(secs.) |
File Remove
(secs.) |
Directory Remove
(secs.) |
| 16MB |
0.40
0.49 |
153.40
2.06 |
13.20
3.25 |
0.10
0.30 |
| 64MB |
0.30
0.46 |
155.00
2.65 |
11.70
3.82 |
0.00
0.00 |
| 256MB |
0.10
0.30 |
155.50
2.33 |
10.70
2.28 |
0.10
0.30 |
| 1GB |
0.30
0.46 |
157.10
1.70 |
10.50
2.87 |
0.10
0.30 |
Table 6 below lists the performance results with an average value and just below it, in red, is the standard deviation.
Table 6 – Performance Results of Medium Files (4 MiB) – Shallow Directory Structure
| Journal Size |
Directory Create
(Dirs/sec) |
File Create
(Files/sec) |
File Create
(KiB/sec) |
File Remove
(Files/sec) |
Directory Remove
(Dirs/sec) |
| 16MB |
122.80
150.40 |
19.80
0.40 |
80,065.70
1,047.94 |
252.30
84.27 |
30.70
92.10 |
| 64MB |
92.10
140.69 |
19.20
0.40 |
79,239.30
1,336.94 |
290.50
88.78 |
0.00
0.00 |
| 256MB |
30.70
92.10 |
19.30
0.46 |
78,988.20
1,184.63 |
298.00
54.11 |
30.70
92.10 |
| 1GB |
92.10
140.69 |
19.00
0.00 |
78,175.30
845.26 |
322.90
118.09 |
30.70
92.10 |
The fourth and final combination tested was for medium files (4 MiB) with a deep directory structure. Table 7 below lists the benchmark times with an average value and just below it, in red, is the standard deviation.
Table 7 – Benchmark Times Medium Files (4 MiB) – Deep Directory Structure
| Journal Size |
Directory Create
(secs.) |
File Create
(secs.) |
File Remove
(secs.) |
Directory Remove
(secs.) |
| 16MB |
5.50
0.50 |
225.90
1.51 |
14.90
3.86 |
2.40
0.49 |
| 64MB |
2.60
0.49 |
221.40
2.01 |
12.40
1.85 |
2.00
1.00 |
| 256MB |
2.60
0.66 |
221.50
1.<63/font> |
14.80
4.28 |
2.40
1.20 |
| 1GB |
2.70
0.64 |
222.90
2.02 |
10.90
2.17 |
2.60
1.80 |
Table 8 below lists the performance results with an average value and just below it, in red, is the standard deviation.
Table 8 – Results of Medium Files (4 MiB) – Deep Directory Structure
| Journal Size |
Directory Create
(Dirs/sec) |
File Create
(Files/sec) |
File Create
(KiB/sec) |
File Remove
(Files/sec) |
Directory Remove
(Dirs/sec) |
| 16MB |
375.00
34.00 |
18.00
0.45 |
72,495.00
485.26 |
296.10
87.63 |
886.60
167.06 |
| 64MB |
818.40
167.06 |
18.00
0.00 |
73,971.30
672.82 |
336.80
48.63 |
1,313.20
620.23 |
| 256MB |
835.40
193.73 |
18.00
0.00 |
73,935.70
541.31 |
306.20
106.08 |
1,132.40
621.95 |
| 1GB |
801.30
187.63 |
17.90
0.30 |
73,473.30
658.96 |
388.10
67.04 |
1,286.00
767.33 |
Jeff Layton is an Enterprise Technologist for HPC at Dell. He can be found lounging around at a nearby Frys enjoying the coffee and waiting for sales (but never during working hours).
Comments on "Size Can Matter: Ramdisk Journal Metadata Performance – Part 2"
I tried creating ram drives on a 3TB ext4 File System.
pvcreate /dev/ram[0-20]
vgcreate jrnlvg /dev/ram[0-20]
lvcreate -L1G -n lv0 jrnlvg
mkfs -O journal_dev /dev/jrnlvg/lv0
tune2fs -L /opt journal device /dev/jrnlvg/lv0
tune2fs -j -J device=/dev/jrnlvg/lv0 /dev/myTBvg/lv10
this all works great
the problem is when the system reboots, you must go through the whole process and 1+TB file systems regenerating their journals takes forever.
Q: how can I dd or copy the ram device data to a regular disk.
so when the system reboot, you setup the ram devices then copy the old jourabck before you fire up the FS?