02 Oct 2009 17:39
TAGS: benchmark filesystems ssd
As I promised I benchmarked some of the Linux filesystems on my solid state disk.
Introduction
I wanted to benchmark the following filesystems:
- ext2, ext3
- ext4
- xfs
- reiserfs, reiser4
- nilfs
- btrfs
- zfs (via fuse)
NILFS2 haven't managed to even finish the Bonnie++ test. This means this filesystem is not yet ready to use (but promises very nice features). Other filesystem that has not been benchmarked is reiser4, because the Ubuntu kernel doesn't have support for it. I would need to patch it and I wasn't happy about it.
Images shown here show results of the standard Bonnie++ tests. Command to do them was:
bonnie -d /dir/on/ssd/partition -n 200:200
The -n parameter was tuned so that for each test some values were returned. With default setting I got many values "++++" indicating test was performed so fast, that Bonnie++ was not able to calculate the performance.
Explanation of test names can be found in bonnie++ documentation.
Before each test, filesystem was created on the prepared partition (25 GB) and some (the same for each test) data was copied to it (about 10 GB) to simulate "used" filesystem.
As it appeared I was not able to disable write-caching with running hdparm -W0 /dev/sda. Instead it stated
/dev/sda:
setting drive write-caching to 0 (off)
write-caching = 1 (on)
Possible write-caching is good thing though (and by default enabled), so I have no problem with that.
All tests were run twice, but the results was nearly the same, so I just removed the second results for each filesystem.
For each test, bigger is better with value being thousands operations per second.
The best filesystem
As some suggest, the preferred I/O-scheduler for SSD disk is "noop", which means there's no IO scheduling in kernel, so we rely on scheduling logic in the hardware (which for various reasons is believed to be good in SSD disks) and profit from no software overhead of queuing.
Let's then compare how well filesystems perform with this scheduler chosen:
This benchmark was performed for all filesystems but NILFS2 and reiser4.
Random seeks
When it comes to random seeks (very important for low-latency systems), the best is ext4 with reiserfs and xfs having almost the same result. Btrfs is next (10% slower), then ext3, ext2 and zfs at the end being 6 time worse than the best.
Creation and deletion of files
Ext4 is the fastest in creating files (both sequentially and randomly) while btrfs is the fastest in deleting files, which small exception of ext2 being 7 times faster than everything in sequentially deleting files. On the other hand it's ability to delete files in random fashion is pretty bad. Comparing only btrfs and ext4, both are fast, the difference is about 10% to the one or the other side. Ext3 performs pretty well in this test, reiserfs reaches about half the performance of ext4/btrfs, while xfs and zfs are really slow.
Read/write
Reading and writing of data is pretty equal through filesystems in terms of benchmark results. The worst results has zfs and ext2 (especially in random read, which is vital in modern use of computers).
As per-character reading/writing is not so important, let's concentrate on the rest of tests. As you can see, btrfs is clearly the best, having first place in 3/5 tests being really close to first in the following two. Reiserfs and ext4 perform really well in this area.
Semi-summary
At this point it's clear that when it comes to performance on SSD disk we have two filesystems to consider the best: btrfs and ext4. Reiserfs is slightly worse, but the most mature from them. Xfs and ext3 are just OK, but don't perform equally good in all tests, while ext2 and zfs (on fuse) being not an option at all.
Let's then compare the features of the three:
| Limits | reiserfs | ext4 | btrfs |
|---|---|---|---|
| file name | 4 KB | 256 B | 255 B |
| max file size | 8 TB | 16 GB to 16 TB (depends on block size) |
16 EB |
| max volume size | 16 TB | 1 EB | 16 EB |
| Features | reiserfs | ext4 | btrfs |
| checksum (error check) | no | yes | yes |
| snapshots (like time machine) | no | no | yes |
| mirroring/stripping on FS layer | no | no | yes |
| compression | no | no | yes |
So it seems, btrfs is full of new features compared to (old) reiserfs and (new) ext4 with only small performance penalty in some areas, while even being faster in some.
The rest tests presented only cover ext4, reiserfs and btrfs filesystems.
What is the best scheduler, then?
Having chosen the best filesystems, let's see, which scheduler works best.
ext4
Comparison of schedulers performance for ext4 filesystem:
As we see, cfq is the best in 5 tests, significantly worse in two (random creation of files and random seeks) and nearly as good as best in the rest. Deadline and noop perform pretty the same (with noop being better at creating files randomly and deadline being better at creating files sequentially).
reiserfs
Schedulers performance for reiserfs filesystem:
For reiserfs, again, cfq does its work really well with only random reads being significantly slower than deadline and noop schedulers.
btrfs
Let's now see what scheduler will be best for btrfs filesystem:
This time not cfq, but deadline scheduler is the man! In random seeks, where cfq is generally worse, this time it's worse by about 30% than the best: deadline scheduler. Only in one test deadline is worse than noop, but this is only slight difference. Cfq is only slightly better in 4 tests, but in the rest, deadline is better.
Ultimate comparison of btrfs, reiserfs and ext4
When we know which scheduler will run best with certain filesystems, let's compare Bonnie++ results for the perfect tandems:
- btrfs with deadline scheduler on underlying disk
- ext4 with cfq
- reiserfs with cfq
Random seeks
Choosing cfq scheduler on any of the tested filesystems degraded random seek performance. This is why btrfs with deadline scheduler is better than its competitors.
Creation and deletion of files
This time, without ext2 having so high bar, you can see the differences in creating and deleting files on the three filesystems I tested. Btrfs is much (about 2 times) faster than ext4 in deleting files, while ext4 is a bit faster in creating files randomly, and significantly faster (about 50%) in creating files sequentially. Reiserfs is about 2 times slower than the slower filesystem in each test.
Read/write
In read/write tests btrfs, ext4 and reiserfs perform almost equally well, with btrfs being slightly better than the latter two filesystems.
Summary
Ext4 and btrfs filesystems perform really well on SSD disks making the users really happy about the speed they get from normal computer use.
With ext4 being the default filesystem for Ubuntu 9.10, having SSD disk, you'll notice that it boots really fast:
- from Grub to GDM in 8 seconds
- from GDM to GNOME in 5 seconds
- OpenOffice launches the first time in 2-3 seconds (second time is 0 seconds)
With btrfs being equally fast (or even faster) than ext4 it's amazing what features it delivers:
- snapshots (you can make a snapshot of filesystem and then roll back to it, or just explore historical version files)
- compression
- mirroring/stripping — things usually done on block-device level now incorporated into filesystem
- nice internal structures and algorithms (copy on write, B-trees, …)
- integrated volume management
On the second hand, btrfs as of version 0.19 has still experimental disk format, which means it can be non-compatible with future kernels, but usually kernel developers create code that either preserve the old format or converts the filesystem to new one on the first mount (and then it's not possible to mount such a partition from an older kernel). Also if I understand well, the biggest changes from 0.18 to 1.0 were just applied in 0.19, so probably this will be the final format of btrfs partitions.
It's clear, that btrfs is the Linux answer to Sun's ZFS, which due to incompatible license can't be incorporated into the kernel (this is why we only have FUSE port available).
Having all this said, it's time for me to migrate my /home (and maybe the system partition too) to btrfs!
You can download raw benchmark data in ODS format.
Thanks for publishing these results.
I noticed that btrfs has an SSD mode now:
https://btrfs.wiki.kernel.org/index.php/FAQ#Is_Btrfs_optimized_for_SSD.3F
Did you use one of the SSD options when you mounted the btrfs partition for these tests?
@Shane, yes btrfs detected it ran on SSD disk and enabled ssd mode (from some version you don't need to pass -o ssd).
Piotr Gabryjeluk
visit my blog
hello my friend. i have a white paper for linux fs on ssd. do you mind if you told me how can i do instalation for bonnie++ on ubuntu 12.04 and wich is the source code for for these experiment, please ????
Sorry, but if you need to ask those questions, you probably won't understand the results.
Piotr Gabryjeluk
visit my blog
I just want to need how you realised this work and this results, please!!!
Can you told me how you realised this work please please !!!
Best regards!
Post preview:
Close preview