ZFSBuild2012 – Benchmark Methods

We took great care when setting up and running benchmarks to be sure we were gathering data that could be used to make useful comparisons.  We wanted to be able to compare the ZFSBuild2012 design with the ZFSBuild2010 design.  We also wanted to be able to compare various configuration options within the ZFSBuild2012 design, so we could make educated decisions about how to configure a variety of options to get the most performance out of the design.  The purpose of this page is to share all of our benchmarking methods.

We used the same blade server that we used for the ZFSBuild2010 testing back in 2010.  The specs for that blade are available at http://www.zfsbuild.com/2010/05/05/test-blade-configuration/ .  Note, we did not merely use a blade with the same specs.  We literally removed that exact blade server from production and reassigned it back to our test lab.  We did install software updates (OS, drivers, OFED, etc) and a firmware update in the InfiniBand card.  We did not make any hardware changes to the test blade server since we run the benchmarks in 2010.  The test blade server was running Windows 2008R2, just like it was for the tests back in 2010.

All of the benchmarks were run on the test blade server using IOMeter.  We used the same IOMeter configuration file that we ran back in 2010.  That configuration file is available at http://www.zfsbuild.com/pics/Graphs/Iometer-config-file.zip .  The only change we made in that file was the disk target (replacing “D:New Volume” with the correct path).

The benchmarks are run over the wire.  IOMeter runs on the Windows 2008R2 test blade server, and connects over the network to the ZFSBuild2012 server.  In every test, the test blade and the ZFSBuild2012 server are plugged into the same network switches.  In some cases this is 1Gbps Ethernet.  In others, it is 20Gbps InfiniBand.

We did not use multipathing during any of the benchmarks.  In fact, we took it a step further and disabled every NIC that we were not using, so there was no chance packets would go down the wrong path.

All of our testing focuses on the SAN capabilities, not the NAS capabilities.  This means we are testing iSCSI or SRP instead of NFS or CFS.  We realize that some people are interested in seeing the performance of a NAS configuration, but that was beyond the scope of our tests.

All of our tests were on a RAID10 style ZFS pool.  We created a ZFS pool called “BigDog”.  The BigDog pool contained 18 1TB SAS drives arranged into 9 mirrored pairs, two 1TB SAS drives as hot spares, two Intel 520 series 240GB SSD drives as L2ARC, and two Intel 313 series 20GB SSD drives as mirrored ZIL drives.  All of our tests were run using ZVols created in this “BigDog” ZFS pool.  When we switched to different storage appliance software, we imported the entire “BigDog” ZFS pool rather than reconfiguring it from scratch each time.  This way we could test the exact same ZFS pool with multiple storage appliance software packages using the exact same hardware.  When switching between storage appliance software, we only changed the contents of the boot drives (the two internal Intel 330 series 60GB SSD drives).

We deleted the IOMeter test file in between each round of benchmarks, and rebooted the test blade server and ZFSBuild2012 box.  We did this to make sure caching from previous benchmark runs did not impact the next benchmark run.

The software versions we used during testing were Nexenta 3.1.3, ZFSGuru 0.2.0-beta 7 (FreeBSD 9.1), and FreeNAS 8.3 (FreeBSD 8.3).  We realize that newer versions have been released since we ran our benchmarks.  We will not be re-running any of the benchmarks at this time, because we have already placed the ZFSBuild2012 server into production.  The ZFSBuild2012 server has spent more than a month as a high performance InfiniBand SRP target for one of our web hosting clusters.

Similarly, the ZFSBuild2010 is also in production, so we could not re-run any benchmarks using the old ZFSBuild2010 box with newer Nexenta builds.  Any references to the performance of the ZFSBuild2010 design will be based on the hardware and software in the ZFSBuild2010 box when the original articles were written back in 2010.

If you have any questions about benchmark methods for the ZFSBuild2012 project, please ask them here.  We will be happy to add more information to this page to better explain our testing methods.

Friday, December 14th, 2012 Benchmarks

9 Comments to ZFSBuild2012 – Benchmark Methods

  • […] hardware designs are running Nexenta with write back caching enabled for the iSCSI shared ZVol.  Click here to read about benchmark methods.  We will be posting InfiniBand benchmarks with ZFSBuild2012 soon, and those InfiniBand benchmarks […]

  • dimms says:

    Thank you for your blog and all the useful and interesting information.

    Let me ask you what’s the purpose of ZIL devices if you’re doing only iSCSI and Infiniband?
    the ZIL is not even being touched if doing IO to zvol using iSCSI. If, while doing IO to iSCSI LUN, you monitor all drives using iostat -xn 5, you’ll see that there’s no IO to the ZIL devices. That is is always the behavior in my case.

  • […] server with Nexenta 3.1.3. Write back caching was enabled for the shared ZVol in each test. Click here to read about benchmark methods.  The 1Gbps performance of the ZFSBuild2012 is included as a point of reference within these […]

  • […] Nexenta includes an option to enable or disable Write Back Cache on shared ZVols. To manage this setting, you must first create your ZVol and then set the ZVol to Shared. Then you can select the ZVol and edit the Write Back Caching setting. The purpose of this article is to find out how much performance is affected by the setting. All benchmarks were run on the ZFSBuild2012 hardware using Nexenta 3.1.3. All tests were run on the same ZFS storage pool using exactly the same hardware. Click here to read about benchmark methods. […]

  • […] The software versions we used during testing were Nexenta 3.1.3, ZFSGuru 0.2.0-beta 7 (FreeBSD 9.1), and FreeNAS 8.3 (FreeBSD 8.3).  Click here to read about benchmark methods. […]

  • virtualstorage says:

    This is a great post.. Few questions I have are:
    1) Iometer config file shows that tests are run for 3 minutes. Isn’t it true that lot of vendors see issuues where iops decrease substantially over time and hence iometer should be run for few hours?

    2) One other post mentions about getting 100,000 iops with this hardware setup. I couldn’t see any graph which represents performance of 100000 iops.

  • 1 – IOmeter runs for hours and hours to complete the entire suite of tests. This simulates IO at several different queue depths. Each “test” runs for 3 minutes per queue depth, and runs at 9 different queue depths to get an accurate picture of the entire queue depth profile. Overall, this is ~27 minutes per IO profile, and with 16 different IO profiles, this benchmark suite runs for nearly 8 hours before completing.

    2 – This post specifically does not contain over 100,000 IOP tests, but if you review the benchmarks from our Infiniband performance post, you’ll see some tests that are well in excess of 100,000 IOPS. http://www.zfsbuild.com/2012/12/15/zfsbuild2012-infiniband-performance/#more-632

  • virtualstorage says:

    Thanks again Matt,

    Another set of questions are:
    1)Have you also tested without L2ARC and ZIL and with minimum RAM in ZFS?
    This will validate the theory that ” With ZFS , you don’t get iops based on per disk iops + workload + raid penalties +frontend iops, But you get overall iops based on number of vdev’s (raid groups) X slowest disk iops in pool “.

    2) Also, Have you tested performance with RAIDz1 OR RAIDz2?.

    3) If my L2ARC size is very large so that it can accommodate all working set sizes and ZIL is sufficiently large ( 2 x5 X speed of SSD), Theoretically, I should be able to get even 100000 iops with around just 10 mirrored SAS 15 K rpm disks? This won’t serve capacity requirements but will meet performance requirements.

  • 1 – We have not tested this at all. ZFSBuild2010 has been retired and the workload moved to ZFSBuild2012, and we are planning on using it as a testing platform (albeit with low memory and SATA drives) and that may be something that we can test.

    2 – We have tested it a little bit, but we have not run the full test suite against it. Again, possibly on the ZFSBuild2010 system.

    3 – If your working set fits inside of the ARC and the L2ARC, getting that kind of performance is not out of the question. The working set size for our tests is (I believe) 10GB. This is why performance is so spectacular in our benchmarks since it’s really only hitting RAM. You would only need 1 spindle to actually hold that data and you’d still see 100,000 IOPS.

  • Leave a Reply

    You must be logged in to post a comment.