ZFSBuild2012 – Write Back Cache Performance

Nexenta includes an option to enable or disable Write Back Cache on shared ZVols. To manage this setting, you must first create your ZVol and then set the ZVol to Shared. Then you can select the ZVol and edit the Write Back Caching setting. The purpose of this article is to find out how much performance is affected by the setting. All benchmarks were run on the ZFSBuild2012 hardware using Nexenta 3.1.3. All tests were run on the same ZFS storage pool. Click here to read about benchmark methods.

iSCSI-WC-E is iSCSI using IPoIB with connected mode disabled and the Write Back Cache enabled.

iSCSI-WC-D is iSCSI using IPoIB with connected mode disabled and the Write Back Cache disabled.

IB-SRP-WC-E is InfiniBand SRP with the Write Back Cache enabled.

IB-SRP-WC-D is InfiniBand SRP with the Write Back Cache disabled.

Generally speaking, enabling the Write Back Cache has no significant impact on read performance, but a huge improvement for write performance.


IOMeter 4k Benchmarks:
IOMeter 4k Benchmarks

IOMeter 4k Benchmarks

IOMeter 4k Benchmarks

IOMeter 4k Benchmarks

IOMeter 8k Benchmarks:
IOMeter 8k Benchmarks

IOMeter 8k Benchmarks

IOMeter 8k Benchmarks

IOMeter 8k Benchmarks

IOMeter 16k Benchmarks:
IOMeter 16k Benchmarks

IOMeter 16k Benchmarks

IOMeter 16k Benchmarks

IOMeter 16k Benchmarks

IOMeter 32k Benchmarks:
IOMeter 32k Benchmarks

IOMeter 32k Benchmarks

IOMeter 32k Benchmarks

IOMeter 32k Benchmarks

Monday, December 17th, 2012 Benchmarks

13 Comments to ZFSBuild2012 – Write Back Cache Performance

  • toronar says:

    Isn’t that dangerous in case of an unplanned power loss or similar problems?
    I did read that to increase write performance you should use a fast drive for ZIL, so that data is committed to stable storage fast instead of being buffered in RAM.

    Or are you comfortable that an UPS is good enough for that job?

  • Depending upon your application, yes that can be dangerous. If you’re handling financial transactions on a heavily loaded Oracle database, I’d want you to be using ZeusRAM drives for your ZIL. With Windows systems, it’s typically been my experience that applications do not flush the disk cache on every access. While this is possibly an issue if you are writing heavily to disk and you lose power immediately, we have enough battery backup to ensure that the data can finish writing to disk. The scenario that I would be a bit more concerned about would be a kernel panic, where some in-flight data could be corrupted regardless of power problems.

    For our business needs however, having suitable battery backup to allow the system to finish in-flight transactions is plenty.

  • Jae-Hoon.Choi says:

    If you did enable WB cache then ZIL don’t use ZIL. Nexenta also too!

    But NFS shared folder use ZIL with WB cache enabled environment.

    Why did you attached ZIL your Zvol?

  • rk says:

    Hi Matt,

    Did you ever get the ARP issue you blogged about in 2010 resolved, or are you still using a CentOS gateway in the new environment?

    I’m working on specing out a similar solution and would love to get in touch to ask a few questions and share experiences. If you’d reach out to me at richardk[aht]leapbeyond(doht)com I’d be grateful.

  • On the ZFSBuild2010 system, we never got it figured out. On ZFSBuild2012, we have not encountered that error. We have upgraded from InfinihostIII cards to ConnectX cards, updated lots of firmware, and the OFED stack has improved significantly since then. Between all of those changes, we have not seen any of the heartache that we saw 2 years ago with InfiniBand.

  • rk says:

    Thanks Matt. Any chance on getting in touch for a few more questions? Willing to pay you to consult, if it helps.

  • Absolutely – connect with me matt.breitbach on skype.

  • david says:

    @Jae-Hoon.Choi

    Thats not correct, with writeback cache enabled, the iSCSI target will still respect sync commands. If an async transfer is done it will go to RAM if its sync it will go to ZIL, if an async is in RAM and the client requests a sync it will be written to disc. If you disable the writeback cache, then all write will be sync, just like setting the ZIL settings to always on the zvol. You can think of the writeback cache as being the same as that on an actual disc. Correctly implemented and well behaved discs will respect the sync requests, which is what the iSCSI / COMSTAR target does.
    With benchmarking, for example iometer, if you have more than 1 outstanding io it will write as async, so will not go to the ZIL (if running on windows, it is always sync on linux, and doesnt matter what the outstanding io is, will also be 1, as it isnt using the correct libraries).

  • fruitfruit says:

    Thanks for the benchmarks, they are very helpful and save us months of work and potential issues.

    Some of the results are hard to explain. I was expecting disabling writeback cache would not impact read performance at all. Unless I’m misinterpreting the graphs, the read performance seems to be fairly drastically impacted. For example, take the 8k random read test (assuming it was 100% random reads), with WC disabled the read IOPS are nearly half!

    Why is WB cache status impacting read performance at all (especially random read?) Is nexenta making other configuration changes beyond just disabling WB cache? (disabling pre-fetching etc.? even 8k sequential reads are half the performance)

    Or is it that zfs has implemented this feature in a suboptimal fashion? i.e. just because writeback was disabled doesn’t mean the writes cannot be cached in memory from an MRU perspective. You can cache it in memory in addition to sync’ing to disk. This would give the read cache locality of time, and is not particularly a ‘smart’ thing, it’s more or less expected from enterprise arrays.

    puzzled.. this is a pretty important test case for us. We were prepared for a hit on write performance, but reads is a bit of a blow and will increase the cost of our build. So any insights would be appreciated.

  • fruitfruit says:

    As clarification to the previous question? Did you do the write tests just before the read tests? Even this shouldn’t have a made a difference in the 8k random read test..?

  • You can download the IOmeter config file from here : http://www.zfsbuild.com/pics/Graphs/Iometer-config-file.zip

    This will configure your IOmeter test to run exactly the way that the tests were run. If you do not wish to open the IOmeter config, the tests were run in the exact order that the results are displayed.

  • stoatwblr says:

    @fruitfruit

    “I was expecting disabling writeback cache would not impact read performance at all. Unless I’m misinterpreting the graphs, the read performance seems to be fairly drastically impacted. For example, take the 8k random read test (assuming it was 100% random reads), with WC disabled the read IOPS are nearly half! ”

    You’re neglecting atime updates. Even when you’re “only” reading, atime gets updated, so every read comes with a write (unless you disable atime).

  • fruitfruit says:

    atime updates on zvols? atime shouldn’t have much of an impact in these test cases, it could have a potential impact in the NFS case.

    Am i mistaken?

  • Leave a Reply

    You must be logged in to post a comment.