ZFSBuild2012 – Still running InfiniBand SRP Benchmarks 32k Random Read

We are still running the IOMeter benchmarks using the new ZFSBuild2012 server and InfiniBand SRP.  The numbers we are seeing with SRP are absolutely awesome.  For example, right now it is running a 32k random read benchmark and it is getting nearly 60,000 IOPS and moving 1854MB/second.  This is amazing because when we ran raw IB performance tests on the network, the max performance of the IB network was 1890MB/second.  The ib_read_bw and ib_write_bw tools showed our IB network moving an average of 1869MB/second and a peak of 1890MB/second.  It is really exciting to see the ZFSBuild2012 box delivering 1854MB/second through IOMeter, which is about 98% of the wirespeed for our IB network.

Thursday, October 11th, 2012 Benchmarks, InfiniBand

7 Comments to ZFSBuild2012 – Still running InfiniBand SRP Benchmarks 32k Random Read

  • ckim44 says:

    wooooow! will you be posting config details on how you got IB SRP working? I hope you do! very very interesting!!! congrats on the speed btw 🙂

  • indigo says:

    Good work and thanks for sharing. Also interested in config details and your experiences once concluded.

  • Yes, we’ll be posting all of the info on how we got things working after we’re done running all the tests.

  • dominicg says:

    Hey! are you available for consulting? Need to size a nexenta supermicro server. Please contact me at

  • schism says:

    Can you explain to me how to know what record size to use when doing bench marks?

    You see 4k used all the time to show off huge IOPS. But what does that really mean in real world use?

    How does one know what their workload will be and therefore what record size to test with?

    I can test with iozone and get tens of thousands of iops when writing with 4k reclen, but I’ll see <5k when doing a 128k reclen. I'm just confused as to how to interpret this and know what I should be using for a reclen and also if I may need to tweak the recordlength within zfs volume itself…

  • twc2013 says:

    Can someone please link me to documentation on how to setup SRP over IB, thanks.

  • We’ve been meaning to get an SRP article posted. We will try to get that done this week.

  • Leave a Reply

    You must be logged in to post a comment.