Archive for October, 2012
We are still running benchmarks for the ZFSBuild2012 SAN building project. We completed all of the Nexenta benchmarks this past week. We ran every combination of networking configurations to deeply test both 1Gbps Ethernet and 20Gbps InfiniBand on the Nexenta platform. We completed ZFSGuru (with FreeBSD 9.1) benchmarks this morning (both 1Gbps Ethernet and 20Gbps InfiniBand). We are currently running FreeNAS 8.3 benchmarks.
We are still running benchmarks. We decided to share a Friday afternoon action shot of the server mounted in the rack.
We are still running the IOMeter benchmarks using the new ZFSBuild2012 server and InfiniBand SRP. The numbers we are seeing with SRP are absolutely awesome. For example, right now it is running a 32k random read benchmark and it is getting nearly 60,000 IOPS and moving 1854MB/second. This is amazing because when we ran raw IB performance tests on the network, the max performance of the IB network was 1890MB/second. The ib_read_bw and ib_write_bw tools showed our IB network moving an average of 1869MB/second and a peak of 1890MB/second. It is really exciting to see the ZFSBuild2012 box delivering 1854MB/second through IOMeter, which is about 98% of the wirespeed for our IB network.
So – just a quick update – we got Infiniband SRP (SCSI RDMA Protocol) working from our Windows system to our Nexenta system. Here’s a screengrab from IOMeter beating the crap out of our 20Gbit Infiniband network. This is a 16k, random read workload. Obviously it’s all fitting in to RAM, but compared to iSCSI w/ IPoIB, there’s no contest. iSCSI/IPoIB manages about 400MB/sec, and 27,000 IOPS. This should be very exciting to see this as a possibility moving forward.
We are re-running the original ZFSBuild2010 tests, and initial results are that this system is significantly faster than the old system. 4k Random Reads are peaking out at over 50,000 IOPS, delivering over 200MB/sec over Infiniband. 8k random reads delivering 40,000 IOPS and over 300MB/sec over Infiniband. These numbers are AWESOME!
Keep in mind though, this is with a 25GB working set. ZFSBuild 2010 only had 12GB of RAM, not nearly enough to cache even 25GB of data. ZFSBuild 2012 has 64GB of RAM, allowing to put all of this data in RAM. We’ll be tailoring the benchmarks after this run to more accurately reflect real world workloads, as we know the 25GB working set size is giving us artificially high results.
Look for more info in the next week or so!