Archive for December, 2012

Dtrace broken with SRP targets?

Anyone that’s using Infiniband, SRP targets, Dtrace, and version of Nexenta Community edition, please raise your hands.

Nobody? ¬†Not suprising ūüôā ¬†Pulled some dtrace scripts off of another system to evaluate performance on the ZFSBuild2012 system, and got a very wierd error :



dtrace: failed to compile script ./arcreap.d: “/usr/lib/dtrace/srp.d”, line 49: translator member ci_local definition uses incompatible types: “string” = “struct hwc_parse_mt”


I’ve never seen this before, and the exact same script on ZFSBuild2010 works flawlessly. ¬†Something in SRP I’m guessing, that’s the part that’s throwing the error, and we aren’t using SRP on the ZFSBuild2010 system. ¬†If anyone at Nexenta or anyone working on the Illumos project sees anything here that make sense, I’d love to hear about it.

Thursday, December 27th, 2012 ZFS 4 Comments

ZFSBuild2012 – Write Back Cache Performance

Nexenta includes an option to enable or disable Write Back Cache on shared ZVols. To manage this setting, you must first create your ZVol and then set the ZVol to Shared. Then you can select the ZVol and edit the Write Back Caching setting. The purpose of this article is to find out how much performance is affected by the setting. All benchmarks were run on the ZFSBuild2012 hardware using Nexenta 3.1.3. All tests were run on the same ZFS storage pool. Click here to read about benchmark methods.

iSCSI-WC-E is iSCSI using IPoIB with connected mode disabled and the Write Back Cache enabled.

iSCSI-WC-D is iSCSI using IPoIB with connected mode disabled and the Write Back Cache disabled.

IB-SRP-WC-E is InfiniBand SRP with the Write Back Cache enabled.

IB-SRP-WC-D is InfiniBand SRP with the Write Back Cache disabled.

Generally speaking, enabling the Write Back Cache has no significant impact on read performance, but a huge improvement for write performance.

› Continue reading

Monday, December 17th, 2012 Benchmarks 13 Comments

ZFSBuild2012 – InfiniBand Performance

We love InfiniBand.  But it is no merely enough to simply install InfiniBand.  We decided to test three different popular connection options with InfiniBand so we could better understand which method offers the best performance.  We tested IPoIB with connected mode enabled (IPoIB-CM), IPoIB with connected mode disabled (IPoIB-UD), and SRP.
› Continue reading

Saturday, December 15th, 2012 Benchmarks 12 Comments

ZFSBuild2012 – Performance compared to ZFSBuild2010

ZFSBuild2012 is faster than ZFSBuild2010 in every way possible.  This page compares the iSCSI 1Gbps Ethernet performance difference between ZFSBuild2012 and ZFSBuild2010.  Both hardware designs are running Nexenta with write back caching enabled for the iSCSI shared ZVol.  Click here to read about benchmark methods.  We will be posting InfiniBand benchmarks with ZFSBuild2012 soon, and those InfiniBand benchmarks show even more performance.

IOMeter 4k Benchmarks:
IOMeter 4k Benchmarks › Continue reading

Friday, December 14th, 2012 Benchmarks 5 Comments

ZFSBuild2012 ‚Äď Benchmark Methods

We took great care when setting up and running benchmarks to be sure we were gathering data that could be used to make useful comparisons.  We wanted to be able to compare the ZFSBuild2012 design with the ZFSBuild2010 design.  We also wanted to be able to compare various configuration options within the ZFSBuild2012 design, so we could make educated decisions about how to configure a variety of options to get the most performance out of the design.  The purpose of this page is to share all of our benchmarking methods.
› Continue reading

Friday, December 14th, 2012 Benchmarks 9 Comments

ZFSBuild2012 – Building Pictures

This article includes pictures taken while we were assembling the ZFSBuild2012 SAN.

Installing the motherboard:
Installing the motherboard
› Continue reading

Thursday, December 13th, 2012 Hardware 11 Comments

ZFSBuild2012 – Specs and Parts Pictures

We took a lot of pictures while we were building and testing the ZFSBuild2012 SAN.

The ZFSBuild 2012 system is comprised of the following :

SuperMicro SC846BE16-R920 chassis – 24 bays, single expander, 6Gbit SAS capable.

SuperMicro X9SRI-3F-B Motherboard – Single socket Xeon E5 compatible motherboard.

Intel Xeon E5 1620 – 3.6Ghz latest generation Intel Xeon CPU.

20x Toshiba MK1001TRKB 1TB SAS 6Gbit HDD’s – 1TB SAS drives.

LSI 9211-8i SAS controller – Moving the SAS duties to a Nexenta HSL certified SAS controller.

Intel SSD’s all around
2x Intel 313 series 20GB SSD drives for ZIL
2x Intel 520 series 240GB SSD drives for L2ARC
2x Intel 330 series 60GB SSD drives for boot (installed into internal cage)

64GB RAM (8x 8GB) – Generic Kingston ValueRAM.

20Gbps ConnectX InfiniBand card

4x ICYDock 2.5″ to 3.5″ drive bay converters

Internal drive bay bracket for boot drives

Y-cable for the rear fans (not enough fan sockets without one Y-cable)

Here are some pictures of the first few parts that showed up:
Stack of computer parts

Stack of computer parts
› Continue reading

Thursday, December 13th, 2012 Hardware 10 Comments