ZFSBuild2012 – Specs and Parts Pictures

We took a lot of pictures while we were building and testing the ZFSBuild2012 SAN.

The ZFSBuild 2012 system is comprised of the following :

SuperMicro SC846BE16-R920 chassis – 24 bays, single expander, 6Gbit SAS capable.

SuperMicro X9SRI-3F-B Motherboard – Single socket Xeon E5 compatible motherboard.

Intel Xeon E5 1620 – 3.6Ghz latest generation Intel Xeon CPU.

20x Toshiba MK1001TRKB 1TB SAS 6Gbit HDD’s – 1TB SAS drives.

LSI 9211-8i SAS controller – Moving the SAS duties to a Nexenta HSL certified SAS controller.

Intel SSD’s all around
2x Intel 313 series 20GB SSD drives for ZIL
2x Intel 520 series 240GB SSD drives for L2ARC
2x Intel 330 series 60GB SSD drives for boot (installed into internal cage)

64GB RAM (8x 8GB) – Generic Kingston ValueRAM.

20Gbps ConnectX InfiniBand card

4x ICYDock 2.5″ to 3.5″ drive bay converters

Internal drive bay bracket for boot drives

Y-cable for the rear fans (not enough fan sockets without one Y-cable)

Here are some pictures of the first few parts that showed up:
Stack of computer parts

Stack of computer parts

Stack of computer parts

Stack of computer parts

Here are pictures of the box the chassis came in:
SuperMicro SC846BE16-R920 chassis

SuperMicro SC846BE16-R920 chassis

SuperMicro SC846BE16-R920 chassis

SuperMicro SC846BE16-R920 chassis

SuperMicro SC846BE16-R920 chassis

Here are pictures of the SSD Drives:
SSD Drives

SSD Drives

SSD Drives

SSD Drives

Here are pictures of the box of 1TB SAS drives:
1TB SAS drives

1TB SAS drives

1TB SAS drives

1TB SAS drives

1TB SAS drives

1TB SAS drives

1TB SAS drives

1TB SAS drives

1TB SAS drives

1TB SAS drives

Here are pictures of the motherboard:
SuperMicro X9SRI-3F-B Motherboard

SuperMicro X9SRI-3F-B Motherboard

SuperMicro X9SRI-3F-B Motherboard

SuperMicro X9SRI-3F-B Motherboard

SuperMicro X9SRI-3F-B Motherboard

SuperMicro X9SRI-3F-B Motherboard

SuperMicro X9SRI-3F-B Motherboard

SuperMicro X9SRI-3F-B Motherboard

SuperMicro X9SRI-3F-B Motherboard

SuperMicro X9SRI-3F-B Motherboard

SuperMicro X9SRI-3F-B Motherboard

SuperMicro X9SRI-3F-B Motherboard

SuperMicro X9SRI-3F-B Motherboard

SuperMicro X9SRI-3F-B Motherboard

SuperMicro X9SRI-3F-B Motherboard

Here are pictures of the CPU Heatsink:
CPU Heatsink

CPU Heatsink

CPU Heatsink

Here are pictures of the fan Y-cable:
fan Y-cable

fan Y-cable

fan Y-cable

fan Y-cable

fan Y-cable

Here are pictures of the InfiniBand Card:
InfiniBand Card

InfiniBand Card

InfiniBand Card

InfiniBand Card

Here are pictures of the ICY Dock 2.5″ to 3.5″ converter:
ICY Dock 2.5

ICY Dock 2.5

ICY Dock 2.5

ICY Dock 2.5

Here are pictures of the Internal drive bay bracket for boot drives:
Internal drive bay bracket for boot drives

Internal drive bay bracket for boot drives

Internal drive bay bracket for boot drives

Thursday, December 13th, 2012 Hardware

10 Comments to ZFSBuild2012 – Specs and Parts Pictures

  • kevrags says:

    Hi,

    Great system! Quick question: Nexenta doesn’t recommend using SATA disks with a SAS expander, and the SSDs are SATA disks. Have you noticed any issues with this setup? We are about to start another server build, using the 847E16-JBOD1 for the disk subsystem, and are in the process of sourcing L2ARC and ZIL drives, so would appreciate any feedback. If we have the budget, the ZIL will be a ZeusRAM, but still need L2ARC. We were thinking about installed the L2ARC disks in the server itself.

    Thanks,

    Kevin

  • schism says:

    So your caching SSDs are sata. Did you have to get any adapters for that? How are those presented to you in nexenta? Just as another cNNWWID device ?

  • Nick Bryant says:

    Hi Matt – Great blog thanks.

    Which IB HBA ConnectX did you use? I’ve heard the cards with no on board memory aren’t supported in Nexenta.

  • We used the MHRH29-XTC. I do not know that there are _any_ IB cards on the HSL. For the InfinihostIII cards, that is true. For the ConnectX cards, you do not need memory on the card from what I can tell. They use different Drivers (Tavor vs Hermon iirc) and the Connectx stuff seems to work significantly better than the Infinihost based cards.

  • Nick Bryant says:

    Thanks for clarifying – didn’t realise it only applied to InifiHostIII cards.

  • matkix says:

    Question regarding the LSI 9211-8i SAS, do you have to configure each drive as a single drive within the cards bios? or do you just attach drives and they are instantly provided to the operating system?

    Thanks!

  • Since this is simply an HBA, rather than a RAID card, we do not have to configure each individual drive in the card bios, they are simply presented to the OS as individual drives. It is recommended to do this rather than use a RAID card and configure each drive as RAID cards typically mask some of the underlying drive behavior which is not desirable in ZFS.

  • somaddict says:

    The case is very nice. I started digging around a bit and found a case/backplane combo that is interesting. It doesn’t seem like a real backplane but thermaltake makes these x6 2.5″ drive per 5.25″ bay “things” that would allow for up to 72 2.5″ hotswappable drives in this arguably bush-league case. The reliability of the thermaltake is questionable. The case has 12 5.25″ external bays and room for 2 PSUs and a LOT of fans, haha. Would have to drop $5k or more on that many drives, but this could maybe provide a feasible 60-70TB JBOD to work with for 5400rpm 8mb archival action.

    Xigmatek Elysium – CCC-HSA0DS-U01 – ATX Super Tower – $169
    http://www.newegg.com/Product/Product.aspx?Item=N82E16811815011

    12 x thermaltake MAX-1562 backplane $900
    http://www.amazon.com/Thermaltake-Backplane-Removable-2-5-Inches-RC1600101A/dp/B004G8QERK/ref=sr_1_cc_1?s=aps&ie=UTF8&qid=1361125237&sr=1-1-catcorr&keywords=2.5+thermaltake+backplane

  • wvandenberge says:

    With the Intel 313 SLC units becoming unavailable, do you have an alternative you like as a ZIL device? Have you ever tried something like the DDRDrive3? It’s only 4GB but looks like its an order of magnitude faster than the 313 which may work ok if the txg commit interval is lowered from 5 seconds.

  • Leave a Reply

    You must be logged in to post a comment.