ZFSBuild2012 – Building Pictures

This article includes pictures taken while we were assembling the ZFSBuild2012 SAN.

Installing the motherboard:
Installing the motherboard

Installing the motherboard

Installing the motherboard

Installing the motherboard

Installing the motherboard

Installing the motherboard

Installing the motherboard

Installing the motherboard

Installing the motherboard

Installing the motherboard

Installing the motherboard

Installing the motherboard

Installing the motherboard

Installing the motherboard

Installing the motherboard

Installing the motherboard

Installing the motherboard

Installing the CPU:
Installing the CPU

Installing the CPU

Installing the CPU

Installing the CPU

Installing the CPU

Installing the CPU

Installing the CPU

Here is when we realized the internal plastic shroud was not going to work with this heatsink:
Installing the CPU

Installing the CPU

Installing the CPU

So we won’t be using the internal plastic shroud:
Installing the CPU

Installing the CPU

Installing the 64GB of RAM:
Installing the 64GB of RAM

Installing the 64GB of RAM

Installing the 64GB of RAM

Installing the 64GB of RAM

Installing internal SSD boot drives:
Installing internal SSD boot drives

Installing internal SSD boot drives

Installing internal SSD boot drives

Installing internal SSD boot drives

Installing internal SSD boot drives

Installing internal SSD boot drives

Installing internal SSD boot drives

Installing internal SSD boot drives

Installing internal SSD boot drives

Installing internal SSD boot drives

Installing internal SSD boot drives

Installing internal SSD boot drives

Needed a Y-cable for the SATA power for the internal boot drives:
Installing internal SSD boot drives

Installing internal SSD boot drives

Installing internal SSD boot drives

Installing internal SSD boot drives

Installing internal SSD boot drives

Pictures of the the internal boot drives mounted in the chassis:
Installing internal SSD boot drives

Installing internal SSD boot drives

Installing internal SSD boot drives

Installing internal SSD boot drives

Installing internal SSD boot drives

Installing SAS Controller Card:
Installing SAS Controller Card

Installing SAS Controller Card

Installing SAS Controller Card

Installing SAS Controller Card

Installing SAS Controller Card

Installing SAS Controller Card

Installing SAS Controller Card

Installing SAS Controller Card

Installing SAS Controller Card

Installing SAS Controller Card

Installing SAS Controller Card

Installing InfiniBand Card:
Installing InfiniBand Card

Installing InfiniBand Card

Installing InfiniBand Card

Installing InfiniBand Card

Installing InfiniBand Card

Installing ICY Dock:
Installing ICY Dock

Installing ICY Dock

Installing ICY Dock

Installing ICY Dock

Installing ICY Dock

Installing ICY Dock

Installing ICY Dock

Here is an ICY Dock in a hot swap tray compared to a standard SAS drive installed in a hot sway tray:
Installing ICY Dock

Installing ICY Dock

We took this next picture to explain why we use an ICY Dock instead of a cheap metal 2.5″ to 3.5″ converter. On top is the cheap converter; middle is ICY Dock; bottom is 1TB SAS drive:
Installing ICY Dock

Installing ICY Dock

Installing ICY Dock

Installing ICY Dock

Installing ICY Dock

Installing ICY Dock

Installing ICY Dock

Installing ICY Dock

Installing ICY Dock

Installing ICY Dock

Installing ICY Dock

Installing ICY Dock

Installing ICY Dock

Installing ICY Dock

Installing ICY Dock

Installing ICY Dock

Installing ICY Dock

In the rack:
In the rack

In the rack

In the rack

In the rack

In the rack

In the rack

In the rack

Note that all of the drive lights work except for the ZIL drives. The lights for the ZIL drives only turn on if you manually turn them on from the Nexenta Web GUI.
In the rack

Thursday, December 13th, 2012 Hardware

11 Comments to ZFSBuild2012 – Building Pictures

  • Jae-Hoon.Choi says:

    01. How do you config your NexentaStor CE boot device?

    Your Intel 330 SSD is 60GB that mirrored for your boot device. But your ZFSBuild 2012 Storage has 64GB Memory. NexentaStor CE installer try to create dump device your mirrored boot device.

    I was experienced that if boot device size smaller than system memory(ARC) then dump device creation faied during NexentaStor CE installation.

    2. I’m also have a plan for building personal ZFS block storage that maybe consist of 40Gb QDR infiniband.

    Why do you connect just only 1 IB cable?

    Do you have any multi-path configuration?

  • Eugene says:

    What the device in the rack between Supermicro?

  • sauce214 says:

    Looks like a Tripp Lite battery backup, something like SU3000RTXR3U.

  • matkix says:

    Question: I see you have a similar backplane to what I have in my box. I did some reading, and it appears if you connect both SAS cables, there is some potential for multipathing, IE getting 2x the speed between the backplane, and the HBA. Any thoughts on this?

  • somaddict says:

    Call me crazy, but how in the heck do you have 20 SAS drives in there with only one 4-port SAS port plugged in?

  • somaddict says:

    Nevermind, I’m an idiot. I can answer matkix’s question. The backplane in this chassis is the SAS-846EL1 which has a single port expander. The SAS-846EL2 has a dual port expander that supports cascading, failover, and multipath. A shame because the cases with the EL2 are not terribly more expensive.

  • We use primarily Hyper-V in our system, so we do not have any plans of using IB and ESX 5.x.

  • somaddict says:

    As a cheaper solution, I picked up a used SM chassis with a TQ backplane (Direct Attached) loaded with 24 1TB Barracuda 3gb ES.2 drives (Can’t believe they all work!). I chose the same motherboard but with the three PCI-X slots for some cheap FC/IB lab options. I’ll use the 8 on-board 3gb SAS ports and a couple old LSI 8-port 3gb mini-SAS cards. I thought this might be helpful for others.

    Also, these guys sell used 24-bay SM chassis with mobo/cpu/ram/TQ Backplane for around $300 in waves. You can get on their waiting list. I got a response that they would more in April. I’m not patient enough for that.

    http://www.tamsolutions.com/

    reference thread: http://lime-technology.com/forum/index.php?topic=21958.60

  • tien113 says:

    how can we use lsi 9211-8i for 24 drives ?

  • You would need to use a chassis that has a built-in backplane expander. The SuperMicro SC846E26 would be a good system to look at.

  • Leave a Reply

    You must be logged in to post a comment.