ZFSBuild2012 – Building Pictures
This article includes pictures taken while we were assembling the ZFSBuild2012 SAN.
Installing the motherboard:

















Installing the CPU:







Here is when we realized the internal plastic shroud was not going to work with this heatsink:



So we won’t be using the internal plastic shroud:


Installing the 64GB of RAM:




Installing internal SSD boot drives:












Needed a Y-cable for the SATA power for the internal boot drives:





Pictures of the the internal boot drives mounted in the chassis:





Installing SAS Controller Card:











Installing InfiniBand Card:





Installing ICY Dock:







Here is an ICY Dock in a hot swap tray compared to a standard SAS drive installed in a hot sway tray:


We took this next picture to explain why we use an ICY Dock instead of a cheap metal 2.5″ to 3.5″ converter. On top is the cheap converter; middle is ICY Dock; bottom is 1TB SAS drive:

















In the rack:







Note that all of the drive lights work except for the ZIL drives. The lights for the ZIL drives only turn on if you manually turn them on from the Nexenta Web GUI.

9 Comments to ZFSBuild2012 – Building Pictures
Leave a Reply
You must be logged in to post a comment.
01. How do you config your NexentaStor CE boot device?
Your Intel 330 SSD is 60GB that mirrored for your boot device. But your ZFSBuild 2012 Storage has 64GB Memory. NexentaStor CE installer try to create dump device your mirrored boot device.
I was experienced that if boot device size smaller than system memory(ARC) then dump device creation faied during NexentaStor CE installation.
2. I’m also have a plan for building personal ZFS block storage that maybe consist of 40Gb QDR infiniband.
Why do you connect just only 1 IB cable?
Do you have any multi-path configuration?
What the device in the rack between Supermicro?
Looks like a Tripp Lite battery backup, something like SU3000RTXR3U.
Question: I see you have a similar backplane to what I have in my box. I did some reading, and it appears if you connect both SAS cables, there is some potential for multipathing, IE getting 2x the speed between the backplane, and the HBA. Any thoughts on this?
Call me crazy, but how in the heck do you have 20 SAS drives in there with only one 4-port SAS port plugged in?
Nevermind, I’m an idiot. I can answer matkix’s question. The backplane in this chassis is the SAS-846EL1 which has a single port expander. The SAS-846EL2 has a dual port expander that supports cascading, failover, and multipath. A shame because the cases with the EL2 are not terribly more expensive.
Are you guys considering/using IB to ESX 5.x yet?
http://www.mellanox.com/related-docs/prod_software/ESX_5_X_release_notes.txt
We use primarily Hyper-V in our system, so we do not have any plans of using IB and ESX 5.x.
As a cheaper solution, I picked up a used SM chassis with a TQ backplane (Direct Attached) loaded with 24 1TB Barracuda 3gb ES.2 drives (Can’t believe they all work!). I chose the same motherboard but with the three PCI-X slots for some cheap FC/IB lab options. I’ll use the 8 on-board 3gb SAS ports and a couple old LSI 8-port 3gb mini-SAS cards. I thought this might be helpful for others.
Also, these guys sell used 24-bay SM chassis with mobo/cpu/ram/TQ Backplane for around $300 in waves. You can get on their waiting list. I got a response that they would more in April. I’m not patient enough for that.
http://www.tamsolutions.com/
reference thread: http://lime-technology.com/forum/index.php?topic=21958.60