Important Considerations

Important items to remember :

While building up our ZFS SAN server, we encountered a few issues in not having the correct parts on hand. Once we identified these parts, we ordered them as needed. The following is a breakdown of what not to forget.

Heatsink Retention bracket – We got all of our parts in, and lo and behold we couldn’t even turn the stupid box on. We neglected to take in to account that the heatsink that we ordered gets screwed down. The bracket needed for this is not included with the heatsink, the processor, the motherboard, or the case. It was a special order item from Supermicro that we had to source before we could even turn the system on.

The Supermicro part number for the heatsink retention bracket is BKT-0023L – a google search will lead you to a bunch of places that sell it.

SuperMicro Heatsink Bracket
SuperMicro Heatsink Retention Bracket

Reverse Breakout Cable – The motherboard that we chose actually has a built in LSI 1068E SAS controller. The unfortunate part about this is that it is 8 discreet SAS ports. Luckily they sell what is called a “reverse breakout” cable. This allows you to go from 4 discrete SAS ports to a single SFF-8087 backplane connection. This allowed us to use the onboard SAS in I/T mode to control our backplane and talk to our drives. We ordered ours from Newegg : Reverse Breakout Cable from Newegg

Reverse Breakout Cable - Discreet ConnectionsReverse Breakout Cable – Discreet Connections.

Reverse Breakout Cable - SFF8087 EndReverse Breakout Cable – SFF8087 End

Fixed HDD trays for Supermicro Chassis – we realized only too late that we did not have anywhere to mount our internal HDD’s for OS boot. The SuperMicro Chassis does not come with any internal drive trays. Those need to be ordered separately to allow you to mount HDD’s. We chose the 3.5″ HDD trays so that we could begin our testing with standard 3.5″ HDD’s before we ordered our SSD drives that we planned on booting from. If you plan on starting out with 2.5″ SSD’s you can order the 2.5″ part instead.

Dual 2.5″ HDD Tray part number – MCP-220-84603-0N
Single 3.5″ HDD Tray part number – MCP-220-84601-0N

LA or RA power and data cables – We also neglected to notice that when using the 3.5″ HDD trays that there isn’t really any room for cable clearance. Depending on how you mount your 3.5″ HDD’s, you will need Left Angle or Right Angle power and data connections. If you mount the power and data connectors at the top of the case, you’ll need Left Angle cabling. If you can mount the drives so the power and data are at the bottom of the case, you could use Right Angle cabling.

Left Angle ConnectorsLeft Angle Connectors

Left Angle Connectors connected to a HDDLeft Angle Connectors connected to a HDD

Power extension cables – We did not run in to this, but we were advised by Supermicro that it’s something they see often. Someone will build a system that requires 2x 8 pin power connectors, and the secondary 8 pin connector is too short. If you decide to build this project up using a board that requires dual 8 pin power connectors, be sure to order an extension cable, or you may be out of luck.

Fan power splitter – When we ordered our motherboard, we didn’t even think twice about the number of fan headers on the board. We’ve actually got more than enough on the board, but the location of those gave us another item to add to our list. The rear fans in the case do not have leads long enough to reach multiple fan headers. On the system board that we selected there was only one fan header near the dual fans at the rear of the chassis. We ordered up a 3 pin fan power splitter, and it works great.

Monday, April 12th, 2010 Hardware

4 Comments to Important Considerations

  • prattyy says:

    Could you explain as to why a “reverse breakout cable” was needed for this configuration.?
    is it a limitation of the motherboard or the back-plane?
    if i use a diffident motherboard with a HBA can i directly connect an SFF-8087 to SFF8087 cable to the back-plane and use all the 24 drives.?

  • The reverse breakout cable was needed because the motherboard has 8 discreet SAS ports, rather than the SFF8087 4x SAS ports. If you got an HBA with SFF8087 ports you would not need the reverse breakout cable.

    As an FYI, I would no longer recommend this system board for a ZFSBuild project. We are working on ZFSBuild 2012, and have received all of the hardware for the new build. We will be going in to detail in the near future the components that go into this build which should provide a significantly improved NexentaStor based server.

  • prattyy says:

    Does it not create a bottleneck when connecting 24 drive to a cable which was designed to plug in 4 drives?

    i know this is a old motherboard but what interested me was the on-board lsi 1068 chip which has IT mode.
    Looking forward to your new ZFSBuild project.i am sure it will have Tons of information as the previous one.i have learnt a lot of information from your article.
    one question though why nexentaStor why not openindiana?

  • The 4x SAS lanes provide an aggregate bandwidth of 12Gb/sec (or around 1.5GB/sec) of throughput. While those drives when sequentially reading, can theoretically push significantly more bandwidth than the 1.5GB/sec that the 4x SAS connector can provide, it’s highly unlikely that you’ll ever end up in that situation. Nearly every workload involves some randomness which decreases the maximum throughput of those drives significantly. Adding RAID and any type of virtualization workload on to it and your throughput drops to a few MB/sec per drive (for spinning media).

    As for the integrated 1068 chipset, the LSI 1068 is _old_. It’s been superseded by several generations of chips, many of which can be found integrated on new motherboards or on inexpensive add-in cards. All of these new chipsets are 6Gbit/sec which doubles the available throughput for a single 4x lane and are much better choices for future support.

    Nexentastor vs Openindiana – Openindiana didn’t exist when we put together ZFSBuild2010, and we wanted something that was a little more graphical. A lot has changed in the last two years, and the next round of reviews and benchmarks will contain a lot more competitors.

  • Leave a Reply

    You must be logged in to post a comment.