Installing the Infiniband Switch

Installing Infiniband switch in a SuperMicro SBE-710E

Our current infrastructure relies completely on iSCSI for our storage solution. As such, we have dual gigabit switch modules in our bladecenter. While this has worked very well for us, we want to expand our bladecenter to accept the SuperMicro 4x DDR Infiniband switch.

Little did we know that we would have to take our bladecenter offline to accommodate that. After researching it a little bit, we decided to practice on one of our spare bladecenters so that we knew exactly how to install the switch and do it quickly on the production system that we would be working on. We didn’t want to have to down the system for two hours and figure out all of the ins and outs of the installation while servers were offline. Overall, we believe that this can be accomplished in 10 minutes or less.

The SuperMicro SBE-710E comes with a single management module, and the ability to expand to two management modules. Herein lies the issue. The management module is a single-wide component, and in the SBE-710E all of the expansion slots are configured for single-wide modules. The Infiniband switch is a double wide module. As such, you need to remove a chassis bracket to allow the double wide module to be inserted at the bottom of the chassis. You can see the bracket in this photo – the red arrow is pointing to it.

To remove this spacer, you have to slide the backplane out of the bladecenter about 8-10 inches to access two screws on the bottom of the backplane assembly. This means that all of the blades will be disconnected from power, network, and management assemblies. Time to announce a maintinance window as that means the entire bladecenter has to come offline to remove two screws. We would have appreciated it if SuperMicro had designed the bladecenter so that you didn’t have to take it offline to add a “hot-swap” infiniband module.

To slide the backplane module out, you have to remove four screws. One in each corner of the bladecenter.

Once those four screws are removed, the entire back of the bladecenter, holding the power supply modules, the ethernet switch modules, and the management module can be slid out.

Once the backplane is slid out, simply remove the two screws holding the bracket in place.

Once that is done, the bracket will slide out, and the bay will be ready to accommodate a double wide module.

After that is done, simply slide the backplane back into the bladecenter and replace the 4 screws that you removed.

Now it’s time for the Infiniband switch to be installed. With the bracket removed, the Infiniband switch really is a “hot-swap” device, since it actually fits. The module slides in easily, and the two levers on top secure it into the bladecenter.

Everything is installed and ready to go now. We simply need to cable the Infiniband switch up to our ZFS_HEAD_END system and we should be ready to configure Infiniband.

Wednesday, April 14th, 2010 Hardware, InfiniBand

1 Comment to Installing the Infiniband Switch

  • […] coaster of fun trying to get the InfiniBand network up and running properly. From the headache of installing the InfiniBand switch module to finding the correct Mezzanine cards for our blades, to getting IPoIB (IP over InfiniBand) […]

  • Leave a Reply

    You must be logged in to post a comment.