RAM, Ranks, and Voltage

Here at ZFSBuild we have come across something unusual that we thought we would share.  This isn’t necessarily related to ZFS, but we encountered it while working on a ZFS/Nexenta system.

We recently had to dig deeper into RAM Ranks and voltage specifications.  This stems from populating a SuperMicro SYS-6026-RFT+ barebones system full of RAM.  The system in question has 18 DDR3 RAM slots.  We ordered 8GB DIMM’s from Wiredzone.com, and based on it’s specifications we were pretty sure it would work.  We got the system in, populated the RAM slots, and started running tests.  The weird thing about it was that the system only ever saw 120GB of RAM.  We started reading….and reading….and reading…  Finally came across some SuperMicro documents here and here.  Turns out the RAM we ordered was substituted with RAM that was assumed to be compatible.  The only difference between the DIMM’s we ordered and the DIMM’s we received was the voltage that they operated at.  We ordered 1.5V DIMM’s, and we were shipped 1.35V DIMM’s.  When using 1.35V DIMM’s the system detected 120GB of usable RAM.

We fought for a few days between SuperMicro, WiredZone, and our own gut feelings and finally got it sorted out.  Wiredzone shipped us new DIMM’s that were Dual Rank and 1.5V, and they worked flawlessly.  We’d like to give a big shout out to the WiredZone staff and to the SuperMicro staff that helped us on this.  It’s not terribly well understood black magic that goes on in these servers, and when working on the boundaries of what’s possible all sorts of odd things come up.  The last week has been one of them.

A side note on this is that we would have seen similar behavior if the RAM would have been quad-ranked ram.  In a quad-ranked configuration, the server will apparently only see 12 DIMM’s also.  In all of our years of building systems and working with servers we had never encountered this, and are very happy that we had the folks at WiredZone and SuperMicro to help us sort this out.

Monday, April 9th, 2012 Hardware

2 Comments to RAM, Ranks, and Voltage

  • shotel says:

    Was this RAM problem related to your previous “When is enough memory too much? Part 2” issue at all? Did you solve that issue? What is your IOPS performance delta as it relates to increasing RAM? i.e., if 48GB RAM delivers x IOPS, does 96GB RAM deliver x*2? 192GB RAM delivers x*3? Lastly, why all of the Storage vMotion activity (We rarely use SvMotion)?

    I do so want to thank you for keeping up this blog, it is of great help to us and countless others.

  • This RAM issue was not related to the previous memory issues at all. This is a separate platform that had a physical incompatibility with the RAM that we were shipped.

    As far as IOPS performance goes – increasing the RAM has done nothing to increase the total number of IOPS available, but it does let us keep a much larger data set in RAM, and accessible at ultra-high speed.

    Originally the issue discussed in “When is enough memory….” was brought on by Snapshot commital, essentially doing what Storage vMotion does (reading data into RAM, then deleting it very quickly). Once we relaized this, it necessetated a migration of all customers off the platform to troubleshoot in a clean environment, thus the SvMotions.

  • Leave a Reply

    You must be logged in to post a comment.