What we’ve learned – RAM

This post is possibly the most important lesson that we learned.  RAM is of MAJOR importance to Nexenta.  You can’t have enough of it.  Our original Nexenta deployment was 12GB of RAM.  It seemed like a silly amount of ram just a year ago.  Today we’re looking at it as barely a starting point.  Consider these facts :

1 – RAM is an order of magnitude (or more) faster than Flash.

2 – RAM is getting cheaper every day.

3 – You can put silly amounts of RAM in a system today.

4 – Data ages, and goes cold, and doesn’t get accessed as it gets older, reducing your Hot data footprint.

Lets go through these statements one by one.

1 – RAM is an order of magnitude (or more) faster than Flash.  Flash will deliver, on average, between 2,000 and 5,000 IOPS, depending on the type of SSD, the wear on the SSD, and garbage collection routines.  RAM has the capability to deliver hundreds of thousands of IOPS.  It doesn’t wear out, and there’s no garbage collection.

2 – RAM is getting cheaper every day.  When we built this platform last year, we paid over US $200 per 6GB of RAM.  Today you can buy 8GB Registered ECC DIMMS for under US $100.  16GB DIMM’s are hovering around US $300-$400.  Given the trends, I’d expect those to drop over the next year or two significantly.

3 – You can put silly amounts of RAM in a system today.  Last year, we were looking at reasonably priced boards that could fit 24GB of RAM in them.  Today we’re looking at reasonably priced barebones systems that you can fit 288GB of RAM in.  Insane systems (8 socket Xeon) support 2TB of RAM.  Wow.

4 – Data ages, goes cold, and doesn’t get accessed as much.  Even with only 12GB of RAM and 320GB of SSD, much of our working set is cached.  With 288GB of RAM, you greatly expand your capability of adding L2ARC (remember, L2ARC uses some of main memory) and increase your ARC cache capacity.  If your working set was 500GB on our old system you’d be running at least 200GB of it from spinning disk.  New systems configured with nearly 300GB of ARC and a reasonable (say 1TB) amount of L2ARC would cache that entire working set.  You’d see much of that working set cached in RAM (delivering hundreds of thousands of IOPS) part of it delivered from Flash (delivering maybe 10,000 IOPS), and only very old, cold data being served up from disk.  Talk about a difference in capabilities.  This also allows you to leverage larger, slower disks for older data.  If the data isn’t being accessed, who cares if it’s on slow 7200RPM disks?  That powerpoint presentation from 4 years ago isn’t getting looked at every day, but you’ve still got to save it.  Why not put it on the slowest disk you can find.

This being said, our new Nexenta build is going to have boatloads of RAM.  Maybe not 288GB (16GB DIMMS are still expensive compared to 8GB DIMMS) but I would put 144GB out there as a high probability.


Tuesday, November 15th, 2011 Configuration, Hardware 6 Comments

Disk Drive Selection

Our search for an affordable yet high performance storage array has led us to use ZFS, OpenSolaris, and commodity hardware. To get the affordable part of the storage under hand, we had to investigate all of our options when it came to hard drives and available SATA technology. We finally settled on a combination of Western Digital RE3 1TB drives, Intel X25-M G2 SSD’s, Intel X25-E SSD’s, and Intel X25-V SSD’s.

All Internal Drives › Continue reading

Wednesday, May 19th, 2010 Hardware 12 Comments

Why go with a BladeCenter instead of 1U systems?

The pro’s and con’s of using a BladeCenter

Many people looking through this site may be saying to themselves – Why would you not just build 1U systems and save yourself the cost of running a BladeCenter. Certainly 1U systems would be more cost effective than running a BladeCenter, right?  We thought so too until we really dug into it and found out that running the BladeCenter was actually less expensive when you look at building more than a few servers.  The detailed answers to how this shook out will be explored in this post in depth. › Continue reading

Monday, May 10th, 2010 Hardware 4 Comments

Test Blade Configuration

Our bladecenters are full of high performance blades that we use to run a virtualized hosting environment at this time. Since the blades that are in those systems are in production, we couldn’t very well use them to test the performance of our ZFS system. As such, we had to build another blade. We wanted the blade to be similar in spec to the blades that we were using, but we also wanted to utilize some of the new technology that has come out since we put many of our blades into production. › Continue reading

Wednesday, May 5th, 2010 Hardware 4 Comments

Installing the Infiniband Switch

Installing Infiniband switch in a SuperMicro SBE-710E

Our current infrastructure relies completely on iSCSI for our storage solution. As such, we have dual gigabit switch modules in our bladecenter. While this has worked very well for us, we want to expand our bladecenter to accept the SuperMicro 4x DDR Infiniband switch.
› Continue reading

Wednesday, April 14th, 2010 Hardware, InfiniBand 1 Comment

Important Considerations

Important items to remember :

While building up our ZFS SAN server, we encountered a few issues in not having the correct parts on hand. Once we identified these parts, we ordered them as needed. The following is a breakdown of what not to forget. › Continue reading

Monday, April 12th, 2010 Hardware 4 Comments

Motherboard, CPU, Heatsink, and RAM selection

Motherboard Selection – SuperMicro X8ST3-F

SuperMicro X8ST3-FSupermicro Packaging

SuperMicro X8ST3-FMotherboard Top Photo

We are planning on deploying this server with OpenSolaris 2009.06. As such we had to be very careful about our component selection. OpenSolaris does not support every piece of hardware sitting on your shelf. We had several servers that we tested with that would not boot into OpenSolaris at all. Granted, some of these were older systems with somewhat odd configurations. In any event, component selection needed to be made very carefully to make sure that OpenSolaris would install and work properly. › Continue reading

Thursday, April 8th, 2010 Hardware 3 Comments

Chassis Selection

SuperMicro SC846E1-R900B
SuperMicro SC846E1-R900B Chassis Photo

We host a lot of websites and need a lot of fast storage for those websites and our Cloud Infrastructure. We currently run a lot of individual iSCSI devices over Gigabit Ethernet. We want to consolidate those individual iSCSI devices in to a centralized unit that is comprised of Hybrid storage (SSD+HDD) that is expandable to support a large amount of drives with redundant connections to our Cloud Infrastructure. › Continue reading

Sunday, March 21st, 2010 Hardware 19 Comments

Welcome To ZFS Build!

Building a ZFS SAN to scale up to hundreds of drives and several enclosures takes a lot of fore-thought.  Also important in that decision is figuring out what kind or RAID levels you expect to be using, what kind of drives you will be using, and what kind of performance that you need.

We will discuss component selection including Chassis Selection, Motherboard Selection, Processor Selection, CPU Selection, Cooling Selection, Memory Selection,  SSD Selection, Hard drive selection, Infiniband selection,  and HBA selection.

We will also focus on the performance of the ZFS SAN using different ZFS RAID levels, and different failure modes.

Saturday, March 13th, 2010 Hardware No Comments