Archive for September, 2012
It’s been two years since we built our last ZFS based server, and we decided that it was about time for us to build an updated system. The goal is to build something that exceeds the functionality of the previous system, while costing approximately the same amount. The original ZFSBuild 2010 system cost US $6765 to build, and for what we got back then, it was a heck of a system. The new ZFSBuild 2012 system is going to match the price point of the previous design, yet offer measurably better performance.
The new ZFSBuild 2012 system is comprised of the following :
SuperMicro SC846BE16-R920 chassis – 24 bays, single expander, 6Gbit SAS capable. Very similar to the ZFSBuild 2010 server, with a little more power, and a faster SAS interconnect.
SuperMicro X9SRI-3F-B Motherboard – Single socket Xeon E5 compatible motherboard. This board supports 256GB of RAM (over 10x the RAM we could support in the old system) and significantly faster/more powerful CPU’s.
Intel Xeon E5 1620 – 3.6Ghz latest generation Intel Xeon CPU. More horsepower for better compression and faster workload processing. ZFSBuild 2010 was short on CPU, and we found it lacking in later NFS tests. We won’t make that mistake again.
20x Toshiba MK1001TRKB 1TB SAS 6Gbit HDD’s – 1TB SAS drives. The 1TB SATA drives that we used in the previous build were ok, but SAS drives give much better information about their health and performance, and for an enterprise deployment, are absolutely necessary. These drives are only $5 more per drive than what we paid for the drives in ZFSBuild 2010. Obviously if you’d like to save more money, SATA drives are an option, but we strongly recommend using SAS drives when ever possible.
LSI 9211-8i SAS controller – Moving the SAS duties to a Nexenta HSL certified SAS controller. Newer chipset, better performance, and replaceability in case of failure.
Intel SSD’s all around – We went with a mix of 2x Intel 313 (ZIL), 2x 520 (L2ARC) and 2x 330 (boot – internal cage) SSD’s for this build. We have less ZIL space than the previous build (20GB vs 32GB) but rough math says that we shouldn’t ever need more than 10-12GB of ZIL. We will have more L2ARC (480GB vs 320GB) and the boot drives are roughly the same.
64GB RAM – Generic Kingston ValueRAM. The original ZFSBuild was based on 12GB of memory, which 2 years ago seemed like a lot of RAM for a storage server. Today we’re going with 64 GB right off the bat using 8GB DIMM’s. The motherboard has the capacity to go to 256GB with 32GB DIMM’s. With 64GB of RAM, we’re going to be able to cache a _lot_ of data. My suggestion is to not go super-overboard on RAM to start with, as you can run into issues as noted here : http://www.zfsbuild.com/2012/03/05/when-is-enough-memory-too-much-part-2/
For the same price as our ZFSBuild 2010 project, the ZFSBuild 2012 project will include more CPU, much more RAM, more cache, better drives, and better chassis. It’s amazing what two years difference makes when building this stuff.
Expect that we’ll evaluate Nexenta Enterprise, OpenIndiana, and revisit FreeNAS’s ZFS implementation. We probably won’t go back over the Promise units, as we’ve already discussed them and they likely haven’t changed (and we don’t have any lying about not doing anything anymore).
We are planning to re-run the same battery of tests that we used in 2010 for the original ZFSBuild 2010 benchmarks. We still have the same test blade server available to reproduce the testing environment. We also plan to run additional tests using various sized working sets. InfiniBand will be benchmarked in additional to standard gigabit Ethernet this round.
So far, we have received nearly all of the hardware. We are still waiting on a cable for the rear fans and a few 3.5 to 2.5 drive bay converters for the ZIL and L2ARC SSD drives. As soon as those items arrive, we will place the ZFSBuild 2012 server in our server room and begin the benchmarking. We are excited to see how it performs relative to the ZFSBuild 2010 server design.
Here are a couple pictures we have taken so far on the ZFSBuild 2012 project:
Just got a note in my inbox that Nexenta has “Important” news. They’ve cancelled the OpenStorage Summit in San Jose, and they’ve refunded my registration fee. Great. Just what I wanted. I absolutely did not want to go to San Jose, meet and greet with Nexenta employees and users, and network with said people. I also didn’t want to get out of Iowa for a few days and enjoy the weather in San Jose. What I _am_ looking forward to is cancelling both my hotel reservation and my airfare. I’m betting the airfare probably doesn’t get refunded. Class act guys. It would have been really nice if you would have decided this a month ago when you opened registration. I wonder how many other people are holding the bag on airfare and room reservations. I would have to think that Nexenta had the forethought to check with the hotel and see how many rooms had been reserved, but maybe not. Maybe they expected everyone to book their rooms at the last minute, along with their airfare, because that’s what most conference attendees do.
Nexenta – you can go ahead and assume I won’t be registering for the virtual open storage summit. Also go ahead and assume that I’m probably not interested in coming next year either, because who knows how much notice I’ll get that you’ve cancelled that one too.
Hotel and airfare booked for the OpenStorage Summit 2012 in San Jose. Last year was spectacular and was well worth the trip. Anyone else thinking about going should stop thinking and register!
It’s been noted over on the Nexentastor.org forums that there was a change in the Nexenta EULA that now prevents commercial usage of Nexenta Community Edition. Forum thread can be found here : http://www.nexentastor.org/boards/1/topics/7593
I found this discouraging, as the ability to use Nexenta Community Edition in our production environment was the reason that we selected Nexenta. All of our original testing was done with OpenSolaris, and it actually performed better than Nexenta. We went with Nexenta Community Edition because of the ease of use and the ability to upgrade to Enterprise Edition and purchase support if we needed it. Removing the option for small businesses to use Nexenta Community Edition in production is not something that I expected to see from Nexenta. I wondered why this happened.
I took some time to think about this and try to figure out why Nexenta’s stance might have changed. After browsing the forums, and seeing posts that say things like “I tried contacting Nexenta support” I stumbled upon the idea that support could be a big part of it. This is a free version of Nexenta, allowing you to use up to 18TB of space, with NO SUPPORT. People then come to the forums and complain that there’s no support, or they don’t get a response, or they got little help.
Support costs money. There have been a number of people that are using Nexenta Community Edition that have been contacting support (at least one noted here –http://www.nexentastor.org/boards/2/topics/5662#message-7672). Even if support doesn’t help you, you’re still tying up time on the phone with them, or forcing them to write an email response. This costs money. The EULA change isn’t going to change peoples behavior, but it does make it easier for Nexenta to send you to the forums for support, and use canned responses for Nexenta Community Edition questions.
The other possibility that I could see would be someone purchasing a single Nexenta Enterprise Edition Silver license, and then installed Nexenta Community Edition on 20 other devices, and try to get support on those devices also. That’s pretty shady, but I can easily envision someone doing that. Saying that Nexenta Community Edition isn’t to be run on production workloads allows Nexenta to punt questions if they come from a non-supported system much easier. This is similar to the problem that Cisco has with their SmartNet support. You buy 40 Cisco switches, put SmartNet on one of them, and voila, you’ve got support for every device that you own. Cisco is starting to get this worked out, and I can see a lot of shops hurting in the next few years when they have to buy SmartNet on every device they own.
My suggestion to potential Nexenta Community Edition users – if you’re considering running Nexenta Community Edition in production, go for it. From what I can tell, Nexenta is not planning to use the terms of the EULA to sue people for using Nexenta Community Edition in production. Nexenta IS likely going to give you grief if you call in to support, and you’ll likely not get any help. They’re a for-profit company, and I can’t fault them for wanting to remain in the black. If it’s a mission critical workload and you absolutely need support, buy Nexenta Enterprise Silver at a minimum instead of using Nexenta Community Edition. Nexenta Enterprise Silver is still cheaper than nearly any other support package you’ll find, and my experiences with support have been nothing less than stellar.
My suggestion to Nexenta – figure this out on the backend. By telling small business that they cannot use Nexenta Community Edition in production, you have opened the door for FreeNAS, OpenFiler and Napp-It to step in and grab these startups that desperately want to use your product. Your product is better than FreeNAS, OpenFiler and Napp-It, but FreeNAS, OpenFiler, and Napp-It don’t include draconian licensing limitations. Figure out how to allow these small businesses to use Nexenta Community Edition and flourish, and when they’re ready to go big time, let ’em buy Nexenta Enterprise Silver/Gold/Platinum licensing and support rather than figuring out how to pay Napp-It or one of the others for their support. If the EULA for Nexenta Community Edition had looked like this when we started using it, we would have thought long and hard about not using it or recommending it to other people. I don’t want to do that, and I don’t want anyone reading this site to do that. The Nexenta WebGUI is comfortable and I’ve gotten quite used to it, I’d hate to go back and have to create iSCSI targets from the command line.
At some point in time, somebody will sit down and write a good web based GUI that runs on OpenSolaris/OpenIndiana. By originally allowing production usage of Nexenta Community Edition, Nexenta took away the desire to code that alternative web GUI. After all, nobody wants to spend months coding something when Nexenta Community Edition existed for free, worked awesome, and allowed production usage. Now that Nexenta Community Edition is not allowed for production usage, there will likely be renewed interest in developing a good open source web GUI.