What we’ve learned – Working Set

So lets start talking about some of the things that we’ve learned over the last year, shall we?  The number one thing that we have learned is that your working set size dramatically impacts your performance on ZFS.  When you can keep that working set inside of RAM, your performance numbers are outrageous.  Hundreds of thousands of IOPS.  Latency is next to nothing, and the world is wonderful.  Step over that line though, and you had better have architected your pool to absorb it.

Flash forward to a year later, and we’ve found that our typical response times and IOPS delivered are much higher than we expected, and latency is much lower.  Why is that you may ask?  With large datasets and very random access patterns, you would expect close to worst-case scenarios.  What we found is that we had a lot of blocks that were never accessed.  We have thick-provisioned most of our VM’s, which results in 100+GB of empty space in most of those VM’s (nearly 50% of all allocated capacity).  With Nexenta and ZFS, all of the really active data was being moved into the ARC/L2ARC cache.  While we still had some reads and writes going to the disks, it was a much smaller percentage.  We quickly figured out that the algorithms and tuning Nexenta employs in ZFS for caching seems to be very very intelligent, and our working set was much smaller than we ever really imagined.

Tuesday, November 15th, 2011 Configuration, Virtualization, ZFS

5 Comments to What we’ve learned – Working Set

  • avatar says:

    Already thank you for all that you share with us.
    What would be nice, this will give us a full story with photos on your new configuration and upgrades both hardware and software …

    Thank you in advance and good back with us.

  • swefred says:

    Hi, have you completely moved over to Nexenta? What level are you paying for?

    I do not like the idé of paying more for licenses than for hardware.

  • We have moved a large part of our workload over to Nexenta. Before we move the rest over, we are working on building a new system with more capability and scalability.

    As far as moving to paid Nexenta, we have not done that internally yet. Personally I’ve been exposed to the Nexenta Silver and Gold support channels with my work at other customer sites, and I love having their support. Internally though there is a cost ceiling that prevents us from using the Enterprise version. If we were building and consuming storage at a faster rate, it would be a much simpler choice, as we’d outstrip our capacity before the corresponding head unit prices came down. As it stands today, our backend capacity is not growing fast enough to warrant head units that will scale for 3-5 years. As such, the community version has suited our needs very well so far.

    This may change in the future as data just keeps growing at an exponential rate. At some point it will make sense to purchase licensing and grow the heads to more and more spindles. Right now we have not hit that point.

  • camperf says:

    Hey are you using NexentaStor or Nexenta Core Platform `?

  • We have run NCP for testing, but are using NexentaStor Community edition in production, and we are evaluating Nexenta Enterprise for future production uses.

  • Leave a Reply

    You must be logged in to post a comment.