Let ZFS use all of your RAM

So we’ve been talking about RAM usage, ram problems, and pretty much everything related to RAM lately, so I figured I’d mention this one too.

Many of you, if you’ve got a large memory system, may notice that your system never uses all of it’s RAM.  I’ve got some 192GB systems that routinely reported that they had 30GB of RAM free.  Why is this?  Everything that I’ve read says ZFS will use all of the RAM in your system that it can, how can it leave 30GB just wasting away doing nothing?

The answer is both surprising, and not surprising at the same time.  When ZFS was written if a system had 8GB of RAM it was a monster, and 16GB was nearly unheard of.  Let’s not even talk about 32GB.  Today having 192GB of RAM in a system isn’t difficult to achieve, and the new Xeon E5 platforms boast RAM capacities of 1TB and more.  Back then there was a need to limit ZFS a little bit from encroaching on system processes.  There is a Solaris “bug” (if you can call it that) that limits ARC cache to 7/8 of total system memory.  This was intentional so that the ARC cache couldn’t cause contention for swapfile space in RAM.  That “bug” still exists in Nexenta 3.1.2.  This is why I have 30GB of RAM free on a 192GB system.  1/8 system memory made sense when monster systems were 8GB.  That left 1GB of RAM that wouldn’t be touched for ARC caching to ensure proper system operation.  Today the amount of RAM in systems dwarfs what most people would have used 10 years ago, and as such we need to make modifications.  Fortunately, we can tune this variable.

From what I’ve read (based on information from Richard Elling (@richardelling on twitter)) the default for Nexenta 4.0 and Illumos distributions will be 256MB.  I take a very cautious approach to this and I’m going to use 1GB.  This is all discussed here.

For those who don’t want to check out the links – the pertinent info is here :

edit /etc/system for permanent fix – this reserves 1GB of RAM

set swapfs_minfree=262144

To change it on the fly

echo swapfs_minfree/W0t262144 | mdb -kw

This change has allowed my ARC cache to grow from 151-155GB utilization to 176GB utilization, effectively allowing me to use all of the RAM that I should be able to use.

FYI – this is unsupported, and Nexenta support will probably give you grief if you do this without discussing it with them (if you have a support contract), so be forewarned.  There may be unintended consequences from making this change that I am not aware of.

Wednesday, April 18th, 2012 Configuration, ZFS

5 Comments to Let ZFS use all of your RAM

  • Scott M says:

    Very interesting…. I’ve got a system similar (72gb) that’s been hovering around the 50gb arc for 18 months now….. It’ll be interesting to see how the usage goes this week with that setting. I have been hunting around for a long time for a fix to this so thanks (if it works).

  • The only thing that I’ve noticed is that I’ll get a memory check notification after a while because it will run my system down to less than 5% free memory. So far I have not seen it completely starve the system of memory, but I suppose it is a possibility. Use with caution!

  • Scott M says:

    Just thought I’d post a followup to this since its been a month since we changed our setting. The system is now using between 55 and 60gb for ARC. So we have a decent improvement and haven’t suffered any set backs. I’ve got some more ram to put into this system when its due for maintenance. That’ll get it up to 96gb, I’ll post again how it goes when we’ve got that running.

  • Thanks for the update Scott – we’ve also seen no negative impacts from the settings change yet on our end either.

  • Leave a Reply

    You must be logged in to post a comment.