Archive for April, 2012
So we’ve been talking about RAM usage, ram problems, and pretty much everything related to RAM lately, so I figured I’d mention this one too.
Many of you, if you’ve got a large memory system, may notice that your system never uses all of it’s RAM. I’ve got some 192GB systems that routinely reported that they had 30GB of RAM free. Why is this? Everything that I’ve read says ZFS will use all of the RAM in your system that it can, how can it leave 30GB just wasting away doing nothing?
The answer is both surprising, and not surprising at the same time. When ZFS was written if a system had 8GB of RAM it was a monster, and 16GB was nearly unheard of. Let’s not even talk about 32GB. Today having 192GB of RAM in a system isn’t difficult to achieve, and the new Xeon E5 platforms boast RAM capacities of 1TB and more. Back then there was a need to limit ZFS a little bit from encroaching on system processes. There is a Solaris “bug” (if you can call it that) that limits ARC cache to 7/8 of total system memory. This was intentional so that the ARC cache couldn’t cause contention for swapfile space in RAM. That “bug” still exists in Nexenta 3.1.2. This is why I have 30GB of RAM free on a 192GB system. 1/8 system memory made sense when monster systems were 8GB. That left 1GB of RAM that wouldn’t be touched for ARC caching to ensure proper system operation. Today the amount of RAM in systems dwarfs what most people would have used 10 years ago, and as such we need to make modifications. Fortunately, we can tune this variable.
From what I’ve read (based on information from Richard Elling (@richardelling on twitter)) the default for Nexenta 4.0 and Illumos distributions will be 256MB. I take a very cautious approach to this and I’m going to use 1GB. This is all discussed here.
For those who don’t want to check out the links – the pertinent info is here :
edit /etc/system for permanent fix – this reserves 1GB of RAM
To change it on the fly
echo swapfs_minfree/W0t262144 | mdb -kw
This change has allowed my ARC cache to grow from 151-155GB utilization to 176GB utilization, effectively allowing me to use all of the RAM that I should be able to use.
FYI – this is unsupported, and Nexenta support will probably give you grief if you do this without discussing it with them (if you have a support contract), so be forewarned. There may be unintended consequences from making this change that I am not aware of.
Here at ZFSBuild we have come across something unusual that we thought we would share. This isn’t necessarily related to ZFS, but we encountered it while working on a ZFS/Nexenta system.
We recently had to dig deeper into RAM Ranks and voltage specifications. This stems from populating a SuperMicro SYS-6026-RFT+ barebones system full of RAM. The system in question has 18 DDR3 RAM slots. We ordered 8GB DIMM’s from Wiredzone.com, and based on it’s specifications we were pretty sure it would work. We got the system in, populated the RAM slots, and started running tests. The weird thing about it was that the system only ever saw 120GB of RAM. We started reading….and reading….and reading… Finally came across some SuperMicro documents here and here. Turns out the RAM we ordered was substituted with RAM that was assumed to be compatible. The only difference between the DIMM’s we ordered and the DIMM’s we received was the voltage that they operated at. We ordered 1.5V DIMM’s, and we were shipped 1.35V DIMM’s. When using 1.35V DIMM’s the system detected 120GB of usable RAM.
We fought for a few days between SuperMicro, WiredZone, and our own gut feelings and finally got it sorted out. Wiredzone shipped us new DIMM’s that were Dual Rank and 1.5V, and they worked flawlessly. We’d like to give a big shout out to the WiredZone staff and to the SuperMicro staff that helped us on this. It’s not terribly well understood black magic that goes on in these servers, and when working on the boundaries of what’s possible all sorts of odd things come up. The last week has been one of them.
A side note on this is that we would have seen similar behavior if the RAM would have been quad-ranked ram. In a quad-ranked configuration, the server will apparently only see 12 DIMM’s also. In all of our years of building systems and working with servers we had never encountered this, and are very happy that we had the folks at WiredZone and SuperMicro to help us sort this out.