Often Overlooked Linux OS Tweaks

There are two simple linux OS level settings that people seem to be overlooking setting correctly on their production systems I have seen. These are documented elsewhere, but they keep coming up and seems like they need some quick review here. It is not like these are some super secret setting or magic bullet performance fixing items necessarily, but they are things that in a production Couchbase DB should be set correctly as below and incorporated into whatever system/process you use to bootstrap the nodes you use for Couchbase. They help with memcached performance and rebalance performance and in some cases stability issues.

Please make sure you test these out in a test environment first before moving to production with them obviously.

Swappiness should to be turned off

This one is pretty straightforward if you know about the Linux virtual memory system. Swappiness levels tell the virtual memory subsystem how much it should try and swap to disk. The thing is, the system will try to swap out items in memory even when there is plenty of RAM available to the system. The OS default is usually 60, which is a little aggressive IMO. You can see what value your system is set to by running the following command:

Since Couchbase is tuned to really operate in memory as much as possible. You can gain or at minimum not lose performance by just changing the swappiness value to 0. In non-tech talk, this tells the virtual memory subsystem of the OS to not swap items from RAM to disk unless it really really has to, which if you have sized your nodes correctly, swapping should not be needed. To set this, perform the following process use sudo or just become root if you ride in the wild west.

Make sure that you either have or modify your process that builds your OSs to do this. This is especially critical for public/private clouds where it is so easy to bring up new instances. You need to make this part of your build process for a Couchbase node.

Disable Transparent Huge Pages (THP)

Starting in Red Hat Enterprise Linux (RHEL) version 6, so this includes CentOS 6 and 7 too, a new default method of managing huge pages was implemented in the OS. Ubuntu has this setting as well starting in 12.02, so it will need this changed as well. THP combines smaller memory pages into Huge Pages without the running processes knowing. The idea is to reduce the number of lookups on TLB required and therefor increase performance. It brings in abstraction for automatation and management of huge pages basically.  Couchbase Engineering has determined that under some conditions, Couchbase Server can be negatively impacted by severe page allocation delays when THP is enabled. Couchbase therefore recommends that THP be disabled on all Couchbase Server nodes

Confirm if the OS settings need to be disabled

Check the status of THP by issuing the following commands:

On some Red Hat or Red Hat variants, you might have to do this:

If in one or both files, the output looks like this, you need the below procedure:

Copy the Init Script

The init script is designed to make sure the changes are made around the same time as Couchbase is loaded on reboot.

How to Register the Code in the OS

Do the following:

Create a file with the above code

Chmod the file to be executable

Execute it so it takes effect right now

Make sure the init script starts at boot

Red Hat variants:

Ubuntu:

Test the Process

Check the status of THP by issuing the following commands:

On some Red Hat or Red Hat variants, you might have to do this instead:

For both files, the output should be like this:

Note: There is a different way to do this that you will find elsewhere and edits /etc/grub.conf. My problem with it is that it would get blown out with each and every kernel update in the future. What I propose is easier to manage in the long run and easy to put into something like Puppet module or Chef recipe to append to the end of rc.local when you boot strap a node.

THP is a great feature for some things, but causes problems with applications like Couchbase. It is not alone in this. If you go search the Internet for transparent huge pages, there are multiple documented issues from other DB and application vendors about this. Until something has been found to work with this, it is just best to turn THP off.