Back to Blogs

Often Overlooked Linux OS Tweaks

Kirk Kirkconnell of Couchbase Published

Filter blogs by:

There are two simple linux OS level settings that people seem to be overlooking setting correctly on their production systems I have seen. These are documented elsewhere, but they keep coming up and seems like they need some quick review here. It is not like these are some super secret setting or magic bullet performance fixing items necessarily, but they are things that in a production Couchbase DB should be set correctly as below and incorporated into whatever system/process you use to bootstrap the nodes you use for Couchbase. They help with memcached performance and rebalance performance and in some cases stability issues.

Please make sure you test these out in a test environment first before moving to production with them obviously.

Swappiness should to be turned off

This one is pretty straightforward if you know about the Linux virtual memory system. Swappiness levels tell the virtual memory subsystem how much it should try and swap to disk. The thing is, the system will try to swap out items in memory even when there is plenty of RAM available to the system. The OS default is usually 60, which is a little aggressive IMO. You can see what value your system is set to by running the following command:

cat /proc/sys/vm/swappiness

Since Couchbase is tuned to really operate in memory as much as possible. You can gain or at minimum not lose performance by just changing the swappiness value to 0. In non-tech talk, this tells the virtual memory subsystem of the OS to not swap items from RAM to disk unless it really really has to, which if you have sized your nodes correctly, swapping should not be needed. To set this, perform the following process use sudo or just become root if you ride in the wild west.

# Set the value for the running system
sudo sh -c 'echo 0 > /proc/sys/vm/swappiness'

# Backup sysctl.conf
sudo cp -p /etc/sysctl.conf /etc/sysctl.conf.`date +%Y%m%d-%H:%M`

# Set the value in /etc/sysctl.conf so it stays after reboot.
sudo sh -c 'echo "" >> /etc/sysctl.conf'
sudo sh -c 'echo "#Set swappiness to 0 to avoid swapping" >> /etc/sysctl.conf'
sudo sh -c 'echo "vm.swappiness = 0" >> /etc/sysctl.conf'

Make sure that you either have or modify your process that builds your OSs to do this. This is especially critical for public/private clouds where it is so easy to bring up new instances. You need to make this part of your build process for a Couchbase node.

Disable Transparent Huge Pages (THP)

Starting in Red Hat Enterprise Linux (RHEL) version 6, so this includes CentOS 6 and 7 too, a new default method of managing huge pages was implemented in the OS. Ubuntu has this setting as well starting in 12.02, so it will need this changed as well. THP combines smaller memory pages into Huge Pages without the running processes knowing. The idea is to reduce the number of lookups on TLB required and therefor increase performance. It brings in abstraction for automatation and management of huge pages basically.  Couchbase Engineering has determined that under some conditions, Couchbase Server can be negatively impacted by severe page allocation delays when THP is enabled. Couchbase therefore recommends that THP be disabled on all Couchbase Server nodes

Confirm if the OS settings need to be disabled

Check the status of THP by issuing the following commands:

cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag

On some Red Hat or Red Hat variants, you might have to do this:

cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
cat /sys/kernel/mm/redhat_transparent_hugepage/defrag

If in one or both files, the output looks like this, you need the below procedure:

[always] madvise never


Copy the Init Script

The init script is designed to make sure the changes are made around the same time as Couchbase is loaded on reboot.

# Provides: disable-thp
# Required-Start: $local_fs
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Disable THP
# Description: disables Transparent Huge Pages (THP) on boot


case $1 in
    if [ -d /sys/kernel/mm/transparent_hugepage ]; then
       echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled
       echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag
    elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then
      echo 'never' > /sys/kernel/mm/redhat_transparent_hugepage/enabled
      echo 'never' > /sys/kernel/mm/redhat_transparent_hugepage/defrag
      return 0


How to Register the Code in the OS

Do the following:

Create a file with the above code

$ sudo vi /etc/init.d/disable-thp


Chmod the file to be executable

$ sudo chmod 755 /etc/init.d/disable-thp


Execute it so it takes effect right now

$ sudo service disable-thp start


Make sure the init script starts at boot

Red Hat variants:

$ sudo chkconfig disable-thp on



$ sudo update-rc.d disable-thp defaults


Test the Process

Check the status of THP by issuing the following commands:

cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag


On some Red Hat or Red Hat variants, you might have to do this instead:

cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
cat /sys/kernel/mm/redhat_transparent_hugepage/defrag


For both files, the output should be like this:

always madvise [never]


Note: There is a different way to do this that you will find elsewhere and edits /etc/grub.conf. My problem with it is that it would get blown out with each and every kernel update in the future. What I propose is easier to manage in the long run and easy to put into something like Puppet module or Chef recipe to append to the end of rc.local when you boot strap a node.

THP is a great feature for some things, but causes problems with applications like Couchbase. It is not alone in this. If you go search the Internet for transparent huge pages, there are multiple documented issues from other DB and application vendors about this. Until something has been found to work with this, it is just best to turn THP off.

«   Back to blogs