Rolling Upgrades with Couchbase
One of the unique capabilities of Couchbase is the ability to do online upgrades without downtime. This is in part due to our auto-sharding and rebalancing features. In this blog I will walk through the process of doing a rolling upgrade to two nodes without taking the application off line.
This is actually a very timely topic, as I am writing this blog just as we announced a new version of our product, Couchbase Server 1.8. (Membase Server is now Couchbase Server - you can read more about that in a separate blog). So, for our example, we have two nodes of Membase Server version 1.7.2 that we would like to upgrade to Couchbase Server version 1.8. To get started, we installed Couchbase Server 1.8 on two new nodes, and to perform the upgrade we will essentially swap the data over from the current nodes to the new ones.
Here is our layout:
IP Version:
10.1.6.222 1.7.2
10.1.6.223 1.7.2
10.1.6.224 1.8
10.1.6.225 1.8
First, ensure that the two nodes of Couchbase Server 1.8 are similar to the nodes in the current cluster in terms of memory allocation. If the new node has less memory available it may not be able to join the cluster.
Rebalancing from the Web console
From there, click on: Manage->Server Nodes and add the new nodes by clicking on Add Server and then specifying the IP address as well as the password for the current cluster.
Note that when you are finished, the pending rebalance tab should have a red "2" on it. In this example we use 2 nodes for simplicity but if you have a 15 node cluster you can just as easily do a rolling upgrade by adding 15 new nodes.
From there, click Remove Server for each of the nodes that are scheduled to be removed.
From the web console you will have to leave one node in and remove/rebalance again to complete the upgrade. The web console has restrictions that will not allow you to remove all the original nodes in a cluster through a rebalance. The command line interface is less restrictive and can be used to remove both original servers in one rebalance.
Note that clicking Remove Server marks the node for removal but does not actually remove it. A rebalance is needed to finish the process of adding or removing nodes.
When you are finished clicking on Remove Server, the pending rebalance tab should have a red 3 on it. In this example, I am removing node 10.1.6.222 and adding in nodes 10.1.6.224/10.1.6.225
Click on the Pending Rebalance tab. Note that the nodes are clearly marked here in terms of what operation will be done to them.
Now click on Rebalance and leave this one window open. The browser cache is partly used to remember which nodes were added and removed so you will need to leave this rebalance window open for the duration of the rebalance in the off chance that the rebalance stalls or needs to be restarted. I recommend opening up a separate browser window if you need to look at something.
The rebalance process itself will regenerate a new vbucket map and redistribute the data across the new nodes in your cluster - in this case it will also handle the removal of data from node 10.1.6.222. Rebalance itself is a online operation. The smart client or Moxi is configured with a streaming URI to the cluster and is notified of updates such as topology changes. You can read more about rebalancing here:
After the rebalance is complete, you will need to go back to the Active servers tab, remove the last remaining server node from the original cluster and rebalance again to complete the rolling upgrade.
Three important items during rebalance:
- If the rebalance stops or pauses for whatever reason - you will need to wait a minimum of 5 minutes before restarting it. The network connections that we use for rebalance between nodes do not get shutdown gracefully on a pause or a unplanned stop so they end up living forever if a rebalance is kicked off immediately. However, the network connection has a timeout of 5 minutes if it does not see any activity and because of this we recommend waiting at least 5 minutes before rebalancing again.
- If the rebalance stops or pauses for whatever reason - you will need to start the rebalance again. The browser cache is partly used to remember which nodes were added and removed so you will need to leave this rebalance window open for the duration of the rebalance. I recommend opening up a separate browser window if you need to look at something. Before you click rebalance again it is crucial that you do so from the original window and that you verify in "pending rebalance" that the nodes are still pending for adding/removal. Note that if the browser window were to close for whatever reason - and a restart of rebalance is needed - you simply need to go back to the pending rebalance tab and verify that the correct nodes ares till pending addition or removal. If the cache has been wiped you will need to repeat the steps above.
- Client configuration - Both Moxi and the Smart clients need to be configured to point to at least one membase node that is part of the actual cluster. The streaming URI that both are configured with allow them to r eceive updates to the clusters topology. As you rebalance out nodes, make sure that the clients are configured with a node that is part of the original cluster with a backup configuration pointing to one of the new 1.8 nodes that is being added in. That way the clients can continue accessing the cluster as nodes are removed and added.
Monitoring Rebalance Progress
You can of course monitor through the rebalance GUI itself but you may need a more detailed view to determine if it stalls. You will need to watch 2 metrics on the nodes the disk write queue and the active vbucket count.
Disk Write Queue
The disk write queue should be under 1 million for each node. If it is over this number the node will send a back off message to other nodes when they try to send it vbuckets during the rebalance process. This is normal and the only thing to do here is to wait for the disk write queue to drain so that it will start accepting rebalance operations again.
[membase@Config349VM0 ~]$ ./check_queue_size.sh
Stats for: 10.1.6.222
ep_flusher_todo: 0
ep_queue_size: 633
Stats for: 10.1.6.223
ep_flusher_todo: 0
ep_queue_size: 571
Stats for: 10.1.6.224
Stats for: 10.1.6.225
[membase@Config349VM0 ~]$ ./check_vb_count.sh
Stats for: 10.1.6.222
vb_active_num: 452
Stats for: 10.1.6.223
vb_active_num: 373
Stats for: 10.1.6.224
vb_active_num: 62
Stats for: 10.1.6.225
vb_active_num: 149
[membase@Config349VM0 ~]$ cat check_queue_size.sh
#!/bin/sh
nodes="`cat host_list`"
for node in ${nodes}
do
echo "Stats for: ${node}"
/opt/membase/bin/mbstats ${node}:11210 all | egrep -i "ep_flusher_todo|ep_queue_size"
done
Active Bucket Count
This number should be changing on the nodes that are being added and removed. If you do not see this number change for 10-20 minutes than the rebalance may have stalled and will need to be stopped and restarted after a 5-minute pause.
[membase@Config349VM0 ~]$ ./check_vb_count.sh
Stats for: 10.1.6.222
vb_active_num: 512
Stats for: 10.1.6.223
vb_active_num: 512
Stats for: 10.1.6.224
Stats for: 10.1.6.225
[membase@Config349VM0 ~]$ ./check_vb_count.sh
Stats for: 10.1.6.222
vb_active_num: 452
Stats for: 10.1.6.223
vb_active_num: 373
Stats for: 10.1.6.224
vb_active_num: 62
Stats for: 10.1.6.225
vb_active_num: 149
[membase@Config349VM0 ~]$ cat check_vb_count.sh
#!/bin/sh
nodes="`cat host_list`"
for node in ${nodes}
do
echo "Stats for: ${node}"
/opt/membase/bin/mbstats ${node}:11210 all | egrep -i "vb_active_num:"
done
Rebalancing from the Command Line
Rebalancing from the command line gives us more flexibility as we can add new nodes and remove all the old nodes in one rebalance.
We will be using the membase cli command to specify which nodes to add as well as which to remove. Below are some of the details for the arguments we use - full details for the CLI command available here.
/opt/membase/bin/membase
usage: membase COMMAND CLUSTER [OPTIONS]
CLUSTER is --cluster=HOST[:PORT] or -c HOST[:PORT]
COMMANDs include:
server-add add one or more servers to the cluster
server-readd readd a server that was failed over
rebalance start a cluster rebalancing
rebalance-stop stop current cluster rebalancing
rebalance-status show status of current cluster rebalancing
OPTIONS:
-u USERNAME, --user=USERNAME admin username of the cluster
-p PASSWORD, --password=PASSWORD admin password of the cluster
server-add OPTIONS:
--server-add=HOST[:PORT] server to be added
--server-add-username=USERNAME admin username for the
server to be added
--server-add-password=PASSWORD admin password for the
server to be added
server-readd OPTIONS:
--server-add=HOST[:PORT] server to be added
--server-add-username=USERNAME admin username for the
server to be added
--server-add-password=PASSWORD admin password for the
server to be added
rebalance OPTIONS:
--server-add* see server-add OPTIONS
--server-remove=HOST[:PORT] the server to be removed
and here is a example of how to run this using the topology above:
[membase@Config349VM2 ~]$ /opt/membase/bin/membase rebalance \
-c 10.1.6.223:8091 \
--server-add=10.1.6.224:8091 \
--server-add=10.1.6.225:8091 \
--server-remove=10.1.6.223:8091 \
--server-remove=10.1.6.222:8091 \
-u Administrator \
-p<your password>
SUCCESS: server-add 10.1.6.224:8091
SUCCESS: server-add 10.1.6.225:8091
INFO: rebalancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
SUCCESS: rebalanced cluster