The Couchbase Autonomous Operator enables you to run Couchbase deployments natively on Open Source Kubernetes or Enterprise Red Hat OpenShift Container Platform. I’m excited to announce the availability of the Couchbase Autonomous Operator 0.8 Beta Refresh today!

Running and managing a Couchbase cluster just got a lot easier with the introduction of the Couchbase Autonomous Operator for Kubernetes. Users can now deploy Couchbase on top of Kubernetes and have the Couchbase Autonomous Operator handle much of the cluster management, such as failure recovery and multidimensional scaling. However, users may feel a bit uncomfortable just sitting back and watching the Couchbase Autonomous Operator do its thing. To alleviate some of their worry, this three-part blog series will walk through the different ways the Quality Engineering team here at Couchbase gives our customers peace of mind when running Couchbase on Kubernetes.

This blog series will highlight three types of testing we do for Couchbase on Kubernetes. The first post in the series will focus on manual testing. The second post will be all about testing the Couchbase Autonomous Operator. And the third post will show our users how to test the Couchbase instances themselves with testrunner, our functional test suite.

Phase 1: Manual Certification of the Couchbase Autonomous Operator

Manual testing is often boring compared to the thrills of automated testing, but with Kubernetes it can actually be quite fun. In this post we will walk through setting up the Couchbase travel-sample app with an operator-managed Couchbase cluster as the datastore, all on Kubernetes. Once the application and cluster are all set up on Kubernetes, we will test some scaling and failure scenarios.

In the following sections, we will cover:

0: Prerequisites and Setup

1: Cluster Deployment

2: Cluster Configuration

3: Application Deployment

4: Verification

Prerequisites and Setup:

To set up that travel-sample app using Couchbase on Kubernetes, we will use minikube. Instructions for setting up minikube can be found here: https://kubernetes.io/docs/getting-started-guides/minikube/.

Once minikube is set up, you may want to increase the memory and CPU count. You will need to bring up the Kubernetes dashboard to monitor the cluster using: minikube dashboard

Cluster Deployment:

After setting up minikube, we will need to initialize the Couchbase Autonomous Operator and the Couchbase cluster that the operator will manage. The following YAML file will tell the Kubernetes master to create a Couchbase Autonomous Operator deployment:

Create a file called deployment.yaml and submit this YAML to Kubernetes with: kubectl create -f path/to/deployment.yaml. After a couple of seconds the operator deployment should show up in the Kubernetes dashboard.

Figure 1: State of the Kubernetes cluster after deploying the Couchbase Autonomous Operator.

Next, a secret must be provided to Kubernetes so that the Couchbase Autonomous Operator can manage the Couchbase nodes.

Create a file called secret.yaml and send the secret to Kubernetes with the following: kubectl create -f path/to/secret.yaml.

Next, let’s bring up a cluster of 2 Couchbase nodes. The following YAML specifies a cluster with two nodes, two buckets (default and travel-sample), and all services enabled:

Submit the cluster configuration with: kubectl create -f path/to/cb-cluster.yaml.

Figure 2: State of the Kubernetes cluster after scaling up to 2 Couchbase nodes.

Cluster Configuration:

Now that we have the a two node cluster managed by the Couchbase Autonomous Operator, updates to the cluster configuration should be made in the cb-cluster.yaml file and resubmitted to Kubernetes. Should any changes be made manually through the Couchbase UI, the operator will take action to re-align the cluster to the configuration specified in cb-cluster.yaml. To make changes to the cluster, first make the changes in cb-cluster.yaml, then update Kubernetes with: kubectl apply -f path/to/cb-cluster.yaml

The next step is to load the travel-sample data and index definitions into the cluster’s  travel-sample bucket. The following command will call cbimport on pod cb-example-0000: kubectl exec pod cb-example-0000 -ti /opt/couchbase/bin/cbimport json -c 127.0.0.1:8091 -u Administrator -p password -b travel-sample -f sample -d /opt/couchbase/samples/travel-sample.zip  

Application Deployment:

The cluster is all set up now, but the travel-sample app is not. Now, we need to build a docker image that Kubernetes will use for the app. The dockerfile below will pull down the travel-sample app from my fork in github, https://github.com/korry8911/try-cb-python, and install all dependencies. The travel.py file has been modified for this blog post to use the Kubernetes python client to grab the IPs of the running Couchbase nodes in the same Kubernetes namespace.

Build the travel-sample app docker image with: docker build -t your-dockerhub-handle/travel-sample:latestThe docker image needs to be loaded on to the Kubernetes worker node. Easiest way to do that is pulling the image from dockerhub. Push the travel-sample image with: docker push your-dockerhub-handle/travel-sample:latest.

The travel-sample app configuration is defined in the following file. Make sure to add your docker handle.

Load the travel-sample app with: kubectl create -f path/to/travel-sample.yaml.

Figure 3: State of the Kubernetes cluster after deploying the travel-sample app.

Kubernetes, by default, does not allow access to pods from outside the cluster. Therefore, to reach the travel-sample app, a node-port service must be created to proxy the port on which the travel-sample app listens for incoming requests.

Create the node-port service for the travel-sample with: kubectl create -f path/to/nodeport.yaml.

Figure 4: State of the Kubernetes cluster after adding a node-port service.

Since we are running the application on minikube, we must access the travel-sample app by running: minikube service travelsample.

Verification:

Now that we have played around with the travel-sample app, lets scale up the Couchbase cluster by one node. To do so, change the size parameter in the server section of the cb-cluster.yaml file from 2 to 3. Then, submit the changes to Kubernetes with: kubectl apply -f path/to/cb-cluster.yaml.

Figure 5: State of the Kubernetes cluster after scaling up to 3 Couchbase nodes.

The Couchbase cluster is now 3 nodes, which is enough nodes to test the Couchbase Autonomous Operators autofailover feature. Let’s kill one Couchbase node to simulate a failure scenario and watch the Couchbase Autonomous Operator automatically rebalance a new node back into the cluster. Kill one Couchbase pod by navigating to Pods in the Kubernetes dashboard and deleting the cb-example-0002 pod.

After the autofailover timeout period of 10 seconds, the Couchbase Autonomous Operator will eject the failed node from the cluster, initialize a new node, and rebalance the node into the cluster. Throughout this process, users can play with the travel-sample app without any service interruption

Figure 6: State of the Kubernetes cluster after recovery from a Couchbase node failure.

Conclusion:

The Couchbase Autonomous Operator takes a lot of the hassle out of running a Couchbase cluster in Open Source Kubernetes and/or Enterprise Red Hat OpenShift Container Platform. The desired state of the cluster is maintained automatically even in the face of node failure. Manual certification of the node recovery feature is fun but more testing is required. In the next, post we will go through, in-depth, our functional testing approach for other features of the Couchbase Autonomous Operator.

Ready to check out the Beta?

If you have any questions, comments, or additional feedback, leave a comment below or post on https://forums.couchbase.com/.

Posted by Korrigan Clark

Leave a reply