At Couchbase we take performance very seriously, and with the launch of our new product, Couchbase Autonomous Operator 1.0, we wanted to make sure it’s Enterprise-grade and production ready for customers.
In this blog post we will discuss the detailed performance results from running YCSB Performance Benchmark tests on Couchbase Server 5.5 using the Autonomous Operator to deploy on Kubernetes platform. One of the big concerns for Enterprises planning to run database on Kubernetes is ‘performance’ .
This document gives a quick comparison of two workloads, namely YCSB A & E with Couchbase Server 5.5 on Kubernetes vs bare metal.
YCSB Workload A: This workload has a mix of 50/50 reads and writes. An application example is a session store recording recent actions.
Workload E: Short ranges: In this workload, short ranges of records are queried, instead of individual records. Application example: threaded conversations, where each scan is for the posts in a given thread (assumed to be clustered by thread id).
In general, we observed no significant performance degradation in running Couchbase Cluster on Kubernetes, Workload A had on par performance compared to bare metal and Workload E had approximately less than 10% degradation.
Setup:
For the setup, Couchbase was installed using the Operator deployment as stated below. For more details on the setup, please refer here
Files:
Operator deployment: deployment.yaml (See Appendix)
Couchbase deployment: couchbase-cluster-simple-selector.yaml (See Appendix)
Client / workload generator deployment: pillowfight-ycsb.yaml (See Appendix) (Official pillowfight docker image from dockerhub and installed java and YCSB manually on top of it)
Hardware:
7 servers
24 CPU x 64GB RAM per server
Couchbase setup
4 servers: 2 data nodes, 2 index+query nodes
40GB RAM quota for data service
40GB RAM quota for index services
1 data/bucket replica
1 primary index replica
Tests:
YCSB WorkloadA and WorkloadE
10M docs
Workflow after new empty k8s cluster is initialized on 7 servers:
1 2 3 4 5 6 7 8 9 |
# assign labels to the nodes so all services/pods will be assigned to right servers: kubectl label nodes arke06-sa09 type=power kubectl label nodes arke07-sa10 type=client kubectl label nodes ark08-sa11 type=client kubectl label nodes arke01-sa04 type=kv kubectl label nodes arke00-sa03 type=kv kubectl label nodes arke02-sa05 type=kv kubectl label nodes arke03-sa06 type=kv |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 |
#deploy Operator: kubectl create -f deployment.yaml #deploy Couchbase kubectl create -f couchbase-cluster-simple-selector.yaml #deploy Client(s): kubectl create -f pillowfight-ycsb.yaml I ran my tests directly from the client node by logging into the docker image of the client pod: docker exec -it --user root <pillowfight-yscb container id> bash And installing YCSB environment there manually: apt-get upgrade apt-get update apt-get install -y software-properties-common apt-get install python sudo apt-add-repository ppa:webupd8team/java sudo apt-get update sudo apt-get install oracle-java8-installer export JAVA_HOME=/usr/lib/jvm/java-8-oracle cd /opt wget http://download.nextag.com/apache/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz sudo tar -xvzf apache-maven-3.5.4-bin.tar.gz export M2_HOME="/opt/apache-maven-3.5.4" export PATH=$PATH:/opt/apache-maven-3.5.4/bin sudo update-alternatives --install "/usr/bin/mvn" "mvn" "/opt/apache-maven-3.5.4/bin/mvn" 0 sudo update-alternatives --set mvn /opt/apache-maven-3.5.4/bin/mvn git clone http://github.com/couchbaselabs/YCSB |
Running the workloads:
1 2 3 4 5 6 7 8 9 10 11 |
Examples of YCSB commands used in this exercise: Workload A Load: ./bin/ycsb load couchbase2 -P workloads/workloade -p couchbase.password=password -p couchbase.host=10.44.0.2 -p couchbase.bucket=default -p couchbase.upsert=true -p couchbase.epoll=true -p couchbase.boost=48 -p couchbase.persistTo=0 -p couchbase.replicateTo=0 -p couchbase.sslMode=none -p writeallfields=true -p recordcount=10000000 -threads 50 -p maxexecutiontime=3600 -p operationcount=1000000000 Run: ./bin/ycsb run couchbase2 -P workloads/workloada -p couchbase.password=password -p couchbase.host=10.44.0.2 -p couchbase.bucket=default -p couchbase.upsert=true -p couchbase.epoll=true -p couchbase.boost=48 -p couchbase.persistTo=0 -p couchbase.replicateTo=0 -p couchbase.sslMode=none -p writeallfields=true -p recordcount=10000000 -threads 50 -p operationcount=1000000000 -p maxexecutiontime=600 -p exportfile=ycsb_workloadA_22vCPU.log |
Test results:
Env | Direct setup | Kubernetes pod resources | Test | Bare metal | Kubernetes | Delta |
Env 1 | 22 vCPU, 48 GB RAM
(cpu cores and RAM available are set on OS core level) |
Limit to:
cpu: 22000m = ~22vCPU mem: 48GB All pods are on dedicated nodes |
WorkloadA
50/50 get/upsert |
Throughput: 194,158 req/sec
CPU usage avg: 86% of all 22 cores |
Throughput: 192,190 req/sec
CPU usage avg: 94% of the cpu quota |
– 1% |
Env 2 | 16 vCPU, 48 GB RAM
(cpu cores and RAM available are set on OS core level) |
Limit to:
cpu: 16000m = ~16vCPU mem: 48GB All pods are on dedicated nodes |
WorkloadA
50/50 get/upsert |
Throughput: 141,909 req/sec
CPU usage avg: 89% of all 16 cores |
Throughput: 145,430 req/sec
CPU usage avg: 100% of the cpu quota |
+ 2.5% |
1 2 3 4 5 6 7 8 9 |
Workload E: Load: ./bin/ycsb load couchbase2 -P workloads/workloade -p couchbase.password=password -p couchbase.host=10.44.0.2 -p couchbase.bucket=default -p couchbase.upsert=true -p couchbase.epoll=true -p couchbase.boost=48 -p couchbase.persistTo=0 -p couchbase.replicateTo=0 -p couchbase.sslMode=none -p writeallfields=true -p recordcount=10000000 -threads 50 -p maxexecutiontime=3600 -p operationcount=1000000000 Run: ./bin/ycsb run couchbase2 -P workloads/workloade -p couchbase.password=password -p couchbase.host=10.44.0.2 -p couchbase.bucket=default -p couchbase.upsert=true -p couchbase.epoll=true -p couchbase.boost=48 -p couchbase.persistTo=0 -p couchbase.replicateTo=0 -p couchbase.sslMode=none -p writeallfields=true -p recordcount=10000000 -threads 50 -p operationcount=1000000000 -p maxexecutiontime=600 -p exportfile=ycsb_workloadE_22vCPU.log |
Env | Direct setup | Kubernetes pod resources | Test | Bare metal | Kubernetes | Delta |
Env 1 | 22 vCPU, 48 GB RAM
(cpu cores and RAM available are set on OS core level) |
Limit to:
cpu: 22000m = ~22vCPU mem: 48GB All pods are on dedicated nodes |
WorkloadE
95/5 scan/insert |
Throughput: 15,823 req/sec
CPU usage avg: 85% of all 22 cores |
Throughput: 14,281 req/sec
CPU usage avg: 87% of the cpu quota |
– 9.7% |
Env 2 | 16 vCPU, 48 GB RAM
(cpu cores and RAM available are set on OS core level) |
Limit to:
cpu: 16000m = ~16vCPU mem: 48GB All pods are on dedicated nodes |
WorkloadE
95/5 scan/insert |
Throughput: 13,014 req/sec
CPU usage avg: 91% of all 16 cores |
Throughput: 12,579 req/sec
CPU usage avg: 100% of the cpu quota |
– 3.3% |
Conclusions:
Couchbase Server 5.5 is production ready to be deployed on Kubernetes with the Autonomous Operator. Performance of Couchbase Server 5.5 on Kubernetes comparable to running on bare metal. There is little performance penalty in running Couchbase Server on Kubernetes platform. Looking at the results Workload A had on par performance compared to bare metal and Workload E had approximately less than 10% degradation.
References:
- YCSB Workloads https://github.com/brianfrankcooper/YCSB/wiki/Core-Workloads
- Couchbase Kubernetes page https://www.couchbase.com/products/cloud/kubernetes
- Download Couchbase Autonomous Operator https://www.couchbase.com/downloads
- Introducing Couchbase Operator https://www.couchbase.com/blog/couchbase-autonomous-operator-1-0-for-kubernetes-and-openshift/
Appendix
My deployment.yaml file
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 |
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: couchbase-operator spec: replicas: 1 template: metadata: labels: name: couchbase-operator spec: nodeSelector: type: power containers: - name: couchbase-operator image: couchbase/couchbase-operator-internal:1.0.0-292 command: - couchbase-operator # Remove the arguments section if you are installing the CRD manually args: - -create-crd - -enable-upgrades=false env: - name: MY_POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: MY_POD_NAME valueFrom: fieldRef: fieldPath: metadata.name ports: - name: readiness-port containerPort: 8080 readinessProbe: httpGet: path: /readyz port: readiness-port initialDelaySeconds: 3 periodSeconds: 3 failureThreshold: 19 |
My couchbase-cluster-simple-selector.yaml file
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 |
apiVersion: couchbase.database.couchbase.com/v1 kind: CouchbaseCluster metadata: name: cb-example spec: baseImage: couchbase/server version: enterprise-5.5.0 authSecret: cb-example-auth exposeAdminConsole: true antiAffinity: true exposedFeatures: - xdcr cluster: dataServiceMemoryQuota: 40000 indexServiceMemoryQuota: 40000 searchServiceMemoryQuota: 1000 eventingServiceMemoryQuota: 1024 analyticsServiceMemoryQuota: 1024 indexStorageSetting: memory_optimized autoFailoverTimeout: 120 autoFailoverMaxCount: 3 autoFailoverOnDataDiskIssues: true autoFailoverOnDataDiskIssuesTimePeriod: 120 autoFailoverServerGroup: false buckets: - name: default type: couchbase memoryQuota: 20000 replicas: 1 ioPriority: high evictionPolicy: fullEviction conflictResolution: seqno enableFlush: true enableIndexReplica: false servers: - size: 2 name: data services: - data pod: nodeSelector: type: kv resources: limits: cpu: 22000m memory: 48Gi requests: cpu: 22000m memory: 48Gi - size: 2 name: qi services: - index - query pod: nodeSelector: type: kv resources: limits: cpu: 22000m memory: 48Gi requests: cpu: 22000m memory: 48Gi |
My pillowfight-ycsb.yaml file
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
apiVersion: batch/v1 kind: Job metadata: name: pillowfight spec: template: metadata: name: pillowfight spec: containers: - name: pillowfight image: sequoiatools/pillowfight:v5.0.1 command: ["sh", "-c", "tail -f /dev/null"] restartPolicy: Never nodeSelector: type: client |
Dear Raju,
It’s a nice blog.
Please update it a bit, so that it’s easier to follow along.
Two comments:
Comment#1:
sequoiatools/pillowfight:v5.0.1 seems to be built on coreos and hence doesn’t have a package manager specifically it doesn’t have apt-get.
Comment#2
Below steps is not working now as there’s some change in Oracle Java license agreement.
sudo apt-add-repository ppa:webupd8team/java
-Thanks
I installed JDK 11.
I got below error when using “-p couchbase.epoll=true”
Loading workload…
Starting test.
Maximum execution time specified as: 120 secs
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.couchbase.client.deps.io.netty.util.internal.PlatformDependent0 (file:/opt/ycsb-couchbase2-binding-0.15.0/lib/core-io-1.3.1.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of com.couchbase.client.deps.io.netty.util.internal.PlatformDependent0
WARNING: Use –illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Aug 23, 2019 9:15:34 AM com.couchbase.client.deps.io.netty.util.internal.PlatformDependent
INFO: Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system unstability.
com.yahoo.ycsb.DBException: Could not connect to Couchbase Bucket.
at com.yahoo.ycsb.db.couchbase2.Couchbase2Client.init(Couchbase2Client.java:208)
at com.yahoo.ycsb.DBWrapper.init(DBWrapper.java:86)
at com.yahoo.ycsb.ClientThread.run(Client.java:424)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: failed to create a child event loop
at com.couchbase.client.deps.io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:68)
at com.couchbase.client.deps.io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:49)
at com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:91)
at com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:67)
at com.yahoo.ycsb.db.couchbase2.Couchbase2Client.init(Couchbase2Client.java:195)
… 3 more
Caused by: java.lang.NullPointerException
at com.couchbase.client.deps.io.netty.util.internal.PlatformDependent0.allocateMemory(PlatformDependent0.java:330)
at com.couchbase.client.deps.io.netty.util.internal.PlatformDependent.allocateMemory(PlatformDependent.java:210)
at com.couchbase.client.deps.io.netty.channel.epoll.IovArray.(IovArray.java:64)
at com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoop.(EpollEventLoop.java:60)
at com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:106)
at com.couchbase.client.deps.io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:64)
… 7 more