At Couchbase we take performance very seriously, and with the launch of our new product, Couchbase Autonomous Operator 1.0, we wanted to make sure it’s Enterprise-grade and production ready for customers.

In this blog post we will discuss the detailed performance results from running YCSB Performance Benchmark tests on Couchbase Server 5.5 using the Autonomous Operator to deploy on Kubernetes platform. One of the big concerns for Enterprises planning to run database on Kubernetes is ‘performance’ .

This document gives a quick comparison of two workloads, namely YCSB A & E with Couchbase Server 5.5 on Kubernetes vs bare metal.

YCSB Workload A: This workload has a mix of 50/50 reads and writes. An application example is a session store recording recent actions.

Workload E: Short ranges: In this workload, short ranges of records are queried, instead of individual records. Application example: threaded conversations, where each scan is for the posts in a given thread (assumed to be clustered by thread id).

In general, we observed no significant performance degradation in running Couchbase Cluster on Kubernetes, Workload A had on par performance compared to bare metal and Workload E had approximately less than 10% degradation. 

Setup:

For the setup, Couchbase was installed using the Operator deployment as stated below. For more details on the setup, please refer here

Files:

Operator deployment: deployment.yaml (See Appendix)

Couchbase deployment: couchbase-cluster-simple-selector.yaml (See Appendix)

Client / workload generator deployment: pillowfight-ycsb.yaml  (See Appendix) (Official pillowfight docker image from dockerhub and installed java and YCSB manually on top of it)

Hardware:

7 servers

24 CPU x 64GB RAM per server

Couchbase setup

4 servers: 2 data nodes, 2 index+query nodes

40GB RAM quota for data service

40GB RAM quota for index services

1 data/bucket replica

1 primary index replica

Tests:

YCSB WorkloadA and WorkloadE

10M docs

Workflow after new empty k8s cluster is initialized on 7 servers:

 

Running the workloads:

 

Test results:

Env Direct setup Kubernetes pod resources Test Bare metal Kubernetes Delta
Env 1 22 vCPU, 48 GB RAM

(cpu cores and RAM available  are set on OS core level)

Limit to:

cpu: 22000m = ~22vCPU

mem: 48GB

All pods are on dedicated nodes

WorkloadA

50/50 get/upsert

Throughput: 194,158 req/sec

CPU usage avg: 86% of all 22 cores

Throughput: 192,190 req/sec

CPU usage avg: 94% of the cpu quota

– 1%
Env 2 16 vCPU, 48 GB RAM

(cpu cores and RAM available  are set on OS core level)

Limit to:

cpu: 16000m = ~16vCPU

mem: 48GB

All pods are on dedicated nodes

WorkloadA

50/50 get/upsert

Throughput: 141,909 req/sec

CPU usage avg: 89% of all 16 cores

Throughput: 145,430 req/sec

CPU usage avg: 100% of the cpu quota

+ 2.5%

Env Direct setup Kubernetes pod resources Test Bare metal Kubernetes Delta
Env 1 22 vCPU, 48 GB RAM

(cpu cores and RAM available  are set on OS core level)

Limit to:

cpu: 22000m = ~22vCPU

mem: 48GB

All pods are on dedicated nodes

WorkloadE

95/5 scan/insert

Throughput: 15,823 req/sec

CPU usage avg: 85% of all 22 cores

Throughput: 14,281 req/sec

CPU usage avg: 87% of the cpu quota

– 9.7%
Env 2 16 vCPU, 48 GB RAM

(cpu cores and RAM available  are set on OS core level)

Limit to:

cpu: 16000m = ~16vCPU

mem: 48GB

All pods are on dedicated nodes

WorkloadE

95/5 scan/insert

Throughput: 13,014 req/sec

CPU usage avg: 91% of all 16 cores

Throughput: 12,579 req/sec

CPU usage avg: 100% of the cpu quota

– 3.3%

Conclusions:

Couchbase Server 5.5 is production ready to be deployed on Kubernetes with the Autonomous Operator. Performance of Couchbase Server 5.5 on Kubernetes comparable to running on bare metal.   There is little performance penalty in running Couchbase Server on Kubernetes platform. Looking at the results Workload A had on par performance compared to bare metal and Workload E had approximately less than 10% degradation.

References:

  1. YCSB Workloads https://github.com/brianfrankcooper/YCSB/wiki/Core-Workloads
  2. Couchbase Kubernetes page https://www.couchbase.com/products/cloud/kubernetes
  3. Download Couchbase Autonomous Operator https://www.couchbase.com/downloads  
  4. Introducing Couchbase Operator https://www.couchbase.com/blog/couchbase-autonomous-operator-1-0-for-kubernetes-and-openshift/

Appendix

My deployment.yaml file

My couchbase-cluster-simple-selector.yaml file

 

My pillowfight-ycsb.yaml file

 

Author

Posted by Raju Suravarjjala, Senior Director, Quality Engineering, Couchbase

Raju Suravarjjala is a Senior Director of Quality Engineering at Couchbase. He is well versed in managing teams of various sizes and specializes in testing distributed systems. He has around 20 years of industry experience working with various technology companies like Gupta SQLBase, Zaplet, Plumtree, BEA Systems, and Oracle. He holds a master's degree in computer science from University of Louisiana at Lafayette and earned his bachelor's in mechanical engineering from Jawaharlal Nehru Technological University, India.

2 Comments

  1. Dear Raju,

    It’s a nice blog.
    Please update it a bit, so that it’s easier to follow along.

    Two comments:

    Comment#1:
    sequoiatools/pillowfight:v5.0.1 seems to be built on coreos and hence doesn’t have a package manager specifically it doesn’t have apt-get.

    Comment#2
    Below steps is not working now as there’s some change in Oracle Java license agreement.
    sudo apt-add-repository ppa:webupd8team/java

    -Thanks

  2. I installed JDK 11.

    I got below error when using “-p couchbase.epoll=true”

    Loading workload…
    Starting test.
    Maximum execution time specified as: 120 secs
    WARNING: An illegal reflective access operation has occurred
    WARNING: Illegal reflective access by com.couchbase.client.deps.io.netty.util.internal.PlatformDependent0 (file:/opt/ycsb-couchbase2-binding-0.15.0/lib/core-io-1.3.1.jar) to field java.nio.Buffer.address
    WARNING: Please consider reporting this to the maintainers of com.couchbase.client.deps.io.netty.util.internal.PlatformDependent0
    WARNING: Use –illegal-access=warn to enable warnings of further illegal reflective access operations
    WARNING: All illegal access operations will be denied in a future release
    Aug 23, 2019 9:15:34 AM com.couchbase.client.deps.io.netty.util.internal.PlatformDependent
    INFO: Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system unstability.
    com.yahoo.ycsb.DBException: Could not connect to Couchbase Bucket.
    at com.yahoo.ycsb.db.couchbase2.Couchbase2Client.init(Couchbase2Client.java:208)
    at com.yahoo.ycsb.DBWrapper.init(DBWrapper.java:86)
    at com.yahoo.ycsb.ClientThread.run(Client.java:424)
    at java.base/java.lang.Thread.run(Thread.java:834)
    Caused by: java.lang.IllegalStateException: failed to create a child event loop
    at com.couchbase.client.deps.io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:68)
    at com.couchbase.client.deps.io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:49)
    at com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:91)
    at com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:67)
    at com.yahoo.ycsb.db.couchbase2.Couchbase2Client.init(Couchbase2Client.java:195)
    … 3 more
    Caused by: java.lang.NullPointerException
    at com.couchbase.client.deps.io.netty.util.internal.PlatformDependent0.allocateMemory(PlatformDependent0.java:330)
    at com.couchbase.client.deps.io.netty.util.internal.PlatformDependent.allocateMemory(PlatformDependent.java:210)
    at com.couchbase.client.deps.io.netty.channel.epoll.IovArray.(IovArray.java:64)
    at com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoop.(EpollEventLoop.java:60)
    at com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:106)
    at com.couchbase.client.deps.io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:64)
    … 7 more

Leave a reply