Last week, we published a blog with recommendation on securing Couchbase data platform in response to industry-wide security vulnerabilities. We continued to analyze the potential performance impact caused by the patched OS binaries and this blog post captures the detailed evaluation.

As mentioned in previous blog, for these attacks to be feasible, attackers must be able to run malicious processes on the same host and processor as the victim processes. The risk of this attack happening on a controlled production environment is low since there are restrictions in place on the applications running.

If you can control access to a machine, you may not need (wherever applicable) to set the kernel parameters to enable the mitigations and hence not incur performance degradation.

Impact Analysis

When you patch your OS with the binaries currently available to mitigate these attacks you will likely see increased CPU utilization. For example, CPU usage increased by about 20% during one of our tests. (Check your Web Console monitoring page to access bucket statistics.)

 

To evaluate performance impact we ran the following tests

  • Scenario ‘A’ – Couchbase Cluster is sufficiently sized with additional capacity for production deployment.
  • Scenario ‘B’ – Couchbase Cluster is sized appropriately to handle existing load, but without further capacity available.

Workload

YCSB Workload A with mixed workload (50/50 read/write).

Scenario A

Ran YCSB Workload A on Couchbase Cluster – 4 nodes, 2 buckets, Intel Processor E5-2630 v4 with 20 hyper-threaded cores, 64GB RAM.

We compared the performance between un-patched ‘CentOS 7 kernel 3.10.0-514.2.2’ and patched ‘CentOS 7 kernel 3.10.0.693.11.6’ version.  

Results – Below graph shows CPU utilization increased roughly by 30% on the patched machines. However, the performance impact to Throughput and Latency was under 5% since these machines had additional CPU capacity to grow.

 

 

Un-patched – CentOS 7 kernel 3.10.0-514.2.2 (in μs) Patched – CentOS 7 kernel 3.10.0-693.11.6 (in μs)
Read Latency Average 187 196
95th 213 228
99th 1,497 1,525
Write Latency Average 209 219
95th 236 254
99th 1,583 1,606

Scenario B

Run YCSB Workload A on Couchbase Cluster – 3 nodes, 1 bucket, Intel Processor E5-2680 v3 with 12 cores, 64GB RAM.

Results: Below graph shows Throughput and Latency on patched machine is affected since CPU is running close to maximum utilization on these machines.

 

 

Un-patched – CentOS 7 kernel 3.10.0-514.2.2 (in μs) Patched – CentOS 7 kernel 3.10.0-693.11.6 (in μs)
Read Latency Average 847 1,318
95th 2,627 4,093
99th 7,207 9,167
Write Latency Average 879 1,352
95th 2,703 4,163
99th 7,315 9,271

Conclusion

We recommend re-evaluating your sizing for Couchbase cluster to ensure it has additional CPU capacity before applying OS patch to have a minimal impact on Throughput and Latency.

Author

Posted by Anil Kumar, Director Product Management, Couchbase Cloud-Native Database

Anil Kumar is the Director of Product Management at Couchbase. Anil’s career spans more than 19+ years of building software products across various domains, including enterprise software and cloud services. He is a hands-on product leader responsible for Couchbase Server, Couchbase Cloud, and Kubernetes product lines, including evangelizing the product strategy and vision with customers, partners, developers, and analysts. Before joining Couchbase, Anil spent several years working at Microsoft Redmond. Anil holds a master’s degree in computer science from the University of Toronto (Canada) and a bachelor’s in information technology from Visvesvaraya Technological University (India).

Leave a reply