MongoDB and DataStax, In the Rearview Mirror

Thumbtack published an excellent blog post today (link). It highlights the preliminary results of performance tests executed with Couchbase Server, MongoDB and DataStax Enterprise (Apache Cassandra). The final results will be included in a benchmark report. Thumbtack consulted with Couchbase, MongoDB and DataStax to ensure the databases were configured for optimal performance based on both the workload and the hardware. The result? Couchbase Server outperformed MongoDB and DataStax Enterprise in both read intensive workloads and balanced workloads.

Performance is a fundamental requirement of databases. It has a direct impact on user experience and customer satisfaction. A high performance database enables many users to access data at the same time and to do so without delay. It has a direct impact on cost. A high performance database requires fewer nodes. If it’s deployed to the cloud, it requires fewer instances, fewer subscriptions and results in less charges. If it’s deployed to a data center, it requires fewer servers, fewer subscriptions and results in less less energy consumption (link). It lowers costs.

Execution

• The performance tests were executed with YCSB.
• The performance tests were executed on dedicated servers.
• The performance tests were executed on four servers.
• The performance tests were executed with two operational workloads:
  • Read Intensive: 95% Read / 5% Write
  • Balanced: 50% Read / 50% Write
• The performance tests were executed with multiple clients loads.

Data

• Values: 20M
• Value Size: 150 bytes
• Replicas: 2
• 90% In-Memory

Read Intensive

Couchbase Server, 6.5x the throughput of MongoDB.
Couchbase Server, 7.5x the throughput of DataStax Enterprise.

MongoDB, 2x the read latency of Couchbase Server.
DataStax Enterprise, 3.1x the read latency of Couchbase Server.

* Latency is measured in milliseconds.
** The workload for MongoDB was reduced from 96 clien threads to 64 client threads to lower the latency.

This is no surprise.

First, Couchbase Server features an integrated, in-memory cache to deliver the highest read performance (link). It was engineered for performance, and that requires efficient access to data in memory.

Second, Couchbase Server leverages smart clients. Couchbase Server clients require one request (client to node) whereas MongoDB and DataStax Enterprise clients require two requests (client to node, node to node). Couchbase Server clients determine which node the data is stored on and send it a direct read request. However, MongoDB clients send read requests to routers. The router determines which node the data is stored on and forwards it the read request (link). DataStax Enterprise clients send read requests to random nodes. The random node determines which node the data is stored on and forwards it the read request (link).

One request is faster than two requests.

Balanced

Couchbase Server, 7.6x the throughput of MongoDB.
Couchbase Server, 2.7x the throughput of DataStax Enterprise.

MongoDB, 3x the write latency of Couchbase Server.

* Latency is measured in milliseconds.
** The workload for MongoDB was reduced from 96 clien threads to 64 client threads to lower the latency.

This is no surprise.

To deliver the highest write performance, Couchbase Server writes to an integrated, in-memory cache first. Then, it synchronizes the cache with the storage device for durability (link).

Like read operations, MongoDB and DataStax Enterprise require two requests for write operations. Couchbase Server does not.

Couchbase Server scales out writes to all nodes within a deployment whereas MongoDB limits writes to the primary nodes (link). As a result, all four Couchbase Server nodes executed write operations. However, only two MongoDB nodes executed write operations. The remaining two nodes were secondary nodes.

Couchbase Server implements document level locking. This enables Couchbase Server nodes to execute many writes at a time. However, MongoDB implements a single lock per database (link). MongoDB nodes are limited to executing a single write at a time per database.

DataStax Enterprise maintained low latency write requests during write-intensive workloads. However, it was unable to do so for read requests. As a result, the increased read latency lowered the throughput.

Notes

Despite being configured for strong consistency, Couchbase Server throughput was 2-7x more than MongoDB and DataStax Enterprise.

MongoDB was configured for eventual consistency.

As a result, MongoDB executes read operations on both primary nodes and secondary nodes for increased performance. It scales out read operations. However, it sacrifices consistency by allowing read operations to return inconsistent data. With the default configuration, MongoDB executes read operations on the primary nodes to maintain consistency (link). However, it decreases performance.

DataStax Enterprise was configured for eventual consistency.

As a result, DataStax Enterprise executes read and write operations with a single node for increased performance. However, it sacrifices consistency by allowing read operations to return inconsistent data. When configured for strong consistency, DataStax Enterprise executes read and write operations on multiple nodes (i.e. quorum) to maintain consistency (link). However, it decreases performance.

Thoughput, Latency and Concurrency

Thumbtack expected Couchbase Server to outperform the competition. It’s engineered for scalability and performance, and this engineering is reflected in the architecture. It’s engineered for high throughput, low latency and concurrency.

Couchbase Server thoughput is about twice that of MongoDB and DataStax Enterprise under a light workload. That’s because its latency is about half that of the others. If the performance test was performed with a single cilent thread, latency would determine throughput. However, concurrency is decisive factor. Couchbase Server is not yet saturated. Under increased workloads, the throughput increased. MongoDB and DataStax Enterprise are more or less saturated under light workloads. Under increased workloads, thier latency increased.

* The x-asix is the total number of YCSB client threads.

Summary

Not only does Couchbase Server delivers the highest performance, it’s easy to scale (link) and it’s consistent (link).

PDF with a complete description of the benchmark and the preliminary results is available.

The full report is now available.