MongoDB published another benchmark performed by United Software Associates.

Benchmarks are a useful tool to evaluate database performance. But to be useful, they must be transparent and repeatable. If they fail to meet these standards, the results are questionable.

In recent benchmarks, Couchbase and MongoDB took two different approaches. Couchbase clearly documented the full configuration and included the results of every test. MongoDB did not.  

The below tables illustrate the different approaches taken by Avalon Consulting, LLC and United Software Associates:

  United Software Associates Avalon Consulting, LLC
YCSB
Entries / Operations 400M / 100M 300M / 100M
Value Size Missing 1K
Data Set Size Missing 286GB
Request Distribution Zipfian Uniform
Database
Nodes Missing MongoDB: 9
Couchbase Server: 9
Replicas Missing 3 (1 primary, 2 secondary)
Replication Missing MongoDB: Async
Couchbase Server: Async
Persistence Missing MongoDB: Async
Couchbase Server: Async
Configured Memory (per Node) Missing MongoDB: 30GB
Couchbase Server: 30GB
Total Data Set Size (with Replicas) Missing 858GB
Primary Data Resident in Memory Missing 32%
Version
Database MongoDB: 3.0.3
Couchbase Server: 3.0.2
MongoDB: 3.0.0
Couchbase Server: 3.0.2
Client MongoDB: 3.0.0
Couchbase Server: 2.1.2
MongoDB: 2.1.3
Couchbase Server: 2.1.0
Hardware
Servers Database: 3
YCSB: 1
Database: 9x AWS EC2 i2.2xlarge
YCSB: 2-23x AWS EC2 r2.8xlarge
Processors Both: 2x 3.0GHz Database: 8 vCPU (2.5GHz)
YCSB: 32vCPU (2.5GHz)
Memory Both: 96GB Database: 61GB
YCSB: 244GB
Storage Both: 2x 960GB SSD Database: 2x 800GB SSD
YCSB: 2x 320GB SSD
Networking Both: 10GbE Both: High
OS Both: Ubuntu 14.10 Database: CentOS 6
YCSB: Amazon Linux
OS
Transparent Huge Pages (THP) Disabled Disabled
NUMA Disabled Disabled

 

I hope MongoDB and DataStax will help me improve this template for future benchmarks.

United Software Associates Benchmark Results

They did not publish the results of all tests, they published the results of one.

The benchmark states the ideal number of threads for every database was 150 or 350 depending on the workload, but it does not state the number of threads for the results published.

Workload A Throughput

Threads MongoDB Couchbase Server
105 Missing Missing
140 Missing Missing
175 Missing Missing
210 Missing Missing
245 Missing Missing
280 Missing Missing
315 Missing Missing
350 Missing Missing

Results – Workload A Latency

Threads MongoDB Couchbase Server
105 Read: Missing
Write: Missing
Read: Missing
Write: Missing
140 Read: Missing
Write: Missing
Read: Missing
Write: Missing
175 Read: Missing
Write: Missing
Read: Missing
Write: Missing
210 Read: Missing
Write: Missing
Read: Missing
Write: Missing
245 Read: Missing
Write: Missing
Read: Missing
Write: Missing
280 Read: Missing
Write: Missing
Read: Missing
Write: Missing
315 Read: Missing
Write: Missing
Read: Missing
Write: Missing
350 Read: Missing
Write: Missing
Read: Missing
Write: Missing

Avalon Consulting, LLC Benchmark Results

They published the results of all tests and stated the number of threads for them.

Workload A Throughput

Threads MongoDB Couchbase Server
105 61K 110K
140 65K 141K
175 67K 154K
210 70K 170K
245 74K 193K
280 Exceeded Max Latency 238K
315 Exceeded Max Latency 245K
350 Exceeded Max Latency 252K

Results – Workload A Latency

Threads MongoDB Couchbase Server
105 Read: 1.42ms
Write: 2.05ms
Read: .78ms
Write: .76ms
140 Read: 2.01ms
Write: 2.97ms
Read: .79ms
Write: .78ms
175 Read: 3.16ms
Write: 3.54ms
Read: .89ms
Write: .88ms
210 Read: 3.5ms
Write: 4.49ms
Read: .93ms
Write: .92ms
245 Read: 4.19ms
Write: 5.38ms
Read: .92ms
Write: .91ms
280 Exceeded Max Latency Read: .92ms
Write: .92ms
315 Exceeded Max Latency Read: 1.06ms
Write: .99ms
350 Exceeded Max Latency Read: 1.22ms
Write: 1.22ms

Benchmark Configuration

A benchmark, if you care, should answer the following questions:

  • What was the hardware configuration?
  • How was the operating system configured?
  • What were the database and client versions?
  • How were the databases configured?
  • How was YCSB configured?

 

So, what configuration is this United Software Associates benchmark missing?

  • The number of nodes
  • The replica configuration (# of replicas
  • The replication configuration (async or sync)
  • The persistence configuration (async or sync)
  • The size of the values
  • The size of the data set
  • The size of the cache (per node)
  • The size of the total data set (including replicas)
  • The percentage of primary data resident in memory
  • The results of all runs
  • The number of threads for all published runs

 

In addition, the GitHub repository referenced in this United Software Associates white paper does not include the client configuration for MongoDB, Cassandra, or Couchbase Server.

And what about the MongoDB routers and config servers?

Were they deployed? If so, where?

I don’t know because this benchmark does not include all of the configuration. The first benchmark included most of the configuration, but there were a number of mistakes. With this benchmark, they may have fixed the mistakes, but they failed to include all of the configuration.

Bottom Line

It's important for benchmarks to be credible. Anyone should be able to reproduce a benchmark and validate its results. All of the configuration must be provided and, preferably, it should be performed on cloud infrastructure. If not, vendors can manipulate the configuration to make unfair comparisons look fair.

That's not helpful to anyone.

Resources

MongoDB + United Software Associates Benchmark
White Paper | Code

Couchbase + Avalon Consulting, LLC Benchmark
White Paper | Code

Discuss on Hacker News

Posted by Shane Johnson, Director, Product Marketing, Couchbase

Shane K Johnson was the Director of Product Marketing at Couchbase. Prior to Couchbase, he occupied various roles in developing and evangelism with a background in Java and distributed systems. He has consulted with organizations in the financial, retail, telecommunications, and media industries to draft and implement architectures that relied on distributed systems for data and analysis.

Leave a reply