MongoDB published another benchmark performed by United Software Associates.

Benchmarks are a useful tool to evaluate database performance. But to be useful, they must be transparent and repeatable. If they fail to meet these standards, the results are questionable.

In recent benchmarks, Couchbase and MongoDB took two different approaches. Couchbase clearly documented the full configuration and included the results of every test. MongoDB did not.  

The below tables illustrate the different approaches taken by Avalon Consulting, LLC and United Software Associates:

 United Software AssociatesAvalon Consulting, LLC
YCSB
Entries / Operations400M / 100M300M / 100M
Value SizeMissing1K
Data Set SizeMissing286GB
Request DistributionZipfianUniform
Database
NodesMissingMongoDB: 9
Couchbase Server: 9
ReplicasMissing3 (1 primary, 2 secondary)
ReplicationMissingMongoDB: Async
Couchbase Server: Async
PersistenceMissingMongoDB: Async
Couchbase Server: Async
Configured Memory (per Node)MissingMongoDB: 30GB
Couchbase Server: 30GB
Total Data Set Size (with Replicas)Missing858GB
Primary Data Resident in MemoryMissing32%
Version
DatabaseMongoDB: 3.0.3
Couchbase Server: 3.0.2
MongoDB: 3.0.0
Couchbase Server: 3.0.2
ClientMongoDB: 3.0.0
Couchbase Server: 2.1.2
MongoDB: 2.1.3
Couchbase Server: 2.1.0
Hardware
ServersDatabase: 3
YCSB: 1
Database: 9x AWS EC2 i2.2xlarge
YCSB: 2-23x AWS EC2 r2.8xlarge
ProcessorsBoth: 2x 3.0GHzDatabase: 8 vCPU (2.5GHz)
YCSB: 32vCPU (2.5GHz)
MemoryBoth: 96GBDatabase: 61GB
YCSB: 244GB
StorageBoth: 2x 960GB SSDDatabase: 2x 800GB SSD
YCSB: 2x 320GB SSD
NetworkingBoth: 10GbEBoth: High
OSBoth: Ubuntu 14.10Database: CentOS 6
YCSB: Amazon Linux
OS
Transparent Huge Pages (THP)DisabledDisabled
NUMADisabledDisabled

 

I hope MongoDB and DataStax will help me improve this template for future benchmarks.

United Software Associates Benchmark Results

They did not publish the results of all tests, they published the results of one.

The benchmark states the ideal number of threads for every database was 150 or 350 depending on the workload, but it does not state the number of threads for the results published.

Workload A Throughput

ThreadsMongoDBCouchbase Server
105MissingMissing
140MissingMissing
175MissingMissing
210MissingMissing
245MissingMissing
280MissingMissing
315MissingMissing
350MissingMissing

Results – Workload A Latency

ThreadsMongoDBCouchbase Server
105Read: Missing
Write: Missing
Read: Missing
Write: Missing
140Read: Missing
Write: Missing
Read: Missing
Write: Missing
175Read: Missing
Write: Missing
Read: Missing
Write: Missing
210Read: Missing
Write: Missing
Read: Missing
Write: Missing
245Read: Missing
Write: Missing
Read: Missing
Write: Missing
280Read: Missing
Write: Missing
Read: Missing
Write: Missing
315Read: Missing
Write: Missing
Read: Missing
Write: Missing
350Read: Missing
Write: Missing
Read: Missing
Write: Missing

Avalon Consulting, LLC Benchmark Results

They published the results of all tests and stated the number of threads for them.

Workload A Throughput

ThreadsMongoDBCouchbase Server
10561K110K
14065K141K
17567K154K
21070K170K
24574K193K
280Exceeded Max Latency238K
315Exceeded Max Latency245K
350Exceeded Max Latency252K

Results – Workload A Latency

ThreadsMongoDBCouchbase Server
105Read: 1.42ms
Write: 2.05ms
Read: .78ms
Write: .76ms
140Read: 2.01ms
Write: 2.97ms
Read: .79ms
Write: .78ms
175Read: 3.16ms
Write: 3.54ms
Read: .89ms
Write: .88ms
210Read: 3.5ms
Write: 4.49ms
Read: .93ms
Write: .92ms
245Read: 4.19ms
Write: 5.38ms
Read: .92ms
Write: .91ms
280Exceeded Max LatencyRead: .92ms
Write: .92ms
315Exceeded Max LatencyRead: 1.06ms
Write: .99ms
350Exceeded Max LatencyRead: 1.22ms
Write: 1.22ms

Benchmark Configuration

A benchmark, if you care, should answer the following questions:

  • What was the hardware configuration?
  • How was the operating system configured?
  • What were the database and client versions?
  • How were the databases configured?
  • How was YCSB configured?

 

So, what configuration is this United Software Associates benchmark missing?

  • The number of nodes
  • The replica configuration (# of replicas
  • The replication configuration (async or sync)
  • The persistence configuration (async or sync)
  • The size of the values
  • The size of the data set
  • The size of the cache (per node)
  • The size of the total data set (including replicas)
  • The percentage of primary data resident in memory
  • The results of all runs
  • The number of threads for all published runs

 

In addition, the GitHub repository referenced in this United Software Associates white paper does not include the client configuration for MongoDB, Cassandra, or Couchbase Server.

And what about the MongoDB routers and config servers?

Were they deployed? If so, where?

I don’t know because this benchmark does not include all of the configuration. The first benchmark included most of the configuration, but there were a number of mistakes. With this benchmark, they may have fixed the mistakes, but they failed to include all of the configuration.

Bottom Line

It's important for benchmarks to be credible. Anyone should be able to reproduce a benchmark and validate its results. All of the configuration must be provided and, preferably, it should be performed on cloud infrastructure. If not, vendors can manipulate the configuration to make unfair comparisons look fair.

That's not helpful to anyone.

Resources

MongoDB + United Software Associates Benchmark
White Paper | Code

Couchbase + Avalon Consulting, LLC Benchmark
White Paper | Code

Discuss on Hacker News

Posted by Shane Johnson, Director, Product Marketing, Couchbase

Shane K Johnson was the Director of Product Marketing at Couchbase. Prior to Couchbase, he occupied various roles in developing and evangelism with a background in Java and distributed systems. He has consulted with organizations in the financial, retail, telecommunications, and media industries to draft and implement architectures that relied on distributed systems for data and analysis.

Leave a reply