In the last two weeks, benchmarks sponsored by Couchbase and MongoDB were released that compared their flagship products versus each other and Cassandra. This is a good thing for the NoSQL industry, so long as the benchmarks are open, reproducible, and are not over-engineered to favor one solution over another. Competitive benchmarks can provide valuable information to developers and ops engineers who are evaluating various solutions.

Benchmarks Increasingly Important in Phase 2 of NoSQL Adoption: Mission Critical Deployments

The release of an increasing number of benchmarks isn’t surprising. During the first phase of NoSQL adoption, benchmarks were somewhat less important because most users were experimenting with NoSQL or using it on fairly lightweight applications that operated at small scale. In 2013 we entered phase two of NoSQL adoption, in which enterprises are deploying NoSQL for mission-critical applications operating at significant scale. The use of benchmarks is increasing during this phase, because performance at scale is critical for most of these applications. Developers and ops engineers need to know which products perform best for their specific use cases and workloads.

Different Benchmarks for Different Use Cases

It’s entirely legitimate for benchmarks to focus on use cases and workloads that align with the target market and “sweet spots” of the sponsoring vendor’s products.  That doesn’t make them invalid. It just points out the importance of highlighting what those use cases and workloads are so developers and ops engineers can assess whether the benchmark is applicable to their specific situations. 

For example, Couchbase is very focused on supporting enterprise-class mission-critical applications that operate at significant scale with mixed read/write workloads. As a result, our benchmarks run on clusters with many servers and reflect those workloads.  By contrast, MongoDB is presumably more focused on supporting applications that operate at much smaller scale and therefore sponsored a benchmark with a small amount of data running on a single server. Both are valid, but for completely different situations and users.

Keeping It Fair

To be useful, however, benchmarks need to be fair. Otherwise, they’re of little value to developers and ops engineers. Vendors may complain that a benchmark isn’t fair because it’s focused on a use case and workload that’s not a sweet spot for them. Those aren’t valid complaints. On the other hand, benchmarks need to make every effort to achieve an apples-to-apples comparison and, for example, use the most recent versions of software. Apples-to-apples comparisons can be difficult, because the architectures and operational setups of each product are so different, but significant effort should be made to achieve this. Using the right version of software should be very easy to achieve and should promptly be fixed when it isn’t.

Keeping It Transparent

Transparency implies at least two things: (1) Clearly communicating the use cases and workloads that are being measured, and (2) making the benchmarks open so others can reproduce them, verify the results, and modify them to align more closely with the specific use cases they care about.

A Sign of NoSQL Growth and Adoption

Vendors will continue to sponsor and publish benchmarks, and they’ll continue to gear them toward the use cases the vendor supports best. All of this is just another indicator of the rising importance of NoSQL, which is growing fast. According to a new report from Allied Market Research, the global NoSQL market is expected to reach $4.2 billion by 2020 – an annual growth rate of 35.1% from 2014 through 2020. When done fairly and transparently, competitive benchmarks can help enterprises choose the right product for their particular set of requirements. 

Posted by Bob Wiederhold

Bob served as President and CEO of Couchbase from 2010 to 2017. Until an acquisition by IBM in 2008, Bob served as chairman, CEO, and president of Transitive Corporation, the worldwide leader in cross-platform virtualization with over 20 million users. Previously, he was president and CEO of Tality Corporation, the worldwide leader in electronic design services, whose revenues and size grew to almost $200 million and had 1,500 worldwide employees. Bob held several executive general management positions at Cadence Design Systems, Inc., an electronic design automation company, which he joined in 1985 as an early stage start-up and helped to grow to more than $1.5 billion during his 13 years at the company. Bob also headed High Level Design Systems, a successful electronic design automation start-up that was acquired by Cadence in 1996.

Leave a reply