Bruce Lindsay once said, “There are three things important in the database world: Performance, Performance, and Performance”.  Most enterprise architects know, as we progress in database features and architectures, it’s important to measure performance in an open way so they can compare the TCO reliably.

YCSB did a great job of benchmarking datastores serving the “Cloud OLTP” applications. These data stores were simple with simple get, put, delete operations.  The original YCSB benchmark consists of a simple insert, update, delete, and scan operations on a simple document of 10 key-values; workloads are defined with a mix of these operations with various percentages.

JSON databases like Couchbase and MongoDB have a more advanced data model with scalars, nested objects, arrays,  arrays of objects, arrays and arrays of objects;  JSON databases have more sophisticated query language, indexes, and capabilities. In addition to CRUD operations, applications routinely use the declarative query languages in these databases to search, paginate, run reports.  So, to help architects to evaluate platforms effectively, we need an additional benchmark to measure these capabilities in addition to the basic CRUD operations.

YCSB paper states: We also hope to foster the development of additional cloud benchmark suites that represent other classes of applications by making our benchmark tool available via open source. In this regard, a key feature of the YCSB framework/tool is that it is extensible—it supports easy definition of new workloads, in addition to making it easy to benchmark new systems.

This benchmark extends YCSB to JSON databases by extending existing operations to JSON and then defining new operations and new workloads.

Here’s the outline.

  1. Introduction
  2. Data Model
  3. Benchmark Operations
  4. Benchmark Workloads
  5. YCSB-JSON implementation
  6. How to run YCSB-JSON?
  7. References
1. Introduction

YCSB was developed to measure the performance of scalable NoSQL key-value datastores. YCSB infrastructure does that job well.   YCSB uses a simple flat key-value. Couchbase uses a JSON model, which customers use to massively interactive applications.  We’ve built and are building features into the product to enable customers to build these applications effectively. We need performance measurements for these use cases.

There are additional databases supporting JSON model: MongoDB, DocumentDB, DynamoDB, RethinkDB, Oracle NoSQL.  When running YCSB on JSON databases (Couchbase, MongoDB, etc), the driver simply stores and retrieves strings in the JSON key-value structure. All of these databases require a new benchmark to measure processing of rich structure of JSON (nested objects, arrays) and operations like paging, grouping, aggregations.

The purpose of YCSB-JSON is to extend the YCSB benchmark to measure JSON database capability to cover these two things:

  1. Operations representative of massively interactive applications.
    • Operations on the JSON data model, including nested objects, arrays.
  2. Create workloads that represent operations from these applications

See these customer use cases:

  1. Marriott built its reservation system on IBM Mainframe and DB2. They’ve run into cost, performance challenges as more and more customer try to browse the available inventory.  Systems on DB2 was originally built to take reservations from a phone-in system or from agents. The look to book ratio is low. Today, this ratio is high since the number of lookup requests has gone up exponentially.   This has increased the database cost dramatically as well.  Marriott moved all of its inventory data to Couchbase with continuous synchronization from its mainframe systems; web applications use Couchbase for the lookup/search operations.
  2. Cars.com is a portal to list and sell cars. They have the listing data on Oracle.  When they serve it up on the web, they not only have to present the basic car information but also provide additional insights like how many users are looking into a car or have saved it in their wish list. This is a way of increasing the engagement and sense of urgency.  All the data required for these interactive operations are stored in Couchbase.

More generally, the massively interactive applications include the following:

  1. Browse rooms availability, pricing details, amenities: (lookups by end customers)
  2. Browse information on car make/model or repair shops (enable web-scale consumers & partners)
  3. Provide information to the customer in context  (location-based services)
  4. Serve both Master Data and Transactional Data (at scale)

To support these requirements, the applications & databases do the following:

  1. Query offload from high-cost Systems of Record (mainframe, Oracle) databases
    • (Reservations & Revenue apps)
  2. Opening up back-office functions  to web / mobile access
    • (enable web users to check room details)
  3. Scale database/queries with better TCO  
    • (scale Mainframes with commodity servers)
  4. Modernize legacy systems with capabilities demanded by new collaboration/engagement applications
    • (Browse inventory, flight, room availability, departmental analysis)

The new benchmark needs to measure the performance of queries implementing these operations.

2. Data Model

We’ve taken customer and orders as two distinct collections of JSON documents.  Each order has a reference to its customer.

Below are the sample customer and order document.  This has been generated via the fakeit data generator.  This tool is available at: https://github.com/bentonam/fakeit

See the appendix for the YAML file used to define the data model and domain.

3. Benchmark Operations:

The first four operations are the same as standard YCSB, except this is on JSON documents. Rest of the operations are new.

  1. Insert: Insert a new JSON document.
  2. Update: Update a JSON document by replacing the value of one scalar field.
  3. Read: Read a JSON document, either one randomly chosen field or all fields.
  4. Delete: Delete a JSON document with a given key.
  5. Scan: Scan JSON documents in order, starting at a randomly chosen record key. The number of records to scan is randomly chosen (LIMIT)
  6. Search: Search JSON documents based on range predicates on 3 fields (customizable to n fields)
  7. Page: Paginate result set of a query with predicate on a field in the document.
    • All customers in zip with randomly chosen OFFSET and LIMIT in SQL, N1QL
  8. NestScan: Query JSON documents based on a predicate on a 1-level nested field.
  9. ArrayScan: Query JSON documents based on a predicate within the single-level array field.
  10. ArrayDeepScan: Query JSON documents based on a predicate within a two-level array field (array of arrays)
  11. Report: Query customer order details for customers in specific zipcode.
    • Each customer has multiple orders.
    • Order document has order details.
  12. Report2: Generate sales order summary for a given day, group by zip.
  13. Load: Data loading
  14. Sync: Data streaming and synchronization from another system
  15. Aggregate: Do some grouping and aggregation
Benchmark Operations implementation with Couchbase examples

The first four operations are the same as standard YCSB, except this is on JSON documents. Rest of the operations are new.

Couchbase implements YCSB in two modes.

KV=true.  KV stands for key-value. The simple YCSB operations INSERT, UPDATE, and DELETE can be implemented via KV APIs instead of queries.  Setting KV=true means, use the KV API and KV=false means use the N1QL (SQL for JSON) query. See the tutorial for N1QL at https://query-tutorial.couchbase.com

  1. Insert: Insert a new JSON document.

2. Update: Update a JSON document by replacing the value of one scalar field.

3. Read: Fetch a JSON document with a given key.

4. Delete: Delete a JSON document with a given key.

5. Scan: Scan JSON documents in order, starting at a randomly chosen record key. The number of records to scan is randomly chosen (LIMIT)

6. Page: Paginate result set of a query with predicate on a field in the document.

7. Search: Search JSON documents based on range predicates on 3 fields (customizable to n fields)

8. NestScan: Query JSON documents based on a predicate on a 1-level nested field.

9. ArrayScan: Query JSON documents based on a predicate within the single-level array field.

10. ArrayDeepscan: Query JSON documents based on a predicate within a two-level array field (array of arrays)

  • Get me list of all customers who have visited Paris, France.

KV=true:

Fetch the actual documents directly using KV calls from the benchmark driver.

KV=false:

11. Report: Query customer order details for customers in specific zipcode.

12. Report2: Generate sales order summary for a given day, group by zip.

13. Load: Data loading

  • LOAD 1 million documents
  • LOAD 10 million documents

14. Sync: Data streaming and synchronization from another system

  • Need to measure the data sync performance.
    1. Sync 1 million documents. 50% update, 50% insert
    2. Sync 10 million documents. 80% update, 20% insert
  • Ideally, this sync would be done from Kafka or some other connector pulling data from a different source.

15. Aggregate: Do some grouping and aggregation

 

4. Benchmark Workloads

Workloads are a combination of these operations.

To begin with, the workload definition can reuse the definitions the YCSB definition: workload-A through workload-E. Details are available at https://github.com/brianfrankcooper/YCSB/wiki/Core-Workloads.   We’ll need to define additional workloads with a combination of operations defined above.

Workload SA, same as workload A on the new model. Ditto with workload B through F.  We’ll call them SB through SF to differentiate from the workload B through F.

Workload Operations Record selection Application Example
SA — Update heavy Read: 50%

Update 50%

Zipfian Session store recording recent actions in a user session
SB — Read heavy Read: 95%

Update: 5%

Zipfian Photo tagging; add a tag is an update, but most operations

Update: 5% are to read tags

SC — Read only Read: 100% Zipfian User profile cache, where profiles are constructed elsewhere (e.g., Hadoop)
SD — Read latest Read: 95%

Insert 5%

Latest User status updates; people want to read the latest statuses
SE — Short ranges Scan: 95%

Insert: 5%

Zipfian/Uniform Threaded conversations, where each scan is for the posts in a given thread (assumed to be clustered by thread id)
SF — Read, modify, write Read: 50%

Write: 50%

Zipfian user database, where user records are read and modified by the user or to record user activity.
SG — Page heavy Page: 90%

Insert: 5%

Update:5%

Zipfian User database, where new users are added, existing records are updated, pagination queries on the system.
SH — Search heavy Search: 90%

Insert: 5%

Update: 5%

Zipfian User database, where new users are added, existing records are updated, search queries on the system.
SI — NestScan heavy Nestscan: 90%

Insert: 5%

Update: 5%

Zipfian User database, where new users are added, existing records are updated, nestscan queries on the system.
SJ — Arrayscan heavy Arrayscan: 90%

Insert: 5%

Update: 5%

Zipfian
SK — ArrayDeepscan heavy ArrayDeepScan: 90%

Insert: 5%

Update: 5%

Zipfian
SL — Report Report: 100%
SL — Report2 Report2: 100%
SLoad — Load Load: 100% Everything Data load to setup SoE
SN — Aggregate

(SN1, SN2)

Aggregation: 90%

Insert: 5%

Update: 5%

SMIX — Mixed workload Page:20%

Search:20%
Nestscan:15%

Arrayscan:15%

ArrayDeepscan:10%

Aggregate: 10%

Report: 10%

See below.
SSync — Sync Sync: 100%

Merge/Update: 70%

New/Insert: 30%

Continuous sync of data from other systems to systems of engagement. See below.

 

Example Configuration for YCSB/JSON Workload

Acknowledgments

Thanks to Raju Suravarjjala, Couchbase Senior director for QE and Performance, for pushing us to do this and the entire performance team for supporting this effort. The YCSB-JSON benchmark was developed in collaboration with Alex Gyryk, Couchbase Principal Performance Engineer.  He developed the data models for customer and orders used in this paper and implemented the operations and workloads in YCSB-JSON for Couchbase and MongoDB.  The YCSB-JSON implementation is available at: https://github.com/couchbaselabs/YCSB

Thanks to Aron Benton, Couchase Solution Architect, for developing an easy to use and efficient JSON data generator, fakeit.  He developed this prior to joining Couchbase. It is available at: https://github.com/bentonam/fakeit

Next part
In the next article on YCSB-JSON, Alex will explain the implementations of this benchmark for Couchbase and MongoDB.  The source code for the implementation is available at: https://github.com/couchbaselabs/YCSB
References
  1. Benchmarking Cloud Serving Systems with YCSB https://www.cs.duke.edu/courses/fall13/cps296.4/838-CloudPapers/ycsb.pdf
  2. JSON: http://json.org
  3. JSON Generator: http://www.json-generator.com/
  4. YCSB-JSON Implementation: https://github.com/couchbaselabs/YCSB
Appendix

YAML to generate the customer dataset.

 

Posted by Keshav Murthy

Keshav Murthy is a Senior Director at Couchbase of N1QL R&D. Previously, he was a Senior Director at MapR, Senior Architect for IBM, with more than 20 years experience in database design & development. He lead the SQL and NoSQL R&D team at IBM Informix. He has received two President's Club awards at Couchbase, two Outstanding Technical Achievement Awards at IBM. Keshav has a bachelors degree in Computer Science and Engineering from the University of Mysore, India, holds eight US patents and invented databases for systems of engagement.

Leave a reply