Concurrency Behavior: MongoDB vs. Couchbase

Multi-User Testing

David Glasser of Meteor wrote a blog on an MongoDB query missing matching documents issue he ran into. It is straightforward to reproduce the issue on both MongoDB MMAPv1 and MongoDB WiredTiger engine. Here are his conclusions from his article (emphasis is mine)

Long story short…

This issue doesn’t affect queries that don’t use an index, such as queries that just look up a document by ID.
It doesn’t affect queries which explicitly do a single value equality match on all fields used in the index key.
It doesn’t affect queries that use indexes whose fields are never modified after the document is originally inserted.
But any other kind of MongoDB query can fail to include all the matching documents!

Here’s another way to look at it. In MongoDB, if the query can retrieve two documents using a secondary index (index on something other than _id) when concurrent operations are going on, the results could be wrong. This is a common scenario in many database applications.

Here is the test:

Create a Container: Bucket, table, or collection.
Load the data with a small dataset, say 300K documents.
Create an index on the field you want to filter on (predicates).
In one session, update the indexed field in one session and query on the other.

MongoDB Testing

Steps to reproduce the issue on MongoDB:

Install MongoDB 3.2.
Bring up mongod with either MMAPv1 or WiredTiger.
Load data using tpcc.py
python tpcc.py –warehouses 1 –no-execute mongodb
Get the count

> use tpcc

> db.ORDER_LINE.find().count();

299890

db.ORDER_LINE.ensureIndex({state:1});

MongoDB Experiment 1: Update to higher value

Setup the state field with the value aaaaaa and then concurrently update this value to zzzzzz and query for total number of documents with the two values ['aaaaaa','zzzzzz'] matching the field. When the value of the indexed field moves from lower (aaaaaa) to higher (zzzzzz) value, these entries are moving from one side of the B-tree to the other. Now, we’re trying to see if the scan returns duplicate value, translated to higher count() value.

> db.ORDER_LINE.update({OL_DIST_INFO:{$gt:””}}, {$set: {state:”aaaaaa”}}, {multi:true});

WriteResult({ “nMatched” : 299890, “nUpserted” : 0, “nModified” : 299890 })

> db.ORDER_LINE.find({state:{$in:['aaaaaa','zzzzzz']}}).count();

299890

> db.ORDER_LINE.find({state:{$in:['aaaaaa','zzzzzz']}}).explain();

{

“queryPlanner” : {

“plannerVersion” : 1,

“namespace” : “tpcc.ORDER_LINE”,

“indexFilterSet” : false,

“parsedQuery” : {

“state” : {

“$in” : [

“aaaaaa”,

“zzzzzz”

]

}

“winningPlan” : {

“stage” : “FETCH”,

“inputStage” : {

“stage” : “IXSCAN”,

“keyPattern” : {

“state” : 1

“indexName” : “state_1”,

“isMultiKey” : false,

“direction” : “forward”,

“indexBounds” : {

“state” : [

“[“aaaaaa”, “aaaaaa”]”,

“[“zzzzzz”, “zzzzzz”]”

]

}

“rejectedPlans” : [ ]

“serverInfo” : {

“host” : “Keshavs-MacBook-Pro.local”,

“port” : 27017,

“version” : “3.0.2”,

“gitVersion” : “6201872043ecbbc0a4cc169b5482dcf385fc464f”

“ok” : 1

}

Update statement 1: Update all documents to set state = “zzzzzz”

db.ORDER_LINE.update({OL_DIST_INFO:{$gt:””}},

{$set: {state: “zzzzzz”}}, {multi:true});

Update statement 2: Update all documents to set state = “aaaaaa”

db.ORDER_LINE.update({OL_DIST_INFO:{$gt:””}},

{$set: {state: “aaaaaa”}}, {multi:true});

3. Count statement: Count documents:(state in [“aaaaaa”, “zzzzzz”])

db.ORDER_LINE.find({state:{$in:['aaaaaa','zzzzzz']}}).count();

Time	Session 1: Issue Update Statement1 (update state = “zzzzzz”)	Session 2: Issue Count Statement continuously.
T0	Update Statement starts	Count = 299890
T1	Update Statement Continues	Count = 312736
T2	Update Statement Continues	Count = 312730
T3	Update Statement Continues	Count = 312778
T4	Update Statement Continues	Count = 312656
T4	Update Statement Continues	Count = 313514
T4	Update Statement Continues	Count = 303116
T4	Update Statement Done	Count = 299890

Result: In this scenario, the index does double counting of many documents and reports more than it actually has.

Cause: Data in the leaf level of B-Tree is sorted. As the update B-Tree gets updated from aaaaaa to zzzzzz, the keys in the lower end are moved to the upper end. The concurrent scans are unaware of this move. MongoDB does not implement a stable scan and counts the entries as they come. So, in a production system with many updates going on, it could count the same document twice, thrice or more. It just depends on the concurrent operations.

MongoDB Experiment 2: Update to lower value

Let’s do the reverse operation to update the data from ‘zzzzzz’ to ‘aaaaaa’. In this case, the index entries are moving from a higher value to a lower value, thus causing the scan to miss some of the qualified documents, shown to be undercounting.

Time	Session 1: Issue Update Statement2 (update state = “aaaaaa”)	Session 2: Issue Count Statement continuously.
T0	Update Statement starts	Count = 299890
T1	Update Statement Continues	Count = 299728
T2	Update Statement Continues	Count = 299750
T3	Update Statement Continues	Count = 299780
T4	Update Statement Continues	Count = 299761
T4	Update Statement Continues	Count = 299777
T4	Update Statement Continues	Count = 299815
T4	Update Statement Done	Count = 299890

Result: In this scenario, the index misses many documents and reports fewer documents than it actually has.

Cause: This exposes the reverse effect. When the keys with values zzzzzz is modified to aaaaaa, items go from the higher to the lower end of the B-Tree. Again, since there is no stability in scans, it would miss the keys that moved from the higher end to the lower end.

MongoDB Experiment 3: Concurrent UPDATES

Two sessions update the indexed field concurrently and continuously. In this case, based on the the prior observations, each of the sessions run into both overcount and undercount issue. The nModified result varies because MongoDB reports only updates that changed the value.

But the total number of modified documents is never more than 299980. So, MongoDB does avoid updating the same document twice, thus handling the classic halloween problem. Because they don’t have a stable scan, I presume they handle this by maintaining lists of objectIDs updated during this multi-update statement and avoiding the update if the same object comes up as a qualified document.

SESSION 1

> db.ORDER_LINE.update({state:{$gt:””}}, {$set: {state:”aaaaaa”}}, {multi:true});

WriteResult({ “nMatched” : 299890, “nUpserted” : 0, “nModified” : 299890 })