New in Couchbase Mobile v1.3: Inter-instance Replication in Sync Gateway

DNA undergoing replication.

Graphic courtesy of Madprime with permission under license CC BY-SA 2.0

Sync Gateway

Sync Gateway forms the “glue” between Couchbase Lite and Couchbase Server in the Couchbase Mobile stack.  It’s a secure web gateway that enables sync and data access over the web.

That’s one way to look at it, which doesn’t really do Sync Gateway justice.  You can intelligently use it to link together clients without any back end, for example.  In a previous post, I talked about OpenID Connect support.  Sync Gateway acts there to ease the whole authorization flow.  There’s more, but in this post I want to focus on a new capability added in version 1.3.

A Deeper Look at Syncing

Syncing (short for synchronizing) refers to keeping data consistent across two or more instances of a database.  Syncing can be a difficult problem.  Any time two writers try to make conflicting changes, the architecture has to deal with it.

Some databases simply ignore the issue, forcing a single-writer only use.  Others depend on detecting conflicts in real time and rejecting them.  These approaches don’t work in cases where different copies of the database can’t always communicate to coordinate.

A good sync architecture is a critical component of a full solution that addresses uses where a device may be on a slow network connection, or be disconnected entirely part of the time.  Sync Gateway implements a core piece of that architecture in Couchbase.

We refer to the full stack as Couchbase Mobile.  Mobile uses are an obvious case where offline performance can be important.  Really, though, you can use Couchbase Mobile, including Sync Gateway, in many other scenarios.  Couchbase Lite and Sync Gateway run on a broad range of platforms, and are typically easy to port to new ones.  This makes Couchbase Mobile useful for anything from desktop (or maybe I should say laptop) to IoT.

Here are some of the key feature to know about Sync Gateway replications in general:

• JSON configuration to specify replications
• Supports multiple replications running concurrently
• Can run both OneShot and Continuous replications
• Does not store anything persistently
• Stateless — can be interrupted/restarted anytime without negative side effects
• Filter replications using channels

Inter-instance Replication

Release 1.3 adds a new capability to Sync Gateway, the ability to replicate (sync) between Sync Gateway instances.  Each replication gets configured as a uni-directional flow between two endpoints.  This makes them quite flexible.

For example, a simple replication can specify two databases on the same Sync Gateway instance.  This might not seem that interesting, but because each database can have its own sync function, you might use one database as a feeder for another, allowing complex business logic to manage what gets passed through.

This diagram shows a more typical use case.  Here, we have two Couchbase Server clusters, and two Sync Gateway clusters.  The Sync Gateway clusters run a bi-directional replication (really, two uni-directional replications).  This could form the basis for a globally distributed system.  Standard network routing procedures would ensure clients connect to the nearest Sync Gateway, improving network performance.  For more details on replication in Sync Gateway, take a look at the documentation here.

Document Revisions and Conflicts

It’s important to understand how Couchbase Mobile handles conflict resolution.  CBM uses what’s known as a multiversion concurrency control.  You can think of CBM as storing not just one version of a document, but rather a tree of revisions.  Two disconnected writes updating the same original revision of a document create two new revisions.  Both exist in the tree.  Couchbase gives you several ways to resolve the conflict.  See the documentation here for more details.

More Resources

Adam Wiggins, co-founder of Heroku, wrote a great article on why syncing offers a better UX.

I’ll be posting a follow-up article soon illustrating a simple example of inter-instance replication.  You can find the code on github. (Please note, this is an intentionally simple example.  It is not intended as production quality code.)

Postscript