In this post, I went into some detail about replications between Sync Gateway (SG) instances in Couchbase Mobile (CBM). I wrote a simple Java application with a Swing-based UI to illustrate a working example. I run the app along with two instances of SG to demonstrate everything.

This diagram shows the flow in the application and the communications with the SG pair.

Data flow diagram

An editing pane allows us to enter JSON data. That data gets saved to the Couchbase Lite (CBL) local storage. We have two replications set up. One pushes data from the CBL database to the Sync Gateway instance I called the primary instance. I’ll refer to it simply as Primary. Primary replicates itself to the backup SG instance (hereafter called Backup).

To see this happen, I monitor the _changes feed from both Primary and Backup.

Everything runs together on my Mac. You can set up a complete CBM stack all running on your development machine. This makes initial development easier. It also helps understand how all the pieces fit together and work. I’ll blog about this more in the future.

Let’s walk through the configurations and the code.

Configuring Sync Gateway

Backup

I run each SG instance with its own JSON configuration file. The configuration for Backup is a little simpler. Here it is.

Since we run two copies of SG, we have to make sure they use different network ports. By default, SG uses ports 4984 and 4985. The first couple of lines for Backup set it to listen on ports 5984 and 5985 instead. There’s nothing special about those ports. Just make sure to pick ones that aren’t used by other apps on your machine.

The rest of the config file causes SG to log everything, sets up to serve data from a special-purpose in-memory database (indicated by the keyword “walrus”), and allows open access to anyone via the “GUEST” user. This is a nice piece to remember for early development. Walrus lets you use SG standalone, without a server backend. Using GUEST with all access means you can get things running without worrying about authentication and channel set up. Note by default SG logs to stderr. You can add an option to have it log to a file instead.

Primary and Inter-instance Replication

We set up the push replication in the config for Primary. It has some of the same boilerplate parameters.

The “replications” section is where the interesting stuff happens. Notice the entry is an array. You can specify any number of replications. Here we just have one.

To specify a replication, you need to provide a source and a target. Recall all replications are one-way.

The “source” parameter indicates we want to replicate the database identified as “db”. You can see the section under “databases” where we set this up with Walrus.

The target can be another database hooked to the same SG instance, or you can specify an HTTP URL like we did here. Use the name of the remote database for the path portion of the URL (/db in this case).

Running Sync Gateway

I saved the configurations in two files, primary_gateway_config.json and backup_gateway_config.json. You can run SG manually from a command line. You can find information about downloading and running SG for different platforms here.

I start Backup first. Otherwise Primary puts out a lot of messages about failures connecting to perform the replication.

The Java App

I wrote the Java app using Swing for the UI, using IntelliJ IDEA for layout. Here’s what the app looks like on startup.

DBExplorer on startup

To use it, paste JSON into the leftmost text pane. Click the Save button. That stores a new JSON document in the CBL database. Syncing is off to begin. Click the Start button to enable it.

The middle text pane monitors the Primary changes feed. The rightmost pane monitors the Backup changes feed.

With sync enabled, every time you save a document you should see both changes feeds respond. Here’s what that looks like.

DBExplorer after saving a document

You can find the source for the project on GitHub. (Please note this sample highlights basics. It is not production quality code.) Let’s examine a few key pieces.

There are four classes. Most of the UI is in DBExplorer.java. DBService.java is a fairly typical helper class. Take a look there to see how replications are set up between CBL and SG. Runtime.java exists mostly to have a shared instance of a Jackson JSON ObjectMapper.

SGMonitor.java takes care of monitoring the two changes feeds. I use the OkHttp client library from Square to make the REST calls.

To monitor the changes feed, I use what’s known as longpolling. With longpoll, if there’s no new data to transfer, the HTTP call doesn’t return immediately. Instead it waits until new data gets sent (or it times out). The connection is then closed. This means you need a loop to keep checking for change records. Note the changes records are sent as JSON. The results get bundled as an array, so you can have more than one record per response.

To simplify creating the HTTP request, I create a URL builder with the parameters needed for every call added.

Reading this out, we’re connecting to the _changes endpoint, using a feed type of longpoll, and setting the timeout to 0 (0 means never time out).

I set up a SwingWorker thread to run the loop. It builds an HTTP request, fires it off, and waits for the response.

The final trick here is in telling Sync Gateway what data you need. Every SG _changes response contains a “last_seq” property. This property is just an integer that helps track which records a client has already received. It is the responsibility of the client to keep track of the data it has processed. Once you’ve handled a response, you need to add or update the “since” parameter to future requests. SG then knows to pass only newer changes.

Here’s the latter part of the loop.

I use the Jackson ObjectMapper to parse the change record. I then add “since” as a query parameter with the value extracted from the last_seq property. That sets up the URL for the next request. The publish call takes care of updating the text area in the UI.

That’s it.  The entire project can be run on one machine.  This will let you explore Couchbase Lite, Sync Gateway, and replications in detail.  Enjoy.

Postscript

Check out more resources on our developer portal and follow us on Twitter @CouchbaseDev.

You can post questions on our forums. And we actively participate on Stack Overflow.

You can follow me personally at @HodGreeley

Author

Posted by Hod Greeley, Developer Advocate, Couchbase

Hod Greeley is a Developer Advocate for Couchbase, living in Silicon Valley. He has over two decades of experience as a software engineer and engineering manager. He has worked in a variety of software fields, including computational physics and chemistry, computer and network security, finance, and mobile. Prior to joining Couchbase in 2016, Hod led developer relations for mobile at Samsung. Hod holds a Ph.D. in chemical physics from Columbia University.

Leave a reply