Welcome to the new Couchbase kafka connector! It provides functionality to direct stream of events from Couchbase Server (3.0 or later) to Kafka. It is still under development, so use with care and open issues if you come across them. Its issue tracker is located at https://issues.couchbase.com/browse/KAFKAC.  Thanks much to Shibi of PayPal who had written another connector based on an older interface; it inspired this one.

Getting it

You can find the project source code in github here. The developer previews are available through our own maven repository, the GA artifacts will be available on maven central. Here are the coordinates:

  • Group ID: com.couchbase.client
  • Artifact ID: kafka-connector
  • Version: 1.0.0-dp1

Usage

The usage of the library is pretty easy. Lets say we would like to receive every modification from the Couchbase Server and send to Kafka only document body (by default the connector serializes document body and metadata to JSON). To achieve that you need to define a filter class which allows only instances of MutationMessage to pass through:

And an encoder class, which takes the document value converts it to byte array:

That's essentially enough to setup Couchbase-Kafka bridge:

The couchbase1.vagrant and kafka1.vagrant addresses above are locations of Couchbase Server and Kafka correspondingly, which could be easily set up using provisioning scripts from env/ directory. Just navigate there and run vagrant up.

Posted by Sergey Avseyev, SDK Engineer, Couchbase

Leave a reply