“ Today we are happy to announce the General Availability of the Couchbase Connector 4.0 for Apache Kafka. The Connector allows you to easily build robust and reactive data pipelines that take advantage of stream processing connecting different datastores, applications, and services in real-time.”
Useful Links
Introduction
Couchbase Server is an open source, distributed, JSON document database. It exposes a scale-out, key-value store with managed cache for sub-millisecond data operations, purpose-built indexers for efficient queries and a powerful query engine for executing SQL-like queries.
Apache Kafka is a community distributed streaming platform capable of handling trillions of events a day. Initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log. Since being created and open sourced by LinkedIn in 2011, Kafka has quickly evolved from messaging queue to a full-fledged event streaming platform.
Couchbase Kafka Connector is an open source plugin for Kafka Connect framework that utilizes high-performance Database Change Protocol to publish documents to Kafka topics in near real-time. Since its inception in 2017, Couchbase Kafka Connector has enabled several customers providing functionality to stream, filter and transform data between Couchbase Server and Apache Kafka by providing Source and Sink connectors.
Availability
You can download the distribution by clicking on the download link available here.
Enhancement
Version 4.0 is a major update that changes how you configure and extend the Source and the Sink connector.
Major features includes
-
-
- Enhanced durability options (requires Couchbase Server 6.5).
- Standardizing configuration properties.
- Even workload distribution.
- More flexible API for extensions.
- Option to omit document contents.
- Packaged as a Confluent Hub component for easy installation.
- Support for Scopes and Collections (planned for Couchbase Server 7.0).
-
Migration
There are two areas that you might want to consider during Migration from its predecessor Version 3.x
-
- Source and Sink Configuration – Version 4.0 comes with standardizing names of configuration in a way that it’s more consistent and provides more appropriate default values. A shell script is made available to make the migration more seamless.
- Java API changes – If you have extended connectors behavior by writing custom SourceHandler or Filter it’s worth noting down that they have changed and are not backward compatible.
For more information visit our migration guide.
Next Steps
Why wait ? Try it out yourself and let us know if you have any questions or feedback. Drop by our developer community, we’d love to hear from you !
