Event-driven architecture with Solace Event Broker & Couchbase

The asynchronous event-driven architecture of Couchbase is the core of its design. Document mutations continuously stream to the Couchbase Index, Analytics, Eventing and Search services.

This architecture guarantees that the Couchbase services are updated in real-time and enables us to integrate Couchbase seamlessly into your event-driven architecture.

This blog post demonstrates how you can integrate Couchbase with an event broker. In this post, I use the popular Solace PubSub+ message broker, but integration with other messaging products, like Kafka, is very similar.

Architecture overview

Couchbase can interact with other systems using the Couchbase SDKs and various Couchbase Connectors, like the one we have for Kafka. For this integration, we use an alternate approach,  Couchbase Eventing Service. The Eventing Service allows you to promptly act on changes (mutations) to your data by triggering JavaScript functions. The Eventing Service natively supports cURL and allows you to call external REST services directly from the JavaScript listener.

We use this capability to send data to Solace PubSub+ using their REST messaging API. The messages are sent to a Solace Topic that its subscribers can consume.

In this integration, we implement a one-way POST to Solace, illustrated in the figure below. Here, the Couchbase Eventing function sends an HTTP POST with the document in the message body to the Solace PubSub+ event broker. The event broker acknowledges receipt of the message with an HTTP 200 OK response. The message is now in the hands of the Solace Event Broker, where consumers can subscribe to the messages on the Topic.

Figure 1. Document mutations in Couchbase trigger an event listener that sends the document to a Solace Topic via an HTTP POST. Consumers subscribe to the Topic.

Solace can further process the message, or subscribers can subscribe to the Topic via JMS, MQTT and other protocols. In this post, I use a REST Consumer. A microservice can consume the messages by providing an HTTP callback endpoint by creating a REST consumer endpoint in Solace. The callback endpoint is called when a new message is added to the Topic.

Sample application

To demonstrate this, we built an example scenario using a dataset that comes out of the box with Couchbase, the travel-sample bucket.

In our test scenario, we want to publish any changes to the hotel documents to Solace. As shown in this figure, a microservice will then subscribe and consume the changes.

The sample application parts are:

1. A Couchbase eventing function is triggered for every mutation to the hotel documents. The eventing function will then send the hotel document to a Solace Topic 
2. Solace Event Broker containing queue and topic as well as REST delivery configuration 
3. A REST consumer subscribed to the Solace Topic

Prerequisites

These three prerequisites are required to follow along with the sample application:

Couchbase 7 Enterprise Edition

You can run Couchbase as a single-node cluster on Docker on a local machine.

We create a single node Couchbase Cluster for development purposes (single-node clusters are not recommended for production use). Include the Eventing and Data services when setting up Couchbase on your one node cluster.

Solace PubSub+ Event Broker Cloud

Sign up for a free Solace cloud trial and create a Service/VPN. We will configure the VPN later on in the instructions.

REST Consumer

We need an HTTP endpoint that accepts HTTP POST requests. Since we are using the Solace PubSub+ Cloud, the REST consumer needs to be accessible from the internet.

Instead of hosting your own REST consumer, you can use a free service (like Beeceptor) that allows you to receive HTTP POST requests and acknowledges an HTTP 200 response. That’s everything we need to verify that the integration is working. If you plan to perform more extensive tests, you could write your own REST consumer microservice.

For this purpose, go to Beeceptor.com and configure an endpoint name. In this example, I use cbtest as the endpoint.

Capture the endpoint address: https://cbtest.free.beeceptor.com, which is needed later when configuring the receiving endpoint in Solace.

Configuration

With the prerequisites in place, it is time to set up the various services for the sample application.

Solace setup

First, we need to set up Solace to create the corresponding Topic, Queue, and REST delivery endpoint. After signing up to Solace cloud, make sure you have created a Service/VPN. 

Each of the following steps is illustrated with a screenshot to show how your configuration should look like as you follow along.

Setup queue and topic

Let’s configure the Topic to receive the messages we send from the Couchbase Eventing Service.

In the Solace administrative interface, go to Queues/Topic Endpoints and create a new Topic Endpoint called T/rest/pubsub using the default settings.

Next, go to Queues and create a queue called Q/couchbase/input. Keep the default values.

Now open the Q/couchbase/input queue in edit mode and choose the Subscriptions tab. Add a T/rest/pubsub topic subscription to the Q/couchbase/input queue.

With the queue and topic in place, the REST client can now subscribe to it.

Go to  Client Connections and switch to the REST tab. Create a new REST Delivery Endpoint with the name RDPCouchbase. Keep the default values and make sure to enable the new endpoint.

Navigate into the newly created endpoint and jump to the REST consumers tab.

Create a new REST Consumer with the name demoConsumer and enable the consumer.

Provide the remote host address for the consuming REST service. As described in the prerequisite, either provide your own endpoint or create a free test endpoint at Beeceptor.

Configure the remote port; we use Beeceptor with port 80 and keep the remaining default values.

Navigate to the Query Bindings tab in the RDPCouchbase REST delivery point and add the binding for the recently created Q/couchbase/input queue. Add the Post Request Target, which corresponds to the URL path of the REST receiver. I use /callback.

Verify the Solace REST configuration

To verify the Solace is configured properly and that REST messages can be consumed by Solace as well as sent to the REST client, run the following cURL command:

Replace the hostname and credentials with your environment’s configuration.

This concludes the Solace setup. We now have a Topic that can receive messages and a REST subscriber that provides a callback function.

Couchbase setup

With our Couchbase node running the Eventing and Data services, we can create the eventing function.

Install the travel-sample sample bucket (see the documentation here). 

Create a bucket named rr100 (for resident ratio 100%) and a scope named eventing. In this scope, create a collection named metadata.

rr100.eventing.metadata will be used by the eventing function to store its metadata. 

In Couchbase, prior to version 7, you had to create a dedicated bucket for the metadata. With version 7 we take advantage of scopes and collections support to create a dedicated collection for the eventing metadata information instead.

Navigate to the Eventing service and create a new eventing function by clicking Add function.

The event listener listens to the hotel collection in the inventory scope within the travel-sample bucket. This will trigger our event listener whenever a document is changed within the hotel collection while changes to documents in other collections or buckets are not picked up.

The rr100.eventing.metadata bucket is used to store the eventing functions metadata.

The curlSolaceRestUrl specifies the URL alias to the receiving Solace REST endpoint. Provide the full URL, including the URL path, representing the Topic that you created during the Solace setup. For example: https://mrd3aaxn3oc4d.messaging.solace.cloud:9443/T/rest/pubsub

The Solace endpoint requires authentication. We configure basic authentication and provide the credentials. 

(Note: There was a bug in Couchbase 7.0.X that can lose the password on your cURL credentials. Issues can be avoided by retying the password if you edit your Eventing Function’s settings and perform a “Save”.)

We can now implement the OnUpdate() JavaScript function with the eventing configuration in place. Couchbase will trigger this function for any document mutation in the hotel collection. 

In this simple example, we send all document mutations to Solace using the cURL support in the Eventing Service. In a real-world use case, we would typically apply filtering based on some document attributes and implement the OnDelete method to handle document deletion.

Test the sample application

Now it’s time to test the sample application:

1. Start the evt_sendto_solace event listener
2. Navigate to Documents in the Couchbase console
3. Select the travel-sample.inventory.hotel collection as shown here:
4. 
5. Open a hotel document, edit any document fields and save the changes. This document change will trigger an update event that our event listener picks up.
6. The event listener will now send the updated document to Solace.
7. We can now verify that our Solace REST consumer has received the updated document. Notice that Solace sent the document to the REST consumer using the /callback specified during the queue binding.

The document mutations to the hotel documents in the hotel collection are now published to Solace and then sent to the Solace subscribers.

Couchbase services rely on the internal Database Change Protocol (DCP) which supports the  eventually consistent view of your data.  DCP does not guarantee that you see all changes but it does guarantee you see the most recent.  For example if a document in travel-sample.inventory.hotel mutates quite fast some of the intermediate mutations can be deduplicated as such the Eventing Service will not see all the changes.

Thank you for joining me in this demonstration.

Conclusion

This article taught us to integrate Couchbase with the Solace event broker using the Couchbase Eventing service.

We see how Couchbase can seamlessly integrate into your event-driven architecture. You can now take full advantage of the quality of service aspects of the Solace platform with guaranteed and persistent messaging.

The Eventing function logic in our example is limited to a basic cURL POST request without adequate error handling. You will want to implement error & retry logic for a production-ready solution.

• Learn about the internals of Couchbase Eventing here
• Get started with a free cloud trial for Couchbase Capella