UPDATE: Starting in v2.8, Sync Gateway includes built-in Prometheus exporter support. Stats are exported in Prometheus-compatible format through the metrics endpoint. This implies that there is no longer the need to have a separate exporter. Read the Couchbase Documentation for more details on monitoring Sync Gateway v2.8+ using the new endpoint.

The Couchbase Mobile 2.5 release introduced extensive stats reporting capabilities on the Sync Gateway. The stats provide key insights into the health of your Couchbase Mobile deployment and constitutes an integral part of any deployment.

This is a step-by-step guide for using Prometheus (open source monitoring and alerting) with Sync Gateway clusters on Kubernetes. We will use Grafana for visualizing the stats.

It is often stated that Prometheus and Kubernetes share a “spiritual ancestry”. Both projects are hosted by the Cloud Native Computing Foundation. They are driven by common design goals, making Prometheus very well suited for monitoring Kubernetes deployments.

In a related post, we discussed setting up monitoring with Prometheus on a Couchbase Mobile cluster in an environment without Kubernetes. If you had followed along with that post, you can skip the initial sections and jump right to the Deployment Architecture section.

Background

Sync Gateway Stats Reporting

Sync Gateway stats are reported in JSON format and are available through the _expvar endpoint via the Sync Gateway Admin REST interface.
These are the categories of stats that are reported.

Prometheus

Prometheus is an open source system monitoring and alerting platform. At its core, Prometheus Server is responsible for polling “Prometheus targets” for stats and storing them as time-series data. Prometheus targets are statically configured or can be discovered by Prometheus.

Grafana

Grafana is an open-source data visualization and alerting platform. It supports Prometheus as a data source and can be used to build comprehensive dashboards.

Introducing the Prometheus Sync Gateway Exporter

In order for Prometheus to monitor the Sync Gateway, we need a “Prometheus target” corresponding to the Sync Gateway. This target is the Sync Gateway Exporter, herein referred to as “the Exporter”. Simply put, the Exporter is responsible for exporting Sync Gateway stats to Prometheus metrics.

Deployment Architecture

When we tie all this together, a typical deployment looks like the following:

Okay…so it looks a whole lot more complex than it actually is. It is in fact quite straightforward:

Sync Gateway Exporter Container

Exporter and Sync Gateway containers must be co-located in the same pod using the sidecar pattern. The Sync Gateway Admin REST API is, by default, only exposed on localhost. This configuration is strongly recommended in production environments as well. Because the Exporter polls the Sync Gateway Admin API for stats, it needs to be on the same host/node as Sync Gateway.

Also, the Sync Gateway stats are reported on a per-node basis. So in a cluster of two or more Sync Gateway nodes, each Sync Gateway node must have its own Exporter.

Sync Gateway Exporter Service

The Sync Gateway pods containing the Exporter must be exposed to other services on the cluster. In Kubernetes, you do that through a Kubernetes Network Service. The Sync Gateway Exporter Service exposes the Sync Gateway pods and makes it accessible to the Prometheus Service.

Prometheus Operator

Continually polls the Exporter for stats, which in turn polls the Sync Gateway REST endpoint. The server uses the rules defined in the rules.yaml to push alerts to the Alerting Manager. The simplest way to deploy Prometheus on the Kubernetes cluster is using the Prometheus Operator. The Prometheus Operator takes care of deploying the Prometheus stack including the Alerting Manager and Grafana.
The Operator acts on a number of Custom Resource Definitions(CRDs). The most interesting one is the Service Monitor. The Service Monitor is responsible for defining how services are to be monitored. In this case, the Sync Gateway pods that are exposed via the Sync Gateway Exporter Service are monitored.

Grafana

The service polls the Prometheus Server for stats and graphs it on a web-based dashboard that can be accessed via the web browser. To access the Grafana dashboard from outside the cluster, you will need to use port forwarding. You could also expose it using NodePort or the equivalent in Kubernetes. Again, the Prometheus Operator should take care of deploying the same.

Exporter Source Code

The Sync Gateway Exporter is open-source couchbaselabs project and available on GitHub. It is compatible with Couchbase Sync Gateway 2.5+. To that end, while we will strive to keep it up-to-date with the latest release of Sync Gateway, we cannot guarantee that. The good news is that it is open source and includes detailed instructions on how you can extend it to support additional stats.

Clone the Exporter source from Github and update relevant submodules. The repo includes all the yaml config files that would be used for the deployment.

The yaml config files referenced in this post are available in the “kubernetes” folder.

Installation

In the remainder of the post, we will walk through the steps to setup monitoring on a Couchbase Mobile cluster on the Kubernetes cluster. The instructions herein should apply to any managed Kubernetes deployment.

Deploying Couchbase Server Cluster

Follow the instructions here to deploy the latest version of the Couchbase Server cluster using Couchbase Autonomous Operator.

Deploying Sync Gateway Cluster

Follow the instructions here to deploy a Sync Gateway cluster on Kubernetes

Deploying Prometheus and Grafana

There are several choices to deploy Prometheus. In fact, most managed Kubernetes services may have options
One of the simplest options is to deploy using the Prometheus Operator using Helm.

The Prometheus chart will deploy the entire monitoring stack including the Prometheus server, Grafana, Alert Manager, etc.

Installing Helm

If you already have Helm running on your cluster, skip this step

Installing Prometheus

Install the Prometheus Operator by running this command from the root of the cloned repo.

  • Create a new configuration file named values.yaml using a text editor. The key thing is the default admin password using to log into the Grafana dashboard. You can change that after the initial login.
  • The values.yaml file is also available in the source repo.

  • Deploy the Helm Chart
    The Prometheus chart is deployed in the prometheus namespace

  • Verify the installation
    The output should show the Pods corresponding to the Prometheus stack

Deploying Sync Gateway Exporter

With the Prometheus monitoring stack set up, it is time to install the Exporter for monitoring.

The Exporter container will be deployed on the same pod as the Sync Gateway using the sidecar container pattern. To do that, we will update the Sync Gateway cluster Deployment to include the deployment of the Exporter container with couchbasesamples/sync-gateway-prometheus-exporter:latest image available on Docker Hub.

  • Open the sgw-deployment.yaml in a text editor and update it to include the Exporter container as shown below.

If you have a Sync Gateway import node, you will have to update sgw-deployment-import.yaml as well.

  • Once you update the yaml file, apply it to your cluster

  • Verify the status of the deployment. This sample cluster includes two Sync Gateway nodes.

  • You can view the details of Sync Gateway pod and check that it contains exporter container

Deploying Sync Gateway Exporter Service

Next, we deploy a Kubernetes service that exposes the Sync Gateway nodes and makes them available to other nodes in the cluster.

  • Create a new configuration file named svc.yaml using a text editor. The svc.yaml file is also available in the source repo.

  • Deploy the service

  • Verify the installation. The key point to note in the output is that the service points to the two Sync Gateway endpoints.

Setting up Prometheus Service Monitor

In order for Prometheus to discover the Sync Gateway Exporter service, we will have to define a ServiceMonitor. As discussed earlier, the Prometheus ServiceMonitor specifies how the services should be monitored. The Prometheus Operator generates the Prometheus scrape configuration based on the definition.

  • Create a new configuration file named servicemonitor.yaml using a text editor. The servicemonitor.yaml file is also available in the source repo.

  • Deploy the service

Configuring Prometheus Alerts

This is an optional step and only required if you want the Prometheus Alerting Manager to send alerts. Alerts are based on a set of configured alerting rules:

  • A sample rules.yaml file is available in the source repo. It includes some example rules defined per this specification. You can edit it to suit your environment.
  • Apply the rules

Tying it all together

Verifying Prometheus Monitoring

Now we will verify that the Prometheus ServiceMonitor is able to access and monitor the Sync Gateway Service.

  • Enable port forwarding from your local machine to the remote Prometheus server on port 9090. This is only needed for testing purposes. You do not need port forwarding enabled for normal operations.

Access the Prometheus webUI on your local browser

If everything goes well, you should see the Sync Gateway Exporter nodes listed in the Targets menu

Verifying Grafana

  • Enable port forwarding from your local machine to the remote Grafana server.

  • Access the Grafana dashboard webUI on your local browser.

  • Log in with the default credentials of “admin” and password of “admin”. You can change it after the initial login. The next step is to set it up with the Sync Gateway dashboard
  • A default “Couchbase Sync Gateway” dashboard called dashboard.jsonnet is available in the source repo as a jsonnet file.
  • The next step would be to add “Prometheus” as the “Data Source” and import the “Sync Gateway dashboard” JSON file that was generated earlier. You could do this manually by following the menu options. We have simplified this process and provided a script that will do all that for you!
    First, ensure that Grafana service is running on port 3000. Run the script from the root of your cloned repo using the command below by running make on the grafana-dev target. This does the following:

    • Regenerates the dashboard.json dashboard from the dashboard.jsonnet file
    • Uses Grafana API to add Prometheus as a data source
    • Uses Grafana API to load the dashboard.json generated in the previous step

Viewing Grafana Stats

  • Once the script successfully executes, you should refresh the Grafana web UI on your web browser. You will see the “Couchbase Sync Gateway Dashboard” in the list of available dashboards. The dashboard graphs all the stats reported by Sync Gateway. You can customize the dashboard.
  • Click on the “Couchbase Sync Gateway Dashboard” to see the stats. You can filter it by Sync Gateway (if you have more than one) or database

That’s it! You have successfully set up monitoring of Couchbase Sync Gateway Kubernetes with Prometheus. You can now drive replications with Couchbase Lite clients and monitor them.

The default Sync Gateway dashboard is a starting point. You can customize the dashboard, either by editing the dashboard.jsonnet file or directly through the Grafana UI.

What’s Next

Prometheus is well suited for monitoring Kubernetes deployments. By using the Sync Gateway Exporter, you can export Sync Gateway stats to Prometheus and visualize them using visualization tools such as Grafana. This vastly simplifies the monitoring of your Couchbase Mobile clusters.

In addition to the Exporter, we have provided a default Grafana dashboard that you can customize. In addition to customizing the dashboard itself, you can also customize the rules for alerts.

For questions or feedback, please leave a comment below. Or feel free to reach out to me on Twitter @rajagp or email priya.rajagopal@couchbase.com.  The Couchbase Forums are another good place to reach out with questions.

Acknowledgments

I would like to thank Carlos Becker for his work on building the first version of the Sync Gateway Exporter.

 

Author

Posted by Priya Rajagopal, Director, Product Management

Priya Rajagopal is a Director of Product Management at Couchbase responsible for the product portfolio for mobile and edge computing. She has been professionally developing software for over 20 years in several technical and product leadership positions, with 10+ years focused on mobile technologies. As a TISPAN IPTV standards delegate, she was a key contributor to the IPTV standards specifications. She has 22 patents in the areas of networking and platform security.

Leave a reply