The Couchbase Mobile 2.5 release introduced extensive stats reporting capabilities on the Sync Gateway.The stats provide key insights into the health of your Couchbase Mobile deployment and constitues an integral part of any deployment.
This post is a step-by-step guide to how you can use Prometheus, an open source monitoring and alerting platform for monitoring your Sync Gateway cluster deployed on Kubernetes. We will use Grafana for visualizing the stats.

It is often stated that Prometheus and Kubernetes share a “spiritual ancestry”. Both projects are hosted by Cloud Native Computing Foundation and driven by common design goals, making Prometheus very well suited for monitoring Kubernetes based deployments.

In a related post, we discussed how you can setup monitoring with Prometheus on a Couchbase Mobile cluster on a non-Kubernetes environment. If you had followed along that post, you can skip the initial sections and jump right to the Deployment Architecture section.

Background

Sync Gateway Stats Reporting

Sync Gateway stats are reported in JSON format and is available through the _expvar endpoint via the Sync Gateway Admin REST interface.
These are the categories of stats that are reported.

Prometheus

Prometheus is an open source systems monitoring and alerting platform. At the core of it is the Prometheus Server that is responsible for polling “Prometheus targets” for stats and storing it as time series data. Prometheus targets are statically configured or can be discovered by Prometheus.

Grafana

Grafana is an open source data visualization and alerting platform. It supports Prometheus as a data source and can be used to build comprehensive dashboards.

Introducing the Prometheus Sync Gateway Exporter

In order for Prometheus to monitor the Sync Gateway, we need a “Prometheus target” corresponding to the Sync Gateway. This target is the Sync Gateway Exporter, herein referred to as “the Exporter”. Simply put, the Exporter is responsible for exporting Sync Gateway stats to Prometheus metrics.

Deployment Architecture

Tying all this together, a typical deployment would like the following –

Okay…so it looks a whole lot more complex than it actually is. It is in fact quite straightforward

  • Sync Gateway Exporter Container The key thing to note is that the Exporter container must be co-located with the Sync Gateway container in the same pod using the sidecar pattern. The Sync Gateway Admin REST API is, by default, only exposed on localhost. This configuration is strongly recommended in production enviornments as well. As a result, since the Exporter polls the Sync Gateway Admin API for stats, the Exporter would need to be on the same host/node as the Sync Gateway. Also,the Sync Gateway stats are reported on a per-node basis. So in a cluster of two or more Sync Gateway nodes, each Sync Gateway node must have it’s own Exporter.
  • Sync Gateway Exporter Service The Sync Gateway pods containing the Exporter must be exposed to other services on the cluster. In Kubernetes, you do that through a Kubernetes Network Service. The Sync Gateway Exporter Service exposes the Sync Gateway pods and makes it accessible to the Prometheus Service.
  • Prometheus Operator continually polls the Exporter for stats, which in turn polls the Sync Gateway REST endpoint. The server uses the rules defined in the rules.yaml to push alerts to the Alerting Manager. The simplest way to deploy Prometheus on Kubernetes cluster is using the Prometheus Operator. The Prometheus Operator takes care of deploying the Prometheus stack including the Alerting Manager and Grafana.
    The Operator acts on a number of Custom Resource Definitions(CRDs). The most interesting one is the Service Monitor. The Service Monitor is responsible for defining how services are to be monitored. In this case, the Sync Gateway pods that are exposed via the Sync Gateway Exporter Service are monitored.
  • Grafana service polls the Prometheus Server for stats and graphs it on a web-based dashboard that can be accessed via the web browser. In order to access Grafana dashboard outside via a client browser outside the cluster, you will need to set up port forwarding or expose it using NodePort or equivalent in Kubernetes. Again,the Prometheus Operator should take care of deploying the same.

Exporter Source Code

The Sync Gateway Exporter is open source couchbaselabs project and available on GitHub. It is compatible with Couchbase Sync Gateway 2.5+. To that end, while we will strive to keep it up-to-date with the latest release of Sync Gateway, we cannot guarantee that. The good news is that it is open source and includes detailed instructions on how you can extend it to support additional stats.

Clone the Exporter source from github and update relevant submodules. The repo includes all the yaml config files that would be used for the deployment.

The yaml config files referenced in this post are available in the “kubernetes” folder.

Installation

In the remainder of the post, we will walk through the steps to setup monitoring on a Couchbase Mobile cluster on Kubernetes cluster. The instructions herein should apply to any managed Kubernetes deployment.

Deploying Couchbase Server Cluster

Follow the instructions here to deploy latest version of Couchbase Server cluster using Couchbase Autonomous Operator.

Deploying Sync Gateway Cluster

Follow the instructions here to deploy Sync Gateway cluster on Kubernetes

Deploying Prometheus and Grafana

There are several choices to deploy Prometheus. In fact, most managed Kubernetes services may have options
One of the simplest options is to deploy using the Prometheus Operator using Helm.

The Prometheus chart will deploy the entire monitoring stack including the Prometheus server, Grafana, Alert Manager etc.

Installing Helm

If you already have Helm running on your cluster, skip this step

Installing Prometheus

Install the Prometheus Operator by running this command from the root of the cloned repo.

  • Create the a new configuration file named values.yaml using a text editor. The key thing is the default admin password using to log into Grafana dashboard You can change that after initial login. The values.yaml file is also available in the source repo.

  • Deploy the Helm Chart
    The Prometheus chart is deployed in the prometheus namespace

  • Verify the installation
    The output should show the Pods corresponding to the Prometheus stack

Deploying Sync Gateway Exporter

Now that our Prometheus monitoring stack is setup, the only remaining piece is to install the Exporter and set it up so it can be monitored.
The Exporter container will be deployed on the same pod as the Sync Gateway using the sidecar container pattern. To do that, we will update the Sync Gateway cluster Deployment to include the deployment of the Exporter container with couchbasesamples/sync-gateway-prometheus-exporter:latest image available on Docker Hub.

  • Open the sgw-deployment.yaml in a text editor and update it to include the Exporter container as shown below.

If you have a Sync Gateway import node, you will have to update sgw-deployment-import.yaml as well.

  • Once you update the yaml file, apply it to your cluster

  • Verify the status of the deployment. This sample cluster includes two Sync Gateway nodes.

  • You can view the details of Sync Gateway pod and check that it contains exporter container

Deploying Sync Gateway Exporter Service

Next we deploy a Kubernetes service that exposes the Sync Gateway nodes and makes it available to other nodes in the cluster.

  • Create the a new configuration file named svc.yaml using a text editor. The svc.yaml file is also available in the source repo.

  • Deploy the service

  • Verify the installation. The key point to note in the output is that the service is points to the two Sync Gateway endpoints.

Setting up Prometheus Service Monitor

In order for Prometheus to discover Sync Gateway Exporter service, we will have to define a ServiceMonitor. As discussed earlier, the Prometheus ServiceMonitor specifies how the services should be monitored. The Prometheus Operator generates Prometheus scrape configuration based on the definition.

  • Create the a new configuration file named servicemonitor.yaml using a text editor. The servicemonitor.yaml file is also available in the source repo.

  • Deploy the service

Configuring Prometheus Alerts

This is an optional step and only required if you want the Prometheus Alerting Manager to send alerts based on a set of configured alerting rules.

  • A sample rules.yaml file is available in the source repo. It includes some example rules defined per this specification. You can edit it to suit your environment.
  • Apply the rules

Tying it all together

Verifying Prometheus Monitoring

Now we will verify that the Prometheus ServiceMonitor is able to access and monitor the Sync Gateway Service.

  • Enable port forwarding from your local machine to the remote Prometheus server on port 9090. This is only needed for testing purposes. You do not need port forwarding enabled for normal operations.

Access the Prometheus webUI on your local browser

If everything goes well, you should see the Sync Gateway Exporter nodes listed in the Targets menu

Verifying Grafana

  • Enable port forwarding from your local machine to the remote Grafana server.

  • Access the Grafana dashboard webUI on your local browser.

  • Log in with the default credentials of “admin” and password of “admin”. You can change it after initial login. The next step is to set it up with the Sync Gateway dashboard
  • A default “Couchbase Sync Gateway” dashboard called dashboard.jsonnet is available in the source repo as a jsonnet file.
  • Next step would be to add “Prometheus” as the “Data Source” and import the “Sync Gateway dashboard” json file that was generated earlier. You could do this manually by following the menu options. We have simplified this process and provided a script that will do all that for you!
    First ensure that Grafana service is running on port 3000. Run the script from the root of your cloned repo using command below by running make on the grafana-dev target. This does the following

    • (Re)Generates the dashboard.json dashboard from the dashboard.jsonnet file
    • Uses Grafana API to add Prometheus as data source
    • Uses Grafana API to load the dashboard.json generated in the previous step

  • Once the script succesfully executes, you should refresh the Grafana web UI on your web browser. You will see the “Couchbase Sync Gateway Dashboard” in list of available dashboards. The dashboard graphs all the stats reported by Sync Gateway. You can customize the dashboard.
  • Click on the “Couchbase Sync Gateway Dashboard” to see the stats. You can filter it by Sync Gateway (if you have more than one) or database

That’s it! You have succesfully setup monitoring of Couchbase Sync Gateway Kubernetes with Prometheus. You can now drive replications with Couchbase Lite clients and monitor it. The default Sync Gateway dashboard is a starting point. You can customize the dashboard, either by editing the dashboard.jsonnet file or directly through the Grafana UI.

What’s Next

Prometheus is well suited as the monitoring platform of choice on Kubernetes deployments. Using the Sync Gateway Exporter, you can export Sync Gateway stats to Prometheus and visualize it using visualization tools such as Grafana. This vastly simplifies the monitoring of your Couchbase Mobile clusters. In addition to the Exporter, we have provided a default Grafana dashboard that you can customize. In addition to customizing the dashboard itself, you can also customize the rules for alerts.

If you have questions or feedback, please leave a comment below or feel free to reach out to me at Twitter @rajagp or email me priya.rajagopal@couchbase.com.  The Couchbase Forums are another good place to reach out with questions.

Acknowledgements

I would like to thank Carlos Becker for his work on building the first version of the Sync Gateway Exporter.

 

Posted by Priya Rajagopal , Principal Product Manager

Priya Rajagopal is a Principal Product Manager for Couchbase focusing on the Couchbase Mobile and IoT platform. She has been professionally developing software for 20 years in several technical leadership positions, with the last 8 years focussed on mobile technologies. She is a co-inventor on 22 patents spanning areas of networking and platform security.

Leave a reply