The Couchbase Mobile 2.5 release introduced extensive stats reporting capabilities on the Sync Gateway.The stats provide key insights into the health of your Couchbase Mobile deployment and constitues an integral part of any deployment.
In this post, we discuss how you can use Prometheus, an open source monitoring and alerting platform for monitoring your Sync Gateway nodes and Grafana for visualizing the stats. In an upcoming related post, we will discuss how you can setup monitoring with Prometheus on a Couchbase Mobile Kubernetes cluster.

Background

Sync Gateway Stats Reporting

Sync Gateway stats are reported in JSON format and is available through the _expvar endpoint via the Sync Gateway Admin REST interface.
These are the categories of stats that are reported.

Prometheus

Prometheus is an open source systems monitoring and alerting platform and hosted by Cloud Native Computing Foundation. At the core of it is the Prometheus Server that is responsible for polling “Prometheus targets” for stats and storing it as time series data. Prometheus targets are statically configured or can be discovered by Prometheus.

Grafana

Grafana is an open source data visualization and alerting platform. It supports Prometheus as a data source and can be used to build comprehensive dashboards.

Introducing the Prometheus Sync Gateway Exporter

In order for Prometheus to monitor the Sync Gateway, we need a “Prometheus target” corresponding to the Sync Gateway. This target is the Sync Gateway Exporter, herein referred to as “the Exporter”. Simply put, the Exporter is responsible for exporting Sync Gateway stats to Prometheus metrics.

Deployment Architecture

Tying all this together, a typical deployment would like the following –

  • Sync Gateway Exporter The Sync Gateway Admin REST API is, by default, only exposed on localhost. This configuration is strongly recommended in production enviornments as well. As a result, since the Exporter polls the Sync Gateway Admin API for stats, the Exporter would need to be on the same host/node as the Sync Gateway. Also,the Sync Gateway stats are reported on a per-node basis. So in a cluster of two or more Sync Gateway nodes, each Sync Gateway node must have it’s own Exporter.
  • Prometheus Server continually polls the Exporter for stats, which in turn polls the Sync Gateway REST endpoint. The server uses the rules defined in the rules.yaml to push alerts to the Alerting Manager.
  • Grafana service polls the Prometheus Server for stats and graphs it on a web-based dashboard that can be accessed via the web browser.

Installation

In the remainder of the post, we will walk through the steps to setup a Couchbase Mobile cluster for monitoring with the Exporter, Prometheus Server and Grafana. While the instructions are intended for a development environment, you can easily customize these for a production environment.

We will be using docker, which will make it extremely simple to get up and going.

Downloading Exporter Source code

The Sync Gateway Exporter is open source couchbaselabs project and available on GitHub. It is compatible with Couchbase Sync Gateway 2.5. To that end, while we will strive to keep it up-to-date with the latest release of Sync Gateway, we cannot guarantee that. But the good news is that it is open source and includes detailed instructions on how you can extend it to support additional stats.

Download the Exporter source from github

Deploying Couchbase Mobile Cluster

All commands in this section must be executed from the root folder of the cloned github repo.

If you already have a Couchbase Mobile cluster running on docker, you can skip this step and move on to the “Deploying Sync Gateway Exporter” step. Just remember to substitute the docker network name corresponding to your existing setup in the instructions below.

Creating a docker network

When using docker, it is recommended to run all the components on same docker network.
Create a docker network with the name “demo”

Deploying Couchbase Server

We will use a special docker image of the Couchbase Server v6.0.1 that includes a preinstalled sample bucket called “TravelSample” as well as a preconfigured RBAC user with “Application Access” that would enable Sync Gateway to connect to the Couchbase Server.
If you prefer, you can install the vanilla docker image of the latest version of Couchbase Server and manually configure it with a bucket and Sync Gateway RBAC user.

Deploying Sync Gateway

The Sync Gateway will be launched with the config file named sync-gateway-config.json that is located in the “testdata” folder of the cloned repo.

Try it out

Confirm that your Sync Gateway is up and running and that the _expvar endpoint is accessible by running the following curl command. If things are working as expected, you should see a whole lot of stats output to the console.

Deploying Sync Gateway Exporter

The docker image of the Sync Gateway Exporter is available at couchbasesamples/sync-gateway-prometheus-exporter:latest.

The sgw.url config option is of particular interest. This points to the Sync Gateway server. If you are working with an existing deployment of Couchbase Mobile, make sure you set the sgw.url to point to the Sync Gateway node in your setup.

The Exporter binds to 0.0.0.0 by default. You can override that using --web.listen-address option.

Try it out

Confirm that the exporter is up and running with the following curl command.

If things are working, there will be a bunch of metrics output to the console. Confirm that the “sgw_up 1”.

Deploying Prometheus Server

Now that we have the Exporter exporting stats from the Sync Gateway, we will setup Prometheus to poll the Exporter for metrics.

Exploring the prometheus.yml file

The prometheus.yml specifies the config with which the Prometheus server is launched. It is located in the “testdata” folder of the cloned repo. Let’s examine the contents of this file.

  • The scrape_interval specifies the polling interval. You can adjust it per your needs
  • The rule_files specifies the location of the Prometheus Rules file. The Prometheus rules file,sync-gateway.rules.yml is located at testdata/rules/ folder in the cloned repo. The rules are used by Prometheus to trigger alerts. We have provided a baseline set of rules as a starting point. You can customize the rules as needed. The rules file is mounted at /etc/prometheus/rules when the docker container is launched.
  • The targets specifies the Prometheus targets which is the Exporter. You could have multiple exporters if you have multiple Sync Gateways. In that case, you can statically specify all the Exporter endpoints or configure Prometheus to discover the targets.

Installing docker image

We will deploy the official docker image prom/prometheus with the prometheus.yml file included in the cloned repo.

Try it out

Open the URL http://localhost:9090/graph in a web browser.
– Click on the “Insert metric at Cursor” button to see list of available Sync Gateway metrics.
– Click on “Status”->“Targets” to see the list of exporters and their stats
– Click on “Alerts” menu button to see the status of the alerts

Deploying Grafana

While you could use Prometheus’s web UI for visualizing the stats, we will be using Grafana as you can create some very compelling dashboards with it and Grafana integrates well with Prometheus.

  • We will install the official grafana/grafana:6.2.0 docker image. The grafana/data folder in the cloned repo contains the grafana.db that is used for persisting the dashboard and other metadata. This is volume mounted at var/lib/grafana

  • A default “Couchbase Sync Gateway” dashboard called dashboard.jsonnet is available in the cloned repo as a jsonnet file. Make the grafana target to generate the corresponding dashboard.json file that can then be imported into Grafana. The dashboard graphs all the metrics exported by the Exporter. You can customize it per your needs.

Try it out

  • Open the URL http://localhost:3000 in a web browser. You should see the login screen. Log in with the default credentials of “admin” and password of “admin”. You can change it after initial login.
  • Next step would be to add “Prometheus” as the “Data Source” and import the “Sync Gateway dashboard” json file that was generated earlier. You could do this manually by following the menu options. We have simplified this process and provided a script that will do all that for you!
    Run the script from the root of your cloned repo using command below by running make on the grafana-dev target. This does the following
    • (Re)Generates the dashboard.json dashboard from the dashboard.jsonnet file
    • Uses Grafana API to add Prometheus as data source
    • Uses Grafana API to load the dashboard.json generated in the previous step

  • Once the script succesfully executes, you should refresh the Grafana web UI on your web browser. You will see the “Couchbase Sync Gateway Dashboard” in list of available dashboards. The dashboard graphs all the stats reported by Sync Gateway. You can customize the dashboard.
  • Click on the “Couchbase Sync Gateway Dashboard” to see the stats. You can filter it by Sync Gateway (if you have more than one) or database

That’s it! You have succesfully setup monitoring with Prometheus and Grafana. You can now drive replications with Couchbase Lite clients and monitor it. The default Sync Gateway dashboard is a starting point. You can customize the dashboard, either by editing the dashboard.jsonnet file or directly through the Grafana UI.

TL;DR

Instead of running each container independently, you can do a multi-container launch using docker-compose.
The repo includes a docker-compose.yml file that you can use to launch all the containers including Couchbase Server, Sync Gateway, Exporter, Prometheus and Grafana . There are some caveats. You can learn more about it here

What’s Next

Using the Sync Gateway Exporter, you can export Sync Gateway stats to Prometheus and visualize it using visualization tools such as Grafana. This vastly simplifies the monitoring of your Couchbase Mobile clusters. In addition to Exporter, we have provided a default Grafana dashboard that you can customize. customizing the dashboard itself, you can also customize the rules for alerts. While we discussed the installation using docker containers, you could also install the binaries directly.
In the next blog, we will discuss how you can deploy the Exporter in a cluster running Kubernetes. So stay tuned for that!
If you have questions or feedback, please leave a comment below or feel free to reach out to me at Twitter @rajagp or email me priya.rajagopal@couchbase.com.  The Couchbase Forums are another good place to reach out with questions.

Acknowledgements

I would like to thank Carlos Becker for his work on building the first version of the Sync Gateway Exporter.

Posted by Priya Rajagopal , Principal Product Manager

Priya Rajagopal is a Principal Product Manager for Couchbase focusing on the Couchbase Mobile and IoT platform. She has been professionally developing software for 20 years in several technical leadership positions, with the last 8 years focussed on mobile technologies. She is a co-inventor on 22 patents spanning areas of networking and platform security.

Leave a reply