Blog Post

Couchbase Tools shipped with the Couchbase C Client Library (libcouchbase)

Sergey Avseyev of Couchbase Published

In this blog post, I will reveal some the less known features of the libcouchbase client library. It describes the libcouchbase tools which are collectively under cbc, the command originally written by Trond Norbye. You probably won’t use them for writing you high-level bindings or implementing your application, but they still useful in many ways.  They're a great example of libcouchbase usage, they're useful as diagnotic or debug tooling or you can even use them in your scripts.

Because there is a current release (1.0) of libcouchbase and a developer preview of the next release (1.1) which supports the new functionality coming in Couchbase Server 2.0, I will describe the 1.1 preview and will note which features are not accessible in current version.

Installation instructions may be found at http://www.couchbase.com/develop/c/current for current stable version and http://www.couchbase.com/develop/c/next for the developer preview of 1.1. Both of them work with current stable server version and latter has extra commands which will work only with new server.

After installation you will have the cbc command available and you can check the installed version:

$ cbc version
cbc built from: libcouchbase 1.1.0dp8
    using libcouchbase: 1.1.0dp8

Note that as of this writing, the currently released version is actually 1.1.0dp6, but packages for the most recent builds are available through the snapshot repository.  More on that at the end.

You could print out the all possible sub-commands using the helpsubcommand:

$ cbc help
Usage: cbc command [options]
command may be:
  help       show this help or for given command
  cat        output keys to stdout
  cp         store files to the cluster
  create     store files with options
  flush      remove all keys from the cluster
  hash       hash key(s) and print out useful info
  lock       lock keys
  unlock     unlock keys
  rm         remove keys
  stats      show stats
  verify     verify content in cache with files
  version    show version
Use 'cbc command --help' to show the options

The "help" command accepts the command name as an argument and shows the command's short description and all options the command can process, e.g. cbc help cp.

This command was added recently and may not be available in all versions. It is still possible to get list of commands in any version you have installed: just execute cbcwithout arguments and "cbc cp --help" correspondingly.

All commands are accessible either as subcommands or symlinks to the main binary, e.g. "cbc cp" and "cbc-cp". The latter form is useful in shell scripts.

Connecting to Cluster

All commands accept a common set of arguments like --host, --bucket, --user, --password, and --timeout. These arguments should be specified to every command which requires interaction with cluster which need special connection options, like bucket name or authorization.

The option --enable-timingswill set make cbc record timings for all operations and output then to STDERR.

By default it will use localhostfor --host, defaultfor --bucket, nothing for --user/--password, and finally 2500000microseconds for --timeout.

If your connection settings are not the defaults, but you still doesn’t like typing them each time, we have solution: just put all of the settings into a file at $HOME/.cbcrcand all libcouchbase tools will use this before starting. The format is pretty simple: all empty lines and lines starting with #are skipped. Here's my example config:

# this config changes connection defaults and sets the timeout to 3 seconds
uri=example.com:8091,example.org:8091
user=Administrator
password=s3cr3t
bucket=mybucket
timeout=3000000

There is also a way to override the .cbcrcvalues but honour command line arguments: set environment variables. Couchbase cbc tools look at the following variables at startup: COUCHBASE_CLUSTER_URI, COUCHBASE_CLUSTER_USER, COUCHBASE_CLUSTER_PASSWORD, COUCHBASE_CLUSTER_BUCKET.

cbc-cpcbc-cp: Copy File(s) to the Cluster

$ cbc help cp
Usage: cp [options] [arguments]

        -?      Print this help text (--help)
        -h val  Hostname to connect to (--host)
        -b val  Bucket to use (--bucket)
        -u val  Username for the rest port (--user)
        -P val  password for the rest port (--password)
        -T      Enable command timings (--enable-timings)
        -t val  Specify timeout value (--timeout)

This command copies file contents to the given cluster using filename for the key. For example the following command will store contents of the JSON file buzz.json:

$ cbc cp buzz.json
Stored "buzz.json" CAS:66f09fa920d0000

After storing you will see the status on standard error. The command cbc-cpuses the file name as the key name and file contents as a value. It is possible to store several files quickly, as the command will pipeline the operations:

$ cbc cp buzz.json barbie.json
Stored "buzz.json" CAS:6c55a2eae30d0000
Stored "barbie.json" CAS:bdf8a2eae30d0000

The command can also read file list from STDIN. For example you can feed output of the lscommand to cbc cp(note that filenames must be separated with a newline char):

$ ls *.json | cbc cp -
Stored "barbie.json" CAS:8eb154e522020000
Stored "buzz.json" CAS:95a67ee922020000

cbc-create: Create Keys with Specific Options

$ cbc help create
Usage: create [options] [arguments]

        -?      Print this help text (--help)
        -h val  Hostname to connect to (--host)
        -b val  Bucket to use (--bucket)
        -u val  Username for the rest port (--user)
        -P val  password for the rest port (--password)
        -T      Enable command timings (--enable-timings)
        -t val  Specify timeout value (--timeout)
        -f val  Flag for the new object (--flag)
        -e val  Expiry time for the new object (--exptime)
        -a      Fail if the object exists (--add)

This command creates the item in the cluster with the given options, like expiration time (--exptime), flags (--flag) or force it to use ADD command (--add) which will fail if the key already in the cluster. The command operates on a single key only and then waits for the value on STDIN (type a ^Dto insert the required EOF). For example:

$ cbc create -f 3735928559 hello
Hello, Couchbase!
Stored "hello" CAS:b79b729f70070000

cbc-cat: Print Keys to STDOUT

$ cbc help cat
Usage: cat [options] [arguments]

        -?      Print this help text (--help)
        -h val  Hostname to connect to (--host)
        -b val  Bucket to use (--bucket)
        -u val  Username for the rest port (--user)
        -P val  password for the rest port (--password)
        -T      Enable command timings (--enable-timings)
        -t val  Specify timeout value (--timeout)

This command simply writes the contents of the all keys to STDOUT like Unix command cat(1). Also it writes useful info to STDERR. For example, lets output the key helloset above:

$ cbc cat hello
"hello" Size 18 Flags:deadbeef CAS:853dcf95bc070000
Hello, Couchbase!

cbc-rm: Remove Key

$ cbc help rm
Usage: rm [options] [arguments]

        -?      Print this help text (--help)
        -h val  Hostname to connect to (--host)
        -b val  Bucket to use (--bucket)
        -u val  Username for the rest port (--user)
        -P val  password for the rest port (--password)
        -T      Enable command timings (--enable-timings)
        -t val  Specify timeout value (--timeout)

This command removes given key from the cluster:

$ cbc rm hello
Removed "hello"

cbc-hash: Perform Hash Calculation

$ cbc help hash
Usage: hash [options] [arguments]

        -?      Print this help text (--help)
        -h val  Hostname to connect to (--host)
        -b val  Bucket to use (--bucket)
        -u val  Username for the rest port (--user)
        -P val  password for the rest port (--password)
        -T      Enable command timings (--enable-timings)
        -t val  Specify timeout value (--timeout)

This command calculates the hash value of the key and locate the server which should store it, using the current cluster config. It also writes useful info about what vbucket is responsible for key, what the address is for that vbucket, the endpoint for requests for Couchbase Views and the list of replica nodes.

$ cbc hash foo bar baz
"foo"   vBucket: 115, Server: "172.16.16.193:12000", Couch API: "http://172.16.16.193:9500/default", Replicas: "127.0.0.1:11210", "172.16.16.193:12000", "172.16.16.193:12003"
"bar"   vBucket: 767, Server: "172.16.16.193:12009", Couch API: "http://172.16.16.193:9503/default", Replicas: "127.0.0.1:11210", "172.16.16.193:12000", "172.16.16.193:12003"
"baz"   vBucket: 36, Server: "172.16.16.193:12000", Couch API: "http://172.16.16.193:9500/default", Replicas: "127.0.0.1:11210", "172.16.16.193:12000", "172.16.16.193:12003"

cbc-stats: Dump Cluster Statistics

$ cbc help stats
Usage: stats [options] [arguments]

        -?      Print this help text (--help)
        -h val  Hostname to connect to (--host)
        -b val  Bucket to use (--bucket)
        -u val  Username for the rest port (--user)
        -P val  password for the rest port (--password)
        -T      Enable command timings (--enable-timings)
        -t val  Specify timeout value (--timeout)

This command returns tab-separated tuples of the server statistics. First is the node hostname with port, then stat key and stat value.

To show a simple scripting example, this is how you can calculate used memory on the whole cluster (this cluster has 5 nodes):

$ cbc stats  | awk '/mem_used/ { print($1 ": " $3); sum += $3 } END {print("total: " sum)}'
192.168.1.3:12000: 20369582
192.168.1.3:12003: 22086752
192.168.1.3:12006: 21664064
127.0.0.1:11210: 27053336
192.168.1.3:12009: 22086752
total: 113260486

The stats output contains not only statistics from the live cluster but also its configuration parameters. You can get the maximum and default time (in seconds) for key locks (which we'll want to know soon):

$ cbc stats | grep getl
127.0.0.1:11210 ep_getl_default_timeout 15
127.0.0.1:11210 ep_getl_max_timeout     30
192.168.1.3:12000       ep_getl_default_timeout 15
192.168.1.3:12000       ep_getl_max_timeout     30
192.168.1.3:12003       ep_getl_default_timeout 15
192.168.1.3:12003       ep_getl_max_timeout     30
192.168.1.3:12009       ep_getl_default_timeout 15
192.168.1.3:12009       ep_getl_max_timeout     30
192.168.1.3:12006       ep_getl_default_timeout 15
192.168.1.3:12006       ep_getl_max_timeout     30

cbc-lock: Lock Key

$ cbc help lock
Usage: lock [options] [arguments]

        -?      Print this help text (--help)
        -h val  Hostname to connect to (--host)
        -b val  Bucket to use (--bucket)
        -u val  Username for the rest port (--user)
        -P val  password for the rest port (--password)
        -T      Enable command timings (--enable-timings)
        -t val  Specify timeout value (--timeout)
        -e val  Expiry time for the lock (--exptime)

This command locks the specified keys, making all mutation requests generate an error. It allows you to specify the expiration time ("--exptime"). You can inspect cluster default and maximum timeouts with the "cbc-stats" command as described above.

$ cbc create hello
Hi
Stored "hello" CAS:57fa8b8c01020000
$ cbc rm hello
Failed to remove "hello":
Invalid arguments
$ sleep 20
$ cbc rm hello
Removed "hello"

The cbc-lockcommand allows multiple keys.

cbc-unlock: Unlock Key

$ cbc help unlock
Usage: unlock [options] [arguments]

        -?      Print this help text (--help)
        -h val  Hostname to connect to (--host)
        -b val  Bucket to use (--bucket)
        -u val  Username for the rest port (--user)
        -P val  password for the rest port (--password)
        -T      Enable command timings (--enable-timings)
        -t val  Specify timeout value (--timeout)

This command unlocks given keys. It accepts pairs of key and CAS values.  If they match, the server releases the lock.

$ cbc lock -e 30 *.json > /dev/null
"barbie.json" Size:139 Flags:0 CAS:900ccb8ea2030000
"buzz.json" Size:93 Flags:0 CAS:e79acb8ea2030000
$ cbc unlock barbie.json 900ccb8ea2030000 buzz.json e79acb8ea2030000
Unlocked "barbie.json"
Unlocked "buzz.json"

cbc-verify: Verify Keys

$ cbc help verify
Usage: verify [options] [arguments]

        -?      Print this help text (--help)
        -h val  Hostname to connect to (--host)
        -b val  Bucket to use (--bucket)
        -u val  Username for the rest port (--user)
        -P val  password for the rest port (--password)
        -T      Enable command timings (--enable-timings)
        -t val  Specify timeout value (--timeout)

This command verify the keys' consistency. It compares the contents of the keys with corresponding files. If all is ok, it just return zero code at exit and prints nothing. So more interesting the case when something wrong. Let’s remove buzz.jsonand add something to barbie.json:

$ cbc cp *.json
Stored "barbie.json" CAS:59c1a52165050000
Stored "buzz.json" CAS:e17da62165050000
$ cbc rm buzz.json
Removed "buzz.json"
$ echo "garbage" >> barbie.json
$ cbc verify *.json
Incorrect size for: "barbie.json"
Failed to get "buzz.json": No such key
$ echo $?
1
 
As you can see, there are a number of pretty useful utilities in "cbc".  In fine UNIX tradition, you can build upon them with your own scripts and logic.  Of course, it's all Open Source as well, so feel free to have a look at the code and maybe contribute some new ideas over on github.
 
To get the very latest code right after it goes through code review, including everything you see in this blog as of this writing, add one of our snapshot repositories and update your libcouchbase package.
 
Thanks!