November 7, 2009

SASL memcached now available!

memcached SASL support

After the initial announcement by Dustin Sallings that the memcached server now has SASL support, (which is a feature in the newly released 1.4.3, announced just today) this past week has also resulted in a flurry of other developments, particularly in the client realm

: A couple weeks ago Dustin added support for SASL in spymemcached (Java) and as of this week, Trond Norbye added SASL support to libmemcached, now making it possible to take advantage of the memcached server's SASL support for your applications that use memcached! So now you probably want to know: "How can I use these new features?". This post will attempt to show you how you can get SASL working with memcached, as well as explain the mechanics of just what SASL is and how it works with memcached to provide authentication that your applications can take advantage of.

What is SASL?

SASL stands for Simple Authentication and Security Layer, which provides you a means of adding authentication support to connection-based protocols, such as the memcached client binary protocol in this case. The way SASL works is that there is a function (API) call that is added to the client that provides a functionality for identifying and authenticating a given user. SASL also provides the means to implement protection throughout the entire connection. Once a user is identified and authenticated, SASL provides a security layer between the client protocol and connection.

How is SASL implemented for memcached?

Until the the binary protocol was added as well as the SASL implementation, memcached had no authentication layer whatsoever. Any client could connect to memcached perform any operation at will. It was up to the memcached user (usually a web developer) to make sure that their network was locked down and that their application was designed securely. Then the binary protocol was added o memcached which made for a more efficient and compact client-server connection. This made it possible to also add SASL support to memcached. With SASL support, functions were added to the server that made is so a client connection was forced to authenticate before the complete connection could continue. This of course required a client that sent the authentication information using SASL. At first, only spymemcached client (java) was support, but as already mentioned in the introduction, libmemcached now also has SASL support. Basically, SASL uses some sort of storage for user credentials-- it can be LDAP or SQL database. In this example, the user credentials are stored in a database file on the server that memcached will be running on. In order for a user to be able to connect to, memcached server won't allow the user to connect unless they provide their credentials, and those credentials must match the user credentials stored in the SASL database. More information on the SASL specifics of the implementation for memcached can be found here.

How do I install SASL with memcached?

Well, the first thing you need to do is get the latest memcached server. One place you'll find this is the memcached github repository or memcached.org, explained in the next step. You also will need a few prerequisites on your server such as sasl development libraries and as well as SASL utilities such as saslpasswd. On Ubuntu, for instance, the packages are installed as:

sudo apt-get -f install libsasl2-2 sasl2-bin libsasl2-2 libsasl2-dev libsasl2-modules

Not to forget, the usual requirements for memcached: libevent Just search accordingly for your packages according to the OS you are using.

How do I set up SASL memcached?

Now you will need to obtain SASL memcached. The code is now officially released from memcached.org wget http://memcached.googlecode.com/files/memcached-1.4.3.tar.gz tar xvzf memcached-1.4.3.tar.gz Or as always it is available via Github

git clone git://github.com/memcached/memcached.git

Now compile and install memcached with SASL enabled:

cd memcached-1.4.3
./configure --enable-sasl
make test
sudo make install

Next, you will set up the SASL database and memcached application configuration file. There are two ways to do this: system-wide or for a specific user.

System-wide SASL database file:

For a system-wide SASL set-up, you will need set up a memcached.conf file to point to the SASL database file which contains the user and password information in /usr/lib/sasl2/memcached.conf, named per application. An example of this is:

mech_list: plainlog_level: 5

Note: on Ubuntu, this database file is already in existence. If you have to have one created, make sure whatever directory you specify for the location of the sasl database exists prior to running the next command) To add a user to the SASL database file:

sudo saslpasswd2 -c -a memcached

As you can see, -a specifies the application, 'memcached', which needs to match the name you gave the conf file in the previous step 'memcached.conf'. When you run this saslpasswd2 command, you will be prompted for a password and password verification as shown below:

patg@ishvara:~/code-dev/memcached-dustin$ sudo saslpasswd2 -c -a memcached capttofu Password:Again (for verification):

You can verify that the user you created now exists. To do so, run:

sudo sasldblistusers2

-or-

sudo sasldblistusers2 -f /etc/sasldb2

You'll see an output such as:

patg@ishvara:~/code-dev/memcached-dustin$ sudo sasldblistusers2test1@ishvara: userPassword

User-specific SASL database file:

The first thing you need to ensure is that you set a very important environment variable, SASL_CONF_PATH whenever you run memcached. In this example, this path will be set to /home/patg/sasl.

export SASL_CONF_PATH=/home/patg/sasl

Make sure to create whatever directory path you specify! Next, you will set up the memcached.conf file for SASL (named per application). In this example, the contents are:

mech_list: plainlog_level: 5sasldb_path: /home/patg/sasl/sasldb2

Next you will have to create the database file that you just specified in the previous step in your memcached.conf file.

sudo saslpasswd2 -c -a memcached -f /home/patg/sasl/sasldb2 capttofu

Notice that the -a flag specifies the name of the application memcached, which must match to the name of the config file you specified in the previous example above, memcached.conf. Also notice that this differs from the system-wide example because you have to specify the SASL database file in this example with the -f flag. When you run saslpasswd2, you will be prompted to enter the password and password verification, as shown in the example below:

patg@ishvara:~$ saslpasswd2 -c -a memcached -f /home/patg/sasl/sasldb2 capttofuPassword:Again (for verification):

Now to run sasldblistusers2 to verify that you in fact added the user. As with saslpasswd2, you will have to specify the SASL database file

patg@ishvara:~$ sasldblistusers2 -f /home/patg/sasl/sasldb2capttofu@ishvara: userPassword

Running SASL-enabled memcached

Using either a system-wide sasl database file or user-level SASL database file, you will need to start memcached with the appropriate flags. When memcached runs with SASL enabled, the binary protocol is used, text protocol turned off. In this example, a user-level SASL database file is used. As the user 'patg', the following was done:

export SASL_CONF_PATH=/home/patg/sasl/usr/local/bin/memcached -S -vvv

Now to use memcached, you will need a SASL-enabled client, which as announced in this post, spymemcached (Java) and libmemcached support now. For simplicity (at least for me!) I hacked up one of the tests that come with memcached, binary-sasl.t, and renamed it patg.t (Reminds me I need to add SASL support to Cache::Memcached!). I also had to export an environment variable that the test suite code uses to state that you use an existing running memcached server:

export T_MEMD_USE_DAEMON=localhost:11211

Then I ran my test:

patg@ishvara:~/code-dev/memcached-dustin$ perl t/patg.t1..20returning handleok 1 - started the serverok 2 - Proper version: 0ok 3 - list_mechs CRAM-MD5 PLAINok 4 - this fails to authenticateok 5 - error code matchesok 6 - this fails to authenticateok 7 - error code matchesok 8 - this fails to authenticateok 9 - error code matchesok 10 - this fails to authenticateok 11 - error code matchesok 12 - bad mechok 13 - bad authok 14 - authenticatedok 15ok 16 - somevalue = somevalueok 17ok 18ok 19 - somevalue = somevalueok 20

Of interest and worth pointing out, notice the list_mechs test. This test lists what your SASL authentication mechanisms your server isconfigured for. What is of interest to me here is that I see my running server authenticating, since I started the test with -vvv. A failed authentication:

<30 0x00 0x00 0x00 0x00authenticated() in cmd 0x21 is true30: going from conn_parse_cmd to conn_nreadmech: ``PLAIN'' with 23 bytes of datasasl result code: -13Unknown sasl response: -13>30 Writing an error: Auth failure.

A successful authentication:

authenticated() in cmd 0x21 is true30: going from conn_parse_cmd to conn_nreadmech: ``PLAIN'' with 16 bytes of datasasl result code: 0

A successful GET using authentication:

authenticated() in cmd 0x00 is true30: going from conn_parse_cmd to conn_nread<30 GET x> FOUND KEY xL 0 9somevalueofu>30 Writing bin response:

The server is working! Now to try out SASL memcached using libmemcached with SASL support.

Installing libmemcached

The first thing to install libmemcached with Trond's new SASL functionality is to clone his tree from Launchpad. You will need the bazaar revision control system for this. For Ubuntu, it is:

apt-get install bzr

Next, clone the tree:

bzr clone lp:~trond-norbye/libmemcached/sasl_rfe_462250cd sasl_rfe_462250/sh config/bootstrap./configure # note, you do not have to use --enable-sasl because SASL will be compiled by defaultmakemake testsudo make install

This installs libmemcached. Now, how do you use it? Well, you will have to write programs to utilize it. For my enjoyment, I wrote a simple C program that uses the information Trond provided on his post. I called it sasl_test.c which you can find on Northscale's downloads. The crux of what it contains is shown below. I have a main function that contains these important lines:

/* initialize the client connection to use SASL */ if (sasl_client_init(NULL) != SASL_OK) { fprintf(stderr, "Failed to initialize sasl library!\n"); return 1; }

 memcached_st *memc = memcached_create(NULL); /* set the sasl callbacks for sasl auth */ memcached_set_sasl_callbacks(memc, sasl_callbacks); memcached_server_st *servers = memcached_servers_parse(servers_list); memcached_server_push(memc, servers); memcached_server_list_free(servers);

 /* you have to use binary protocol to use SASL */ memcached_behavior_set(memc, MEMCACHED_BEHAVIOR_BINARY_PROTOCOL, 1);

 /* set a value */ rc= memcached_set(memc, key, keylen, value, vallen, 0, 0); if (rc == MEMCACHED_SUCCESS) printf("setting key %s value %s\n", key, value); assert(rc == MEMCACHED_SUCCESS);

 /* get a value */ retval= memcached_get(memc, key, keylen, &retlen, (uint32_t)0, &rc); if (rc == MEMCACHED_SUCCESS) printf("fetched key %s value %s\n", key, retval); assert(rc == MEMCACHED_SUCCESS);

 /* free the connection */ memcached_free(memc); sasl_done();

Then, per Trond's instructions (lifted from his page), added callbacks, which I declare in another file I have sasl_test.h

static int get_username(void *context, int id, const char **result, unsigned int *len);static int get_password(sasl_conn_t *conn, void *context, int id, sasl_secret_t **psecret);

static sasl_callback_t sasl_callbacks[] = {{ SASL_CB_USER, &get_username, NULL}, { SASL_CB_AUTHNAME, &get_username, NULL}, { SASL_CB_PASS, &get_password, NULL}, { SASL_CB_LIST_END, NULL, NULL}};

Then I defined these in callbacks sasl_test.c (notice the password is set according to what was added to the sasl db file):

static char *username = "capttofu";static char *passwd = "s3kr1t";

static int get_username(void *context, int id, const char **result, unsigned int *len){ if (!result || (id != SASL_CB_USER && id != SASL_CB_AUTHNAME)) { return SASL_BADPARAM; }

 *result= username; if (len) { *len= (username == NULL) ? 0 : (unsigned int)strlen(username); }

 return SASL_OK;}

static int get_password(sasl_conn_t *conn, void *context, int id, sasl_secret_t **psecret){ static sasl_secret_t* x;

 if (!conn || ! psecret || id != SASL_CB_PASS) { return SASL_BADPARAM; }

 if (passwd == NULL) { *psecret = NULL; return SASL_OK; } size_t len = strlen(passwd); x = realloc(x, sizeof(sasl_secret_t) + len); if (!x) { return SASL_NOMEM; }

 x->len = len; strcpy((void *)x->data, passwd);

 *psecret = x; return SASL_OK;}

I compile this with the following command:

gcc -g -O0 -I/usr/local/include/libmemcached -lmemcached -lmemcachedutil -o sasl_test sasl_test.c

Then I can run the test!

./sasl_test localhost:11211 testkey testvalue

If I observe my SASL-enabled memcached server, I see that SASL authentication works. Connection:

<30 new binary client connection.30: going from conn_new_cmd to conn_waiting30: going from conn_waiting to conn_read30: going from conn_read to conn_parse_cmd

Authentication:

<30 Read binary protocol data:<30 0x80 0x20 0x00 0x00<30 0x00 0x00 0x00 0x00<30 0x00 0x00 0x00 0x00<30 0x00 0x00 0x00 0x00<30 0x00 0x00 0x00 0x00<30 0x00 0x00 0x00 0x00authenticated() in cmd 0x20 is true

Set, authenticated:

30: going from conn_read to conn_parse_cmd<30 Read binary protocol data:<30 0x80 0x01 0x00 0x07<30 0x08 0x00 0x00 0x00<30 0x00 0x00 0x00 0x18<30 0x00 0x00 0x00 0x00<30 0x00 0x00 0x00 0x00<30 0x00 0x00 0x00 0x00authenticated() in cmd 0x01 is true30: going from conn_parse_cmd to conn_nread<30 SET testkey Value len is 9> FOUND KEY testkey0 0 10test valueofu

Get, authenticated:

30: going from conn_read to conn_parse_cmd<30 Read binary protocol data:<30 0x80 0x0c 0x00 0x07<30 0x00 0x00 0x00 0x00<30 0x00 0x00 0x00 0x07<30 0x00 0x00 0x00 0x00<30 0x00 0x00 0x00 0x00<30 0x00 0x00 0x00 0x00authenticated() in cmd 0x0c is true30: going from conn_parse_cmd to conn_nread<30 GET testkey> FOUND KEY testkey0 0 9testvaluettofu>30 Writing bin response:

And it works! Now I can modify other libmemcached-based programs to use SASL. Of course, you could also use spymemcached for using SASL as well.

Summary

With this post, you should come away having a better understandingof what SASL is, and how you can use it with memcached. You should be fully able to now obtain, compile and install both the memcached server as well as the client, libmemcached with SASL enabled, as well as be able to start writing memcached applications that use libmemcached to use SASL. Have fun!

Comments