Blog Post

Memcached, go-derper, Black Hat and an Amazon Web Services (AWS) Security Bulletin

Frank Weigel of Couchbase Published

If you are a user of memcached and have deployed instances on Amazon EC2, you may have received a message from Amazon over the weekend (we received one on 8/7/2010) indicating you may have a "Possible Insecure Memcached Configuration." Here's the body of the message we received:

We've sent you this email to let you know that we have observed that you may be running memcached in an insecure configuration. Specifically, we have noticed that you have at least one security group that allows the whole internet to have access to the port most commonly used by memcached (11211).

There has been a lot of recent attention by the security community about the lack of access controls on memcached and recently some exploits have been published. This has highlighted the importance of running with strict access controls. While we are not aware of any unauthorized access to your Amazon EC2 instances, we do believe you should have your technical team look at this immediately.

We suggest that you audit your security group settings and restrict access to only the instances and IP addresses that need access. Most users only authorize other Amazon EC2 instances to access their memcached server. If you need to access your memcached server from outside of Amazon EC2, you can also authorize just trusted addresses to access your security group.

If you need additional assistance, you can reach our Premium Support team by sending email to aws-security-support@amazon.com.

Regards,
The Amazon Web Services Team

Great email and service from the AWS team, and the suggested fix is spot on.

This posting is meant to provide some background on the issue and the alluded to "recent attention" the issue has received. The issue is relevant to all users of memcached, not just those deploying on Amazon EC2.

The vulnerability
The genesis of this bulletin was almost certainly the result of the development of go-derper by the team at sensepost, highlighted at the blackhat USA 2010 conference on July 30, 2010.

The highlighted vulnerability can be summarized as: if you deploy memcached on a server, leave the TCP port on which memcached is configured to listen (11211, by default) exposed to the Internet, leave the memcached ASCII protocol enabled, AND you are not using SASL authentication with the memcached binary protocol, then there is a trivial way for Bad Guys to retrieve and replace most of the contents of your cache. go-derper.rb is a simple Ruby application, built by sensepost, that can be used to exploit the vulnerability.

Eliminating the vulnerability
Let"s examine the vulnerability, clause-by-clause, and highlight what can be done to eliminate it, starting at the top:

"If you deploy memcached on a server,"

This may seem silly to consider, but there are actually options here. Not everyone needs to deploy and configure memcached on a server themselves in order to use the technology. If you are deploying memcached on a cloud platform, for example, you may simply leverage a pre-built image or even an add-on service.

We run the memcached add-on service for Heroku (itself run on Amazon infrastructure), the leading platform-as-a-service cloud provider for Ruby applications. Because we manage the memcached add-on, our deep expertise with memcached is implicitly brought to bear on behalf of the thousands of applications deployed on Heroku that leverage our memcached add-on.

Additionally, we are working closely with our friends at RightScale to make pre-configured memcached images available for those who want to deploy pre-configured memcached and membase instances on Amazon AWS.

If you are using one of these deployment options, we've ensured the configuration is secure.

"[if you] leave the TCP port on which memcached is configured to listen (11211, by default) exposed to the Internet,"

If you have deployed your own instance of memcached, either on your own equipment or in a cloud computing environment, then you need to ensure a firewall is protecting the system.

Amazon provides a rich set of capabilities for expressing and enforcing access control for instances running on EC2.

NorthScale co-founder Dustin Sallings also weighed in over the weekend; his blog provides great additional detail, especially regarding firewalling.

"[if you] leave the memcached ASCII protocol enabled,"

As built, the go-derper exploit depends on use of the ASCII protocol.

Memcached provides both an ASCII protocol and a binary protocol. The binary protocol was co-developed by NorthScale's own Trond Norbye while employed by SUN Microsystems.

The fact of this vulnerability is that it also exists in the binary protocol, but the binary protocol supports authentication and access control, providing a mechanism for securing the data.

The ASCII protocol, the original protocol developed for memcached, does not have any facility for authentication or access control, and thus is not suitable for hanging on the public Internet. This protocol was explicitly developed for use behind a firewall, as a "back-end," protected system.

In the unlikely event that you have some good reason to make the memcached port available to any host on the public Internet, but want to control access to the data, then you should disable ASCII protocol support (and enable SASL authentication on the binary protocol, as described next). The NorthScale distribution of memcached makes it easy to configure memcached to NOT bind the ASCII protocol listener to the memcached port.

"AND [if] you are not using SASL authentication with the memcached binary protocol,"

As mentioned above, the memcached binary protocol in recent releases of memcached does support authentication and access authorization via the SASL protocol.

The NorthScale distribution of memcached makes it very easy to leverage this capability. Creating a new "bucket" in our memcached distribution provides both multi-tenancy capability (allowing multiple applications to securely bind to a single memcached cluster) and serves as the vehicle for SASL credential binding. It is this capability that allows us to securely support thousands of memcached add-on users up on Heroku without running thousands of individual servers.

If you are using an older version of memcached (most linux distributions ship with antiquated versions of the software), and you need authenticated access support, you should look at a more recent version of the software. I certainly recommend our distribution.

The memcached historical context
The vulnerability is not surprising. Memcached was initially built by Brad Fitzpatrick for use at LiveJournal, in an environment where control over servers and network security was managed by a skilled team of system administrators. With many lines of defense in front of memcached, there was little need to build yet another layer of security into memcached itself, where the inevitable price would be development effort (effort better spent building blogging features) and performance (in an environment where many millions of memcached transactions are processed per day, and every single microsecond counts).

In a perfect world, every person developing and deploying software should fully understand the characteristics of all the underlying software infrastructure components on which their software is dependent; have a firm understanding of network security, policy formulation and policy enforcement; and regularly audit their operational environment while tracking emerging threats. Few systems would get deployed. In fact, I think it is fair to say that some of the most popular web applications on the Internet today would never have seen the light of day under those constraints.

In the real world, there is a lot of interesting software being developed and deployed by people who are not themselves, and who frequently do not have the resources to employ, experienced system administrators and network security specialists. They just want to get their software or service in the hands of as many users as possible, as quickly as they possibly can. If and when the billions of operations per second materialize, then the microseconds can be wrung out, hopefully by a competent team of system administrators which the organization can then attract, and afford.

An aside on cloud computing
Ultimately, this is one of the promises of cloud computing, as outlined in one of our white papers. Cloud computing is not just about transforming capital to operating expenses, or about leveraging service provider economies of scale. Managed hosting providers have been doing that for over a decade. Cloud computing ultimately enables software developers to develop and deploy software, without also building up expertise in system administration and network security. Ultimately the world is a better place as a result. More developers are empowered to build and deliver software solutions.

Amazon has demonstrated part of the value they provide to their customer base: they have tracked a newly highlighted vulnerability that is widely relevant (given the broad deployment of memcached) to their users, identified specific users at risk (possible given the metadata used to configure the virtual machine instances that ultimately underlie running systems on EC2) and notified them in a timely manner precisely how to deal with the problem. Serious value add.

Where do you get your memcached?
The team here at NorthScale provides the vast majority of the contributed source code to the memcached and membase open source projects. We respect the clearly expressed desire of the larger memcached development community that the core of memcached should remain raw, fast and best suited for those who know what they are doing.

We also make available commercially-supported, certified, less "raw" versions of those systems, making it easier for organizations to deploy, configure, secure and manage the software. While the memcached development community doesn't want the code base polluted with "ease of use" features, there is a much larger potential community of users of the software that will be better off with those features present. Same goes for things like replication and live cluster reconfiguration. Many users want these capabilities, but the core community would prefer to keep them out of memcached proper. We make them freely available in our distribution and make the source code available in related projects (e.g. http://github.com/northscale/bucket_engine).