Not so long ago, we went to lunch at a Sushi joint on Castro street in Mountain View across from NorthScale’s modest HQ. On the way over we were talking about Japanese robatayaki and Dave told a story about an Oscar speech for a Japanese animator who works for a company named Robot.
I have plenty of other stories about the HQ, like describing a “Pink” day… but that’s not the purpose of this post…. It was over lunch that Dustin, Dave, Steve and I, right after the memcached 1.4.0 release, said we should just throw together an AMI with memcached bundled into it for people to use. We didn’t want it to be modified in any way from what you could get from memcached.org…. it’d just be built by some folks who work on and use memcached. It’d be designed to simply boot and use as much available memory on the system to run a nice big memcached instance. We know lots of developers and we know that some of them use EC2 instances to run memcached. We would just make it simpler to obtain. Then, I opened my mouth and said I thought I could throw together an even better AMI, which used some of the technology I’d worked with in the Sun Web Stack to simplify an EC2 deployment even further…. Thus began MrRoboto. As I said above, we know lots of developers, so we know how they use memcached. As people get started with memcached, they tend to run it from a terminal with “-vvv”. That level of verbosity is really only something a memcached developer could love. It’s a bit too much for your average developer using Rails, PHP or Java. Speak up about an idea at a startup, and you’ve just given yourself a software development project to run! The concept was we’d mash up a DTrace script, Dustin’s slosh server modified to run said script when a user asks for it, and a simple AJAX browser UI getting info to show it off to a user. Rather than try to describe it, have a look at this screencast:
memcached AMI with tools from NorthScale from Matt Ingenthron on Vimeo. Effectively, this is “-vvv” in a browser. It’s really, really basic for a first release, but we can think of a few places to grow from here now that the base is in place. While it’s not immediately apparent from this simple, but useful “keyflow”, we can do a lot of other interesting things. Some other experiments are already working. Other than Trond Norbye and myself, I don’t think many people have poked at memcached with DTrace even though it’s been in the code for over a year. I won’t go into huge detail on DTrace here, but the huge advantage you get for ‘free’ with DTrace is that it is simple and safe to use in production, and the overhead of a probe which is not enabled is nil. That’s exactly how we used it in this AMI. If you aren’t looking at it with a browser, there isn’t any overhead. Even if you are looking, the overhead is very low, and out of the critical path of memcached’s execution. In my humble opinion, the technology behind DTrace is awesome; it just needs some published examples and tools to help people ramp up. Thanks much to my colleagues Dustin Sallings for the slosh example, Rod Ebrahimi for helping to get the UI into shape and Steve Yen and Dave Neilsen for the brainstorming and enthusiasm behind the project. Also, thanks to Trond Norbye for authoring the DTrace probes, Chad Mynhier and Adam Leventhal for helping with my simple DTrace-fu issues and Rich McDougall/Jim Mauro and Brendan Gregg (initially of DTrace Toolkit fame and later of Fishworks analytics fame) for the tutilidge on DTrace over the years. More to come in future posts on project MrRoboto, but if this is something you find cool or useful… or you have an idea for something you’d like to see in a future MrRoboto AMI, please let us know! p.s.: the AMI is based on OpenSolaris. Amazon’s EC2 management console (currently in Beta) bugs are keeping it listed as “Other Linux”, which is the default for anything they can’t identify with a substring.
