Hello folks, over the next couple of months we’ll be doing a series of blogs that will follow the development of the Couchbase .NET 2.0 SDK. In this post I will go over the high level architecture, some motivation and features you should expect, and what the overall goals are of the .NET SDK with respect to the other Couchbase clients: Node, PHP, Java, and C. Discussing the goals of the .NET SDK within the scope of the entire suite of Couchbase clients is an important step in unifying the API’s, so that developers have a consistent programming experience no matter which SDK they chose.

History and Motivation

First up, let’s talk about what are motivation is for re-writing the Couchbase .NET SDK. Note that for the most part, this is a complete rewrite from scratch…pretty much every component is being redesigned and redeveloped to better suit the needs of our customer and the user community at large. By now, most of us  have been warned that the single worst strategic mistake a software company can do, is rewrite working, functional code from scratch, but we here at Couchbase feel it’s the best decision given the history and state of the .NET 1.X SDK and the direction we want to go with the client.

A little history might be in order here. The current client is based off of the original Memcached .NET client: Enyim.Caching. It was written several years back and originally supported the Memcached Text protocol and eventually the Memcached Binary Protocol as well. In itself, it was designed to be a stand-alone API, not framework for creating other API’s. That being said, the current .NET Couchbase Client was built on top of a fork of the Enyim.Caching source; we forked because the original author, while a great supporter, no longer had the time needed to keep the project going. The Couchbase client added view support and other Couchbase Server specific features on top of the K/V stuff already supported by Enyim.Caching.

Since the client was originally based off a K/V based protocol, fitting in some of the Couchbase features has made for a bit of a thorny API. Furthermore, as more layers have been added the complexity has increased as well making supporting and maintaining the client more and more difficult. Additionally, since the client is really based off of .NET 3.5 (while we do build for 4.0, 3.5 is the LCD), we weren’t progressing with the current state-of-the-art .NET features like the powerful new Task (async/await) based asynchronous libraries/features and improvements to the networking API’s. Maintaining backwards compatibility and progressing the client is therefore impossible; we need to take a whole new direction.

Goals and Features

The goals/objectives of the Couchbase .NET 2.0 SDK are fairly simple:

  • Unification and consistency of the programming interface across all Couchbase SDK platforms (Java, .NET, PHP, Node, C, etc)
  • A flexible, test-driven design – extensibility at its core
  • Highly configurable – but easy to get up and running with the default configuration
  • Asynchronous I/O – a non-blocking model for network programming
  • A simple and easy to use programming interface
  • Minimalistic use of client/server connections
  • Improved documentation

Some likely features include:

  • Support for Cluster Carrier Configuration Publication (CCCP) – a new way of updating the clients configuration based on the clusters current state via a pull mechanism
  • Task-based Asynchronous Pattern (TAP) support for K/V and View operations
  • First class N1QL support – N1QL is the awesome new SQL-like query language in Couchbase
  • A LINQ provider implementation over the core API
  • Integration points with other .NET technologies like ASP.NET and Frameworks for quickly developing data-driven applications

High-Level Architecture

Over the last couple years, the SDK team at Couchbase have developed several Couchbase Server clients on different platforms and frameworks: .NET, Java, Node.js, etc. As each of these clients have been developed and evolved with changes in the Server, several patterns have emerged for developing a cluster aware client, or more specifically in Couchbase’s case, a “smart” client. In the goal of “unifying the programming interface”, we implement these patterns within each client. We will be discussing those patterns in detail in later posts.

Couchbase Client Components

These components make up a Couchbase client:

  • IO – Resource management and general network programming access
  • Configuration – Client and Server configuration
  • Operations – Key/Value and View operations on JSON documents: Get, Set, etc.
  • Cluster Map – Manages active nodes and where keys are stored across the cluster
  • Logging and Instrumentation – General logging, tracing and monitoring of client activity

Next Up

Well, that’s it for now. Next up we will be going over the design and development of one portion of the Configuration component: Server Configuration. The Server Configuration drives the state of the client which parallels the state of the cluster. A change in the cluster generally, means a change in the client state as well, which brings with it various complexity and issues which must be addressed to keep things stable.

Author

Posted by Jeff Morris, Senior Software Engineer, Couchbase

Jeff Morris is a Senior Software Engineer at Couchbase. Prior to joining Couchbase, Jeff spent six years at Source Interlink as an Enterprise Web Architect. Jeff is responsible for the development of Couchbase SDKs and how to integrate with N1QL (query language).

9 Comments

  1. All of this sounds amazing. Now the big question, when have you planned to launch it?

    1. We will be publishing dates for developer previews, beta\’s and finally the GA release in the near future. Stay tuned!

  2. Hi Jeff – we are kicking off a new and exciting product build using 1.3. I\’m guessing they\’ll be lots of changes coming with 2.0 so would you advise us to keep our approach simple until the 2.0 details come out to save a ton of rewiring? Such a subjective question…I know. thx

    1. Hi Jake –

      There will be significant changes to how the client created, how you interact with a bucket via a client and the configuration elements themselves. Additionally, while the same K/V and view operations will be supported, the interface will be quite different then the current .NET one.

      You may want to consider your own wrapper for the client, so that you can depend upon that interface instead of the CB, thus making upgrading easier in the future. Yes, keeping things simple will help in the future upgrade as well.

      I\’ll be doing another post shortly that gives a high-level overview of what the client interfaces will look like shortly, which should help some.

      -Jeff

  3. Hi Jeff, We are developing an application where I must be absolutely sure that the data is persisted when the client interface returns no errors. From the current documentation I learn that when the client returns no errors this only means that the data has arrived in memory and is queued for persistence to disk and will be transfered by the queue handling process to one or more replica servers. When at that moment the server that is handling the clientrequest is fail-overed or simply crashes, the data is subject of being lost. This is realy a showstopper to our application and we tend to use a ordinary sqldbs instead to be protected by the transaction system. Will the 2.0 interface address this problem and only return with status ok when data is delivered at a server AND data is in memory (as a replica) in at least one other server. So assuring the data will get saved and will be visible after the failover process has dealt with the failed server. The application tolerates not being able to read all stored data as long as the failover is not handled by the cluster, so the temporary unavailability of the stored data (for a couple of minutes) is no show-stopper.

    We like the flexible other features of couchbase but this is a no-go situation if this will not get fixed.

    Regards, Martien

    1. Martien –

      We have mechanism called \”Observe\” to handle durability constraints such what you are asking for. You can read more about it here: http://www.couchbase.com/wiki/

      Let me know how things go!

      Thanks,

      Jeff

      1. Hello Jeff,

        I completely overlooked that feature, that is what I was looking for. So now I can continue the evaluation!! Thanks a lot and happy Easter.

        Martien

      2. Hello Jeff,

        Indeed there are observe functions but unfortunately the observe functions are not supported with .ExecuteCas which I need to assure I replace the data correctly en consistently. Any ideas?

        Regards, Martien

Leave a reply