Node.js + Couchbase 3.0 + SDK 2.0 + Bootstrap

Requirements

When it came time to build a product demonstration for the Couchbase Connect Conference the following requirements were established:

[1] It must be a live demo, with audience participation — no canned movies or powerpoint only presentations.  The bar was set high with significant risk.   
[2] It must demonstrate the power of Couchbase 3.0 and show realtime consistency of data between clusters in different datacenters (XDCR).
[3] It must be agile, leveraging the power of the new Couchbase 2.0 SDK's.  
[4] The application needs to be code complete in 4 days to reinforce how development with Couchbase is ideal for the agile enterprise. 
[5] Tell the world how it was done, and prove it by publishing the source code.  

It was decided that an auction application, spread over two web servers in two different datacenters talking to two different clusters would achieve the objective.   A Salt Lake City Cluster, and a London Cluster will maintain consistency with bidirectional replication between them using XDCR.   The new DCP protocol in Couchbase 3.0 means that consistency would only be limited by wirespeed.   An application server in each datacenter running node.js handles traffic in each region.  The code can be downloaded from couchbaselabs repo on github. **  

Application Design

In addition to using node.js with Express and the Couchbase 2.0 SDK, bootstrap was used for front end development.   The 2.0 SDK's are a remarkable achievment, and include significant new advancements that simplify development.   Information on the 2.0 SDK's and why Couchbase continues to extend development leadership within the market can be found in our developer portal.  

Session Management

While no authentication is needed for a just for fun auction,  a means of identifying users and concurrent sessions is still required.  A simple login form stores a cookie with a user id for each session.   The login function checks if the user exists, and if there is any filtered words violations prior to creating a session cookie.  

module.exports.login = function (newUser,req,res,done) {
    filterScan(newUser, function (filtcb) {
        if (filtcb) {
            db.read(newUser, function (err, user) {
                if (err && err.code === couchbase.errors.keyNotFound) {
                        res.cookie(‘user’, newUser);
                    db.upsert(newUser, 0, function (err, res) {
                        if (err) {
                            console.log(“LOGIN:ERROR CREATING:” + err);
                            done(“LOGIN:ERROR CREATING:” + err, null);
                            return;
                        }
                        console.log(“LOGIN:” + newUser + “:SUCCESS”);
                        done(null, “LOGIN:” + newUser + “:SUCCESS”);
                        return;
                    });
                } else {
                    if (user) {
                        console.log(“LOGIN:ERROR:” + newUser + ” EXISTS”);
                        done(“LOGIN:ERROR:” + newUser + ” exists — please choose a different username”, null);
                        return;
                    }
                    console.log(“LOGIN:ERROR:END:” + err);
                    done(“LOGIN:ERROR” + err, null);
                    return;
                }
            });
        } else {
            console.log(“LOGIN:ERROR:WORD VIOLATION”);
            done(“LOGIN:ERROR, PROHIBITED TERM IN USERNAME”, null);
            return;
        }
    });
}

Then a very small  “middleware” function checks if a user has setup a session prior to routing incoming requests.

function isLoggedIn(req, res, next) {
    if (req.cookies.user) {
        return next();
    }
    res.redirect(‘/’);
}

A further “middleware” function checks to see if a countdown is set or if this auction is closed.

module.exports.isActive = function(req,res,next) {
    db.read(‘state’,function(err,done){
        if(done.value.active) {
            return next();
        }else{
            if(done.value.countdown != “none”)
            {
                res.render(‘countdown.jade’);
            }else {
                db.read(‘bike’, function (err, cb) {
                    res.render(‘winner’, {user: req.cookies.user, amount: cb.value.bid, high: cb.value.high});
                    return;
                });
            }
        }
    });
}

User Interaction

The app.js file controls how the application functions.   All of the routes are defined in the routes.js object in a specific file.  The application object is  passed into the routes object.  This is a stylistic preference that facilitates easier middleware functionality (as used in the above examples).   The REST API is straight forward: The following methods are exposed:

POST   api/auction/login [set a session, cookie, for each unique user]
GET     api/auction/login [safety method, in case of page refresh]    
GET     api/auction/load [load the auction page]                 
POST   api/auction/bid [set a bid, with validation]                
GET     api/auction/bid [safety method, in case of page refresh]                
GET     api/auction/get [get current high bidder]
POST   api/auction/open/:password [reset auction, flush bucket, and set auction to open]
POST   api/auction/close/:password [set auction to closed]
POST   api/auction/view/build    [set a view to be used for looking at bid history]
GET     api/auction/view/get [get view for bid history]
GET     api/auction/countdown/get [get time in seconds between auction “go live” and now]
POST   api/auction/countdown/set/:yyyy/:mm/:dd/:hh [set auction go live date]
POST   api/auction/countdown/del/:password [set auction countdown to none, and auction to live]

Dynamic Updates

In the keynote demo multiple users bid concurrently against two different clusters, and any updates need to be immediately propagated to any other user currently viewing the auction page.  There are several ways of  performing this, each with advantages and disadvantages.   This particular application loads the auction page through a jade template.  The jade template instantiates a jquery ajax polling loop.  This loop polls the /api/auction/get REST endpoint twice each second to determine the highest bidder.   

setInterval(function () {
    pollBid()
}, 500);

function pollBid() {
    $.get(‘/api/auction/get’, function (cb) {
        if (cb) {
            $(“#bid”).html(“$” + cb.bid);
            $(“#high”).html(cb.high);
        }
        else {
        }
    });
}

Administration 

The following methods are called using curl commands during the demo to set the auction states.   The sequence for the demo is reset the auction, set a go live date, set the auction live when the keynote starts, and close the auction.  

curl -X POST http://localhost:3001/api/auction/open/un5ecure_pa55word
curl -X POST http://localhost:3001/api/countdown/set/2014/10/06/13
curl -X POST http://localhost:3001/api/countdown/del/un5ecure_pa55word
curl -X POST http://localhost:3001/api/auction/close/un5ecure_pa55word

Source and Considerations

Much of the functionality in this app is hard coded and specific to this keynote demo use case.   The data model is specific to this use case and is only applicable as a proof of concept.   The use case has no secure access or authentication.  When deploying to different clusters linked by XDCR additional administration is required to “reset” the auction.   Buckets cannot be “flushed” that are currently participating in XDCR.   

** Disclaimer: Couchbase Labs provides experimental code for research and development purposes only.   Code and applications from Couchbase Labs are not supported under any Couchbase Support Agreement and are provided as is with no warranty of any kind.  

Posted by Todd Greenstein

As a Solution Architect at Couchbase. I specialize in API design, architecture, data modeling, nodejs and golang development.

2 Comments

  1. Todd, thanks for the post and demo. Could you share the cluster configs that you used on the AWS version to achieve the 3M/sec performance? IOW, what kind of server horsepower was needed?

    TIA,

    Dave

  2. Hey Dave, these were c3.4xlarge instances. Couchbase Server 3.0 was installed with default settings.

Leave a reply