Node.js + Couchbase 3.0 + SDK 2.0 + Bootstrap
Requirements
When it came time to build a product demonstration for the Couchbase Connect Conference the following requirements were established:
[1] It must be a live demo, with audience participation — no canned movies or powerpoint only presentations. The bar was set high with significant risk.
[2] It must demonstrate the power of Couchbase 3.0 and show realtime consistency of data between clusters in different datacenters (XDCR).
[3] It must be agile, leveraging the power of the new Couchbase 2.0 SDK’s.
[4] The application needs to be code complete in 4 days to reinforce how development with Couchbase is ideal for the agile enterprise.
[5] Tell the world how it was done, and prove it by publishing the source code.
It was decided that an auction application, spread over two web servers in two different datacenters talking to two different clusters would achieve the objective. A Salt Lake City Cluster, and a London Cluster will maintain consistency with bidirectional replication between them using XDCR. The new DCP protocol in Couchbase 3.0 means that consistency would only be limited by wirespeed. An application server in each datacenter running node.js handles traffic in each region. The code can be downloaded from couchbaselabs repo on github. **
Application Design
In addition to using node.js with Express and the Couchbase 2.0 SDK, bootstrap was used for front end development. The 2.0 SDK’s are a remarkable achievment, and include significant new advancements that simplify development. Information on the 2.0 SDK’s and why Couchbase continues to extend development leadership within the market can be found in our developer portal.
Session Management
While no authentication is needed for a just for fun auction, a means of identifying users and concurrent sessions is still required. A simple login form stores a cookie with a user id for each session. The login function checks if the user exists, and if there is any filtered words violations prior to creating a session cookie.
module.exports.login = function (newUser,req,res,done) {
filterScan(newUser, function (filtcb) {
if (filtcb) {
db.read(newUser, function (err, user) {
if (err && err.code === couchbase.errors.keyNotFound) {
res.cookie(‘user’, newUser);
db.upsert(newUser, 0, function (err, res) {
if (err) {
console.log(“LOGIN:ERROR CREATING:” + err);
done(“LOGIN:ERROR CREATING:” + err, null);
return;
}
console.log(“LOGIN:” + newUser + “:SUCCESS”);
done(null, “LOGIN:” + newUser + “:SUCCESS”);
return;
});
} else {
if (user) {
console.log(“LOGIN:ERROR:” + newUser + ” EXISTS”);
done(“LOGIN:ERROR:” + newUser + ” exists — please choose a different username”, null);
return;
}
console.log(“LOGIN:ERROR:END:” + err);
done(“LOGIN:ERROR” + err, null);
return;
}
});
} else {
console.log(“LOGIN:ERROR:WORD VIOLATION”);
done(“LOGIN:ERROR, PROHIBITED TERM IN USERNAME”, null);
return;
}
});
}
filterScan(newUser, function (filtcb) {
if (filtcb) {
db.read(newUser, function (err, user) {
if (err && err.code === couchbase.errors.keyNotFound) {
res.cookie(‘user’, newUser);
db.upsert(newUser, 0, function (err, res) {
if (err) {
console.log(“LOGIN:ERROR CREATING:” + err);
done(“LOGIN:ERROR CREATING:” + err, null);
return;
}
console.log(“LOGIN:” + newUser + “:SUCCESS”);
done(null, “LOGIN:” + newUser + “:SUCCESS”);
return;
});
} else {
if (user) {
console.log(“LOGIN:ERROR:” + newUser + ” EXISTS”);
done(“LOGIN:ERROR:” + newUser + ” exists — please choose a different username”, null);
return;
}
console.log(“LOGIN:ERROR:END:” + err);
done(“LOGIN:ERROR” + err, null);
return;
}
});
} else {
console.log(“LOGIN:ERROR:WORD VIOLATION”);
done(“LOGIN:ERROR, PROHIBITED TERM IN USERNAME”, null);
return;
}
});
}
Then a very small “middleware” function checks if a user has setup a session prior to routing incoming requests.
function isLoggedIn(req, res, next) {
if (req.cookies.user) {
return next();
}
res.redirect(‘/’);
}
if (req.cookies.user) {
return next();
}
res.redirect(‘/’);
}
A further “middleware” function checks to see if a countdown is set or if this auction is closed.
module.exports.isActive = function(req,res,next) {
db.read(‘state’,function(err,done){
if(done.value.active) {
return next();
}else{
if(done.value.countdown != “none”)
{
res.render(‘countdown.jade’);
}else {
db.read(‘bike’, function (err, cb) {
res.render(‘winner’, {user: req.cookies.user, amount: cb.value.bid, high: cb.value.high});
return;
});
}
}
});
}
db.read(‘state’,function(err,done){
if(done.value.active) {
return next();
}else{
if(done.value.countdown != “none”)
{
res.render(‘countdown.jade’);
}else {
db.read(‘bike’, function (err, cb) {
res.render(‘winner’, {user: req.cookies.user, amount: cb.value.bid, high: cb.value.high});
return;
});
}
}
});
}
User Interaction
The app.js file controls how the application functions. All of the routes are defined in the routes.js object in a specific file. The application object is passed into the routes object. This is a stylistic preference that facilitates easier middleware functionality (as used in the above examples). The REST API is straight forward: The following methods are exposed:
POST api/auction/login [set a session, cookie, for each unique user]
GET api/auction/login [safety method, in case of page refresh]
GET api/auction/load [load the auction page]
POST api/auction/bid [set a bid, with validation]
GET api/auction/bid [safety method, in case of page refresh]
GET api/auction/get [get current high bidder]
POST api/auction/open/:password [reset auction, flush bucket, and set auction to open]
POST api/auction/close/:password [set auction to closed]
POST api/auction/view/build [set a view to be used for looking at bid history]
GET api/auction/view/get [get view for bid history]
GET api/auction/countdown/get [get time in seconds between auction “go live” and now]
POST api/auction/countdown/set/:yyyy/:mm/:dd/:hh [set auction go live date]
POST api/auction/countdown/del/:password [set auction countdown to none, and auction to live]
Dynamic Updates
In the keynote demo multiple users bid concurrently against two different clusters, and any updates need to be immediately propagated to any other user currently viewing the auction page. There are several ways of performing this, each with advantages and disadvantages. This particular application loads the auction page through a jade template. The jade template instantiates a jquery ajax polling loop. This loop polls the /api/auction/get REST endpoint twice each second to determine the highest bidder.
setInterval(function () {
pollBid()
}, 500);
function pollBid() {
$.get(‘/api/auction/get’, function (cb) {
if (cb) {
$(“#bid”).html(“$” + cb.bid);
$(“#high”).html(cb.high);
}
else {
}
});
}
Administration
The following methods are called using curl commands during the demo to set the auction states. The sequence for the demo is reset the auction, set a go live date, set the auction live when the keynote starts, and close the auction.
curl -X POST http://localhost:3001/api/auction/open/un5ecure_pa55word
curl -X POST http://localhost:3001/api/countdown/set/2014/10/06/13
curl -X POST http://localhost:3001/api/countdown/del/un5ecure_pa55word
curl -X POST http://localhost:3001/api/auction/close/un5ecure_pa55word
Source and Considerations
Much of the functionality in this app is hard coded and specific to this keynote demo use case. The data model is specific to this use case and is only applicable as a proof of concept. The use case has no secure access or authentication. When deploying to different clusters linked by XDCR additional administration is required to “reset” the auction. Buckets cannot be “flushed” that are currently participating in XDCR.
** Disclaimer: Couchbase Labs provides experimental code for research and development purposes only. Code and applications from Couchbase Labs are not supported under any Couchbase Support Agreement and are provided as is with no warranty of any kind.
Todd, thanks for the post and demo. Could you share the cluster configs that you used on the AWS version to achieve the 3M/sec performance? IOW, what kind of server horsepower was needed?
TIA,
Dave
Hey Dave, these were c3.4xlarge instances. Couchbase Server 3.0 was installed with default settings.