A few years ago, Brett Lawson made a great blog series on using Couchbase Server and Node.js for the development a game server framework. Since then, the Node.js SDK for Couchbase has grown significantly from version 1.x to 2.x.
In this article we’re going to revisit those original three posts and change them to keep up with the latest Node.js and Express Framework standards as well as the latest Node.js SDK version of Couchbase.
The Prerequisites
- Node.js 0.12
- Couchbase Server 4.0
Preparing the Project
Whether you’re on Mac, Windows, or Linux, creating a new Express Framework application should be consistent between them. Create a new directory called gameapi-nodejs probably on your Desktop and from your Terminal (Mac / Linux) or Command Prompt (Windows), run the following command:
1 2 3 |
npm init |
Answer the questions asked to the best of your ability. Of course, the project directory must be your current directory in the Command Prompt or Terminal for this to be successful. Alternative to the command, you could manually create and populate this file. Create a new file called package.json in your project’s directory and fill it with the following:
1 2 3 4 5 6 7 8 9 |
{ "name": "gameapi-nodejs", "version": "1.0.0", "description": "An example of using the Node.js SDK for Couchbase to create a simple game server", "author": "Couchbase, Inc.", "license": "MIT" } |
We’re not done. We need to install the project dependencies before we can start planning out this application. From the Command Prompt or Terminal, run the following command:
1 2 3 |
npm install body-parser couchbase express node-forge uuid --save |
This will install Express Framework, the Couchbase Node.js SDK, Forge for password hashing, UUID for generating unique values, and the body-parser middleware for handling URL encoded and JSON POST data.
Preparing the Database
Before we start coding, Couchbase Server must be installed with a bucket called gaming-sample created.
Since this project is going to make use of a major feature of Couchbase 4.0, we’ll need to further configure the bucket to have a primary index created. If you’re using a Linux, Mac or Windows machine, this can be easily accomplished through the Couchbase Query (CBQ) client.
Mac
To open CBQ on Mac, from the Terminal, run the following:
1 2 3 |
./Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/cbq |
Windows
To open CBQ on Windows, from the Command Prompt, run the following:
1 2 3 |
C:/Program Files/Couchbase/Server/bin/cbq.exe |
Creating a Primary Index
With CBQ open, run the following:
1 2 3 |
CREATE PRIMARY INDEX ON `gaming-sample` using GSI; |
Your bucket is now ready for use with the rest of the project!
The Project Structure
Our project is going to be composed of the following:
Item | Parent | Description |
---|---|---|
models | All database class files will go in here | |
routes | All API endpoint definitions will go in here | |
accountmodel.js | models | Creating and retrieving account information |
sessionmodel.js | models | Authenticating users and maintaining session information |
statemodel.js | models | Create, update, and retrieve game state information |
routes.js | routes | All endpoints for GET, POST, PUT will be in here |
app.js | Server setup information | |
config.json | Static variables | |
package.json | Dependency information |
The Basics
Before we get deep into the Node.js game server logic, it is best to get the base Express Framework application configured.
In the project root, create and open a file called config.json as it will hold static information such as Couchbase connectivity information. Include the following when it is open:
1 2 3 4 5 6 7 8 |
{ "couchbase": { "server": "127.0.0.1:8091", "bucket": "gaming-sample" } } |
In the project root, create and open a file called app.js as it will hold all the basic information about running the Node.js server. Include the following when you’ve opened the file:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
var express = require("express"); var bodyParser = require("body-parser"); var couchbase = require("couchbase"); var config = require("./config"); var app = express(); app.use(bodyParser.json()); app.use(bodyParser.urlencoded({ extended: true })); // Global declaration of the Couchbase server and bucket to be used module.exports.bucket = (new couchbase.Cluster(config.couchbase.server)).openBucket(config.couchbase.bucket); // All endpoints to be used in this application var routes = require("./routes/routes.js")(app); var server = app.listen(3000, function () { console.log("Listening on port %s...", server.address().port); }); |
Let’s break down what we see here. The first few lines are us including the dependencies into our application. Nothing special there. What is important is the following:
1 2 3 4 |
app.use(bodyParser.json()); app.use(bodyParser.urlencoded({ extended: true })); |
This means that we’re going to be parsing JSON data and URL encoded data from request bodies. In particular POST and PUT requests. The next thing we’re doing is initializing the Couchbase cluster in the application and opening a single bucket for use within the application:
1 2 3 |
module.exports.bucket = (new couchbase.Cluster(config.couchbase.server)).openBucket(config.couchbase.bucket); |
By including module.exports.bucket we’re saying that we’re going to use it through the application. Now in other JavaScript files, if we want to access the bucket we can just do:
1 2 3 |
var bucket = require("relative/path/to/app").bucket; |
Next you’ll see that we include our soon to be created routes/routes.js file and passing it the app variable as one of the arguments. What that does will be obvious soon.
Finally, by calling app.listen we’re telling Node.js to listen on port 3000 for requests. The application is almost useable in its most basic state. Create and open routes/routes.js and add the following lines:
1 2 3 4 5 6 7 8 9 10 11 |
var appRouter = function(app) { app.get("/", function(req, res) { res.status(403).send("Not a valid endpoint"); }); } module.exports = appRouter; |
The application can now be run by executing node app.js from the Command Prompt or Terminal. Landing on http://localhost:3000 should leave you with a “Not a valid endpoint” message.
The API Data Model
Before diving into the code that matters, it is best to know how the data will look in Couchbase. For any one user, there will four documents that look like the following:
The User Document
The user document will hold all information about a user. For this application that information will be a name, username, and password.
The name of this document will be prefixed with user:: and have a unique uid value appended to it. This document naming strategy makes use of what is called compound keys.
1 2 3 4 5 6 7 8 9 |
{ "type": "user", "uid", "", "name": "", "username": "", "password": "" } |
The Username Document
The username document will hold only the uid value that is found in the user document. The purpose of the username document can be thought of like a login method document. For example it could represent simple sign in where the user enters a username and password. Being that the document contains the linking uid, it can be tied to the user document. The username document is prefixed with username:: and the actual username is appended to it. A similar strategy can be used if using Facebook or Twitter as login methods and linking them as well through the uid field.
1 2 3 4 5 6 |
{ "type": "username", "uid": "" } |
The Session Document
The session document is an auto-expiring document that acts as a users route into more secure information. In theory, the users front-end will store the sid value and pass it between protected endpoints. With it, a uid tied to a user can be accessed.
1 2 3 4 5 6 7 |
{ "type": "session", "sid": "", "uid": "" } |
The State Document
The state document will hold information about the games particular state. For example if the game character has five lives left and twenty potions, that information would get saved here. There is version information to prevent conflicts between two active game sessions from saving under the same account.
1 2 3 4 5 6 7 8 9 10 11 12 |
{ "type": "state", "uid": "", "states": { "state-name": { "version": 1, "data": { } } } } |
Creating The Account Model
The account model will serve three particular purposes:
- Creating a user account
- Retreiving a user account
- Comparing a hashed password with an unhashed password
Before we start coding, we need to get our includes in order. Add the following to the top of the models/accountmodel.js file:
1 2 3 4 5 6 |
var uuid = require("uuid"); var forge = require("node-forge"); var db = require("../app").bucket; var N1qlQuery = require('couchbase').N1qlQuery; |
Based on the data model seen above, creating a user account will require a name, username, and password. With that information in hand, the password will be hashed, and the user information will be stored along-side a reference document like so:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
AccountModel.create = function(params, callback) { var userDoc = { type: "user", uid: uuid.v4(), name: params.name, username: params.username, password: forge.md.sha1.create().update(params.password).digest().toHex() }; var referenceDoc = { type: "username", uid: userDoc.uid }; db.insert("username::" + userDoc.username, referenceDoc, function(error) { if(error) { callback(error, null); return; } db.insert("user::" + userDoc.uid, userDoc, function(error, result) { if(error) { callback(error, null); return; } callback(null, {message: "success", data: result}); }); }); }; |
Notice in the above code, we first try to insert a new reference document. If it fails (maybe it already exists), neither documents will be saved and instead the error message will be returned. Whether it is a success or failure, the callback from the routes file is executed for displaying any kind of answer to the requestor.
In terms of reading user data, one of two things could happen. We could either pass in a username because we’re doing a simple login of-sorts, or we could pass in a user id. Depends on what we’re after. Let’s say we’re just signing in with the username, you’d probably want to call a function like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
AccountModel.getByUsername = function(params, callback) { var query = N1qlQuery.fromString( "select users.* from `gaming-sample` as usernames " + "join `gaming-sample` as users on keys ("user::" || usernames.uid) " + "where meta(usernames).id = $1" ); db.query(query, ["username::" + params.username], function(error, result) { if(error) { return callback(error, null); } callback(null, result); }); }; |
Notice how we’re using a N1QL query, new in Couchbase 4.0. It is very similar to traditional SQL and it also gives us the convenience of not having to crunch or format data in the application layer. Couchbase Server will do all this for us. We do however, have the option to request data like in older versions of Couchbase and other NoSQL platforms.
In the N1QL statement above, we’re selecting a document with the compound key appended with our plain text username. A join is happening with the uid property of the usernames document (foreign key) and the users document (primary key).
This brings us to validating a password. No database calls are made here. We’re simply going to take a raw password, hash it, then compare the hash with the stored password.
1 2 3 4 5 |
AccountModel.validatePassword = function(rawPassword, hashedPassword) { return forge.md.sha1.create().update(rawPassword).digest().toHex() === hashedPassword ? true : false; }; |
To make models/accountmodel.js useable in our routes file we must export it like so at the bottom of the
models/accountmodel.js code:
1 2 3 |
module.exports = AccountModel; |
Creating The Session Model
The session model will serve three particular purposes:
- Creating a user session
- Retreiving a user session
- Validating a user session
Before we start coding, we need to get our includes in order. Add the following to the top of the models/sessionmodel.js file:
1 2 3 4 |
var uuid = require("uuid"); var db = require("../app").bucket; |
Creating A Session
When a user wishes to create a session, a uid must be provided. With this in hand, a unique session id is created and inserted into the database with an expiration. When the document expires, it will automatically be removed from Couchbase with no user intervention, thus logging the user out.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
SessionModel.create = function(uid, callback) { var sessionDoc = { type: "session", sid: uuid.v4(), uid: uid }; db.insert("session::" + sessionDoc.sid, sessionDoc, {"expiry": 3600}, function(error, result) { if(error) { callback(error, null); return; } callback(null, sessionDoc.sid); }); }; |
Authenticating A User
With the user session created, the session id must be used every time the user wants to reach a protected endpoint.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
SessionModel.authenticate = function(req, res, next) { if(!req.headers.authorization) { next("Must be authorized to use"); } var authInfo = req.headers.authorization.split(" "); if(authInfo[0] === "Bearer") { var sid = authInfo[1]; SessionModel.get(sid, function(error, result) { if(error) { return next(error); } SessionModel.refresh(sid, function() {}); req.uid = result.value.uid; next(); }); } }; |
In the above code, the function will receive a session id which it will then use to look up to see if a session already exists. If it does, it will reset the session expiration time and return the uid associated to it. Getting the session information can be seen as follows:
1 2 3 4 5 6 7 8 9 10 11 |
SessionModel.get = function(sid, callback) { db.get("session::" + sid, function(error, result) { if(error) { callback(error, null); return; } callback(null, result); }); }; |
Not too bad right? Finally you can see that the session reset happens in a similar fashion:
1 2 3 4 5 6 7 8 9 |
SessionModel.refresh = function(sid, callback) { db.touch("session::" + sid, 3600, function(error, result) { if(error) { callback(error, null); } }); }; |
The touch method won’t add time, it will instead reset time. In this case it will reset the timer to one hour
To make models/sessionmodel.js useable in our routes file we must export it like so at the bottom of the models/sessionmodel.js code:
1 2 3 |
module.exports = SessionModel; |
Creating The State Model
The state model will serve two particular purposes:
- Creating or updating a save-state
- Retreiving a save-state by name
Before we start coding, we need to get our includes in order. Add the following to the top of the models/statemodel.js file:
1 2 3 4 5 6 |
var uuid = require("uuid"); var couchbase = require("couchbase"); var N1qlQuery = require('couchbase').N1qlQuery; var db = require("../app").bucket; |
Creating A Save State
The goal behind creating or updating a save-state is that we will first check to see if one exists. If it does not, we will create it. If it does, then we will get whatever information exists, change it, then replace whatever exists currently in the database. This is all done while increasing the state version to avoid conflicts between game saves. Not really to prevent conflicts in saving to Couchbase, just to make sure you don’t pick up the game on two devices and override game data with a much older save.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
StateModel.save = function(uid, name, preVer, data, callback) { db.get("user::" + uid + "::state", function(error, result) { if(error && error.code !== couchbase.errors.keyNotFound) { callback(error, null); return; } var stateDoc = { type: "state", uid: uid, states: {} }; if(result != null && result.value) { stateDoc = result.value; } var stateBlock = { version: 0, data: null }; if(stateDoc.states[name]) { stateBlock = stateDoc.states[name]; } else { stateDoc.states[name] = stateBlock; } if(stateBlock.version !== preVer) { return callback({"status": "error", "message": "Your version does not match the server version"}); } else { stateBlock.version++; stateBlock.data = data; } var stateOptions = {}; if(result != null && result.value) { stateOptions.cas = result.cas; } db.upsert("user::" + uid + "::state", stateDoc, stateOptions, function(error, result) { if(error) { return callback(error, null); } callback(null, stateBlock); }); }); }; |
See the upsert in there. In Couchbase that means create if it doesn’t exist, or replace if it does. Very convenient for things like save-states for a game.
Getting The States
This leaves us with getting any save-state that we might have created.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
StateModel.getByUserIdAndName = function(uid, name, callback) { db.get("user::" + uid + "::state", function(error, result) { if(error) { if(error.code !== couchbase.errors.keyNotFound) { return callback(null, {}); } else { return callback(error, null); } } if(!result.value.states[name]) { return callback({"status": "error", "message": "State does not exist"}, null); } callback(null, result.value.states[name]); }); }; |
The concept behind this is that we’re doing a document lookup based on a user id. If the state document for a particular uid exists then do a lookup on the associative array to see if the state name exists. If it does, return whatever state content exists for the name.
To make models/statemodel.js useable in our routes file we must export it like so at the bottom of the models/statemodel.js code:
1 2 3 |
module.exports = StateModel; |
Creating The API Routes
We’ve created all the necessary data models above, so it is time to string it all together with user accessible routes. Going back to the routes/routes.js file, we’ll start by adding the account model routes:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
app.post("/api/user", function(req, res) { if(!req.body.name) { return res.status(400).send({"status": "error", "message": "A name is required"}); } else if(!req.body.username) { return res.status(400).send({"status": "error", "message": "A username is required"}); } else if(!req.body.password) { return res.status(400).send({"status": "error", "message": "A password is required"}); } AccountModel.create(req.body, function(error, result) { if(error) { return res.status(400).send(error); } res.send(result); }); }); |
The above endpoint expects a POST request with a name, username, and password body parameter We are listening for POST because it is best practice to use POST when creating or inserting data over an HTTP request. If all three exist, then the AccountModel.create() method is called, finally returning either an error or result depending on how successful the method was. If at least one of the required parameters does not exist, an error is returned. A list of error codes can be seen here.
An endpoint for getting user information isn’t so important in this example, so we’ll jump straight into authenticating the user and creating a session. In the routes/routes.js file, add the following:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
app.get("/api/auth", function(req, res, next) { if(!req.query.username) { return next(JSON.stringify({"status": "error", "message": "A username must be provided"})); } if(!req.query.password) { return next(JSON.stringify({"status": "error", "message": "A password must be provided"})); } AccountModel.getByUsername(req.query, function(error, user) { if(error) { return res.status(400).send(error); } if(!AccountModel.validatePassword(req.query.password, user[0].password)) { return res.send({"status": "error", "message": "The password entered is invalid"}); } SessionModel.create(user[0].uid, function(error, result) { if(error) { return res.status(400).send(error); } res.setHeader("Authorization", "Bearer " + result); res.send(user); }); }); }); |
The authentication endpoint expects a username and password. If both are found, the user will be looked up. If the user is found a password comparison is made and if successful then a session will be created.
The final two API endpoints that are useful to us are for getting and creating save states. Starting with creating a save state endpoint, in your routes/routes.js, add the following:
1 2 3 4 5 6 7 8 9 10 |
app.put("/api/state/:name", SessionModel.authenticate, function(req, res, next) { StateModel.save(req.uid, req.params.name, parseInt(req.query.preVer, 10), req.body, function(error, result) { if(error) { return res.send(error); } res.send(result); }); }); |
The above endpoint expects a URL parameter representing the state name, a query parameter representing the current state version, and a request body that can contain any JSON imaginable as it represents the game data worth saving.
Finally, we’re left with getting states that have been saved.
1 2 3 4 5 6 7 8 9 10 |
app.get("/api/state/:name", SessionModel.authenticate, function(req, res, next) { StateModel.getByUserIdAndName(req.uid, req.params.name, function(error, result) { if(error) { return res.send(error); } res.send(result); }); }); |
The above endpoint expects a URL parameter representing the particular save state to find. Of course it also expects the user to be authenticated first as well.
Testing The API
The API endpoints we created in the routes/routes.js file can be tested a few ways. Two of my favorite ways to test are with the Postman extension for Chrome or with cURL. Try it for yourself using cURL:
1 2 3 4 5 6 7 8 9 10 11 12 |
curl -X POST http://localhost:3000/api/user --data "name=Nic%20Raboy&username=nraboy&password=12345" RESPONSE: { "message": "success", "data": { "cas": "16588775686144" } } |
Above we went ahead and created a new user account
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
curl -X GET "http://localhost:3000/api/auth?username=nraboy&password=12345" RESPONSE: [ { "name": "Nic Raboy", "password": "8cb2237d0679ca88db6464eac60da96345513964", "type": "user", "uid":"c3e834b0-867e-4a43-aede-b15a4e139adc", "username": "nraboy" } ] |
Above we went ahead and created a user session. This same strategy can be applied for the other endpoints as well.
Conclusion
Using Node.js and the Couchbase Server SDK, you can easily create an API backend for your games. In Couchbase 4.0 you now have the freedom to use N1QL as an option for querying data in your application.
The full Node.js application that was written about in this article can be downloaded for free from the Couchbase Labs GitHub repository.
Hi,
Do you still need to enable N1QL in couchbase 4 by running \’bucket.enableN1ql(\’localshot:8093\’);\’? Thanks
nope, it just works with the latest SDK
i want to ask you for a client`s error:\”
[com.couchbase.client.deps.io.netty.util.ResourceLeakDetector] LEAK: ByteBuf.release() was not called before it\’s garbage-collected. Enable advanced leak reporting to find out where the leak occurred. To enable advanced leak reporting, specify the JVM option \’-Dcom.couchbase.client.deps.io.netty.leakDetectionLevel=advanced\’ or call ResourceLeakDetector.setLevel()\”
What is the problem?How to solve it
Hi,
Can N1QL only be used for queries or can it be used to do the UPSERT\’s and INSERT\’s? Thanks
As of right now INSERT, UPDATE, DELETE, UPSERT are beta. You can use them, but until they are out of beta we don\’t recommend using them in a production environment.
http://developer.couchbase.com…
Best,