With the boom of Twitter, SMS text messages, and other forms of short message interactions, there has been a boom in URL shortening services. For example, you can use TinyURL, Bitly, Owly, and so many others. The purpose here is to take very long URLs and make them significantly shorter for distribution in a message.
But how do these URL shortening services work?
We’re going to see how to create our own URL shortener using Node.js with Express Framework and Couchbase Server with N1QL. In our example, the short URLs will be generated using Node.js and they will be stored and accessed using Couchbase.
The Requirements
There aren’t too many requirements to make this project possible. At a minimum you’ll need the following to be successful:
- Couchbase Server 4.1+
- Node.js 4.0+
We need to use a version of Couchbase Server that supports N1QL queries. The Node.js version is less strict, but we will need it for serving our application and obtaining dependencies using the Node Package Manager (NPM).
Preparing Couchbase and Understanding the Data Format
Before we can start developing our Node.js application we need to understand the data plan as well as configure Couchbase Server to allow for N1QL queries.
In Couchbase, the goal is to store our data in the following format:
1 2 3 4 5 |
{ "id": "5Qp8oLmWX", "longUrl": "https://www.thepolyglotdeveloper.com/2016/08/using-couchbase-server-golang-web-application/", "shortUrl": "http://localhost:3000/5Qp8oLmWX" } |
We will pass the application a long URL and generate a unique short-hash based on a piece of data. This hash will be used when constructing the short URL which will also be stored in the Couchbase document. The id of the document itself will also be that of the hash value.
Now we need to create a Couchbase Server bucket for storing the data for our application. This bucket can be created via the Couchbase Server Administration Dashboard. Let’s call this bucket example.
When querying for data we will not always be doing lookups based on the id value. This means we’ll need to create indexes on the document values to allow for N1QL queries. To keep things simple, create a simple primary index like the following:
1 |
CREATE PRIMARY INDEX ON `example` USING GSI; |
This query can be executed using the Couchbase Server Query Workbench (Enterprise Edition) or the Couchbase Shell known as CBQ. The index won’t be the quickest because it is so general, but it will accomplish the needs of our very simple application.
Developing the Node.js URL Shortener Application with Express Framework
With the database properly configured we can worry about the code behind the application. This application will be heavily dependent on Express Framework and Couchbase, but also a hashing library known as Hashids.
Creating the Project with the Dependencies
Let’s create a fresh Node.js project from the Command Prompt (Windows) or Terminal (Mac and Linux):
1 |
npm init --y |
The above command will create a package.json file wherever you’re currently navigated to via your command line. So hopefully you’re within a fresh directory.
Now we need to install the project dependencies. Execute the following from the command line:
1 |
npm install couchbase body-parser express hashids --save |
At this point we can worry about the JavaScript development.
Bootstrapping the Node.js Application
Too keep this project simple and easy to understand, we’re going to keep all application logic in a single file. In a production application you’ll probably want to break it up for cleanliness and maintainability, but for this example we’re going to be fine.
Create a file called app.js at the root of your project directory. In this file, the first thing we’re going to do is import the installed dependencies like so:
1 2 3 4 |
var Couchbase = require("couchbase"); var Express = require("express"); var BodyParser = require("body-parser"); var Hashids = require("hashids"); |
Because I’ve yet to explain the body-parser
dependency, I’ll explain it now. This dependency allows us to make requests that contain a body. For example, when making POST or PUT requests it is common to include a JSON body rather than URL parameters or query parameters.
With the dependencies imported we need to initialize Express Framework and the Couchbase N1QL Engine:
1 2 |
var app = Express(); var N1qlQuery = Couchbase.N1qlQuery; |
While we’ve imported the body-parser
plugin, we’ve yet to initialize it. We want to be able to accept JSON and URL encoded values, so we need to configure the dependency like so:
1 2 |
app.use(BodyParser.json()); app.use(BodyParser.urlencoded({ extended: true })); |
At this point all of our dependencies are initialized. It would be a good idea now to establish a connection to our Couchbase Server cluster and application bucket. This can be accomplished with the following line:
1 |
var bucket = (new Couchbase.Cluster("couchbase://localhost")).openBucket("example"); |
Finally let’s start serving our Node.js application with the following:
1 2 3 |
var server = app.listen(3000, function() { console.log("Listening on port %s...", server.address().port); }); |
You’re probably realizing that we haven’t really added any logic. You’re correct in that realization as we’ve only really bootstrapped our application up until now.
Creating the URL Shortener Application Logic
What we want to do now is create some RESTful API endpoints. We’re going to create following endpoints:
1 2 3 |
/ /expand /create |
The root endpoint will be used for navigating to a long URL that is masked behind a short URL. The /expand endpoint will take a short URL and reveal the long URL without navigating to it and the /create endpoint will take a long URL and create a short URL in the database.
Starting with the /create and probably most complicated endpoint, we have the following:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
app.post("/create", function(req, res) { if(!req.body.longUrl) { return res.status(400).send({"status": "error", "message": "A long URL is required"}); } bucket.query(N1qlQuery.fromString("SELECT `" + bucket._name + "`.* FROM `" + bucket._name + "` WHERE longUrl = $1"), [req.body.longUrl], function(error, result) { if(error) { return res.status(400).send(error); } if(result.length == 0) { var hashids = new Hashids(); var response = { id: hashids.encode((new Date).getTime()), longUrl: req.body.longUrl } response.shortUrl = "http://localhost:3000/" + response.id; bucket.insert(response.id, response, function(error, result) { if(error) { return res.status(400).send(error); } res.send(response); }); } else { res.send(result[0]); } }); }); |
This endpoint is a POST request that expects a JSON body. If the longUrl
JSON property does not exist, an error will be returned to the user.
Before we actually create the short URL, we want to make sure one hasn’t already been created. We do this because we want one short URL for every one long URL. We can accomplish this by creating a parameterized N1QL query based on the longUrl
property. If the response contains a document, we’ll return it because the document already exists. If the response does not have a document, we need to create one.
Using the hashids
dependency we can create a hash based on the timestamp and use that as our id and our short URL. After inserting this new document we can return it back to the user.
Now let’s take a look at how to expand those short URLs.
1 2 3 4 5 6 7 8 9 10 11 |
app.get("/expand", function(req, res) { if(!req.query.shortUrl) { return res.status(400).send({"status": "error", "message": "A short URL is required"}); } bucket.query(N1qlQuery.fromString("SELECT `" + bucket._name + "`.* FROM `" + bucket._name + "` WHERE shortUrl = $1"), [req.query.shortUrl], function(error, result) { if(error) { return res.status(400).send(error); } res.send(result.length > 0 ? result[0] : {}); }); }); |
The above code uses a similar concept to the /create endpoint. We take a shortUrl
value and query for it using N1QL. If found, we can return the long URL with the response.
Finally we can worry about navigation.
1 2 3 4 5 6 7 8 9 10 11 |
app.get("/:id", function(req, res) { if(!req.params.id) { return res.status(400).send({"status": "error", "message": "An id is required"}); } bucket.get(req.params.id, function(error, result) { if(error) { return res.status(400).send(error); } res.redirect(result.value.longUrl); }) }); |
Remember, our short URLs are in the format of http://localhost:3000/5Qp8oLmWX which is the same location as our API service. What this means is that 5Qp8oLmWX is just a URL parameter to our root endpoint.
With the id we can do a document lookup based on the key value. If successful we’ll have the document that is currently stored.
The Full Application Source Code
In case you wanted to see the full source code to the application we had just created, it can be found below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 |
var Couchbase = require("couchbase"); var Express = require("express"); var BodyParser = require("body-parser"); var Hashids = require("hashids"); var app = Express(); var N1qlQuery = Couchbase.N1qlQuery; app.use(BodyParser.json()); app.use(BodyParser.urlencoded({ extended: true })); var bucket = (new Couchbase.Cluster("couchbase://localhost")).openBucket("example"); app.get("/expand", function(req, res) { if(!req.query.shortUrl) { return res.status(400).send({"status": "error", "message": "A short URL is required"}); } bucket.query(N1qlQuery.fromString("SELECT `" + bucket._name + "`.* FROM `" + bucket._name + "` WHERE shortUrl = $1"), [req.query.shortUrl], function(error, result) { if(error) { return res.status(400).send(error); } res.send(result.length > 0 ? result[0] : {}); }); }); app.post("/create", function(req, res) { if(!req.body.longUrl) { return res.status(400).send({"status": "error", "message": "A long URL is required"}); } bucket.query(N1qlQuery.fromString("SELECT `" + bucket._name + "`.* FROM `" + bucket._name + "` WHERE longUrl = $1"), [req.body.longUrl], function(error, result) { if(error) { return res.status(400).send(error); } if(result.length == 0) { var hashids = new Hashids(); var response = { id: hashids.encode((new Date).getTime()), longUrl: req.body.longUrl } response.shortUrl = "http://localhost:3000/" + response.id; bucket.insert(response.id, response, function(error, result) { if(error) { return res.status(400).send(error); } res.send(response); }); } else { res.send(result[0]); } }); }); app.get("/:id", function(req, res) { if(!req.params.id) { return res.status(400).send({"status": "error", "message": "An id is required"}); } bucket.get(req.params.id, function(error, result) { if(error) { return res.status(400).send(error); } res.redirect(result.value.longUrl); }) }); var server = app.listen(3000, function() { console.log("Listening on port %s...", server.address().port); }); |
There are probably many optimizations that can be done, but we cared more about the logic in making this a successful project.
Conclusion
You just saw how to create a very basic URL shortener using Node.js for the application logic, Couchbase Server as the NoSQL database, and N1QL as the query technology.
If you wanted to take this to the next level you could keep track of analytic information. For example, if someone navigates to the root endpoint, increase a counter, or store the browser agent. Simple things to add a cool-factor to the URL shortener application.