Ratnopam Chakrabarti is a software developer currently working for Ericsson Inc. He has been focused on IoT, machine-to-machine technologies, connected cars, and smart city domains for quite a while. He loves learning new technologies and putting them to work. When he’s not working, he enjoys spending time with his 3-year-old son.
This is Part 1 of a series of posts where I’ll introduce you to a photo gallery application which allows users to upload any image of their choice (in .png or .jpg format) and view all the images that have been uploaded by other users. The app lets users “like” a particular image. Once a user uploads an image, the app uses Amazon Web Services’ Rekognition API to scan and detect labels for that image. The labels returned with highest confidence are used to tag the images. The app also lets users click on a particular tag to view all images by that tag.
Demo
Here’s a link to the demo of the finished app.
The Tech Stack
Couchbase – For storing image metadata
Express – For building the front end with node
AWS SDK and S3 – For storing the image files and also for image Rekognition
Node.js – For building the app backend
In addition, jQuery is used to make Ajax calls from the client side to server.js files.
Node.js is used to build all the RESTful APIs that the app leverages. It uses the following dependencies to get the job done:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
package.json "aws-sdk": "*", "couchbase": "^2.3.3", "express": "4.13.0", "formidable": "1.0.17", "gm": "^1.20.0", "hogan-express": "^0.5.2", "knox": "^0.9.2", "socket.io": "1.0.5", "uuid": "^2.0.1" |
Here are what these modules are for:
- The aws-sdk – used to call AWS Rekognition APIs
- knox – used to interact with Amazon S3
- gm – used for resizing the image (we don’t want to store large-sized images onto S3, it has a price tag after all)
- formidable – used to process the input files
- hogan-express – provides Mustache template engine for the express framework
- socket.io – used to emit events from the server to client which can be acted upon
- couchbase – used to interact with the Couchbase database
- uuid – used to generate random document IDs for storing JSON documents in Couchbase
Role of Couchbase
I am using Couchbase in this app to
- store image metadata such as filename
- store other useful information such as number of “likes” for an image and the tags assigned to an image
In this post (Part 1), I’ll describe the steps required to use Couchbase with the photo gallery app. Full source code of the app is available at GitHub.
Install Couchbase and Other Node Modules
To install all the modules listed in package.json above, go to the app root directory and type:
1 |
npm install |
Once all the modules, including Couchbase, are installed, your task now is to use it in the application.
Config.json
Define some of the configurable information in a .json file named config.json.
1 2 3 4 5 6 7 8 9 10 11 |
{ "couchbase": { "server": "127.0.0.1:8091", "bucket": "photogallery" } } |
Here, I have specified the Couchbase Server host and port and also the bucket where the image metadata information will be stored.
server = 127.0.0.1:8091 (if you are running Couchbase locally)
Once the config is defined, the next task is to import this configuration into the express app.
Here’s a snippet of the code from the app:
1 2 3 4 5 6 7 |
var couchbase = require("couchbase"); var config = require("./config"); module.exports.bucket = (new couchbase.Cluster(process.env.COUCHBASE_HOST || config.couchbase.server)).openBucket( process.env.COUCHBASE_BUCKET || config.couchbase.bucket); |
We are making use of some environment variables here to identify the Couchbase host and the bucket. If these are not supplied at runtime, the values default to the config.json mentioned above.
Run and Configure Couchbase
As a prerequisite to run the app locally, one first needs to install and run Couchbase Server. Head over to the COUCHBASE DOWNLOAD PAGE and install as per your own OS. In my case, I’m running a Couchbase 4.6 installation on my Windows system. Once Couchbase is up and running, create a bucket called “photogallery.” This bucket will store all the image meta information. One important thing to note, one must create an index on the bucket before using it. To create an index:
1 |
CREATE PRIMARY INDEX ON <bucketname> |
Once the index is created, head over to the Index tab and the index should be listed.
In a production environment, it’s not recommended to have a global secondary index since it could cause performance degradation. However, for the scale of our app, it’s ok to have it.
Save and Retrieve Document in Couchbase
The photogallery app does the following things:
- Once the user selects the file and clicks upload the app stores the file in an Amazon S3 bucket.
- Once the file is saved to S3 successfully, it calls AWS Rekognition API to assign tags to the image.
- Once the tags are assigned, the app stores the metadata information of the image in the Couchbase photogallery bucket.
To save the data in Couchbase, we need the following snippet of code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 |
// import variables var uuid = require("uuid"); var db = require("../upload").bucket; var config = require("../config"); var N1qlQuery = require('couchbase').N1qlQuery; //define a model function ImageModel() { }; // define the save function ImageModel.save = function(data, callback) { var jsonObject = { filename: data.filename, likes: data.likes, tags: data.tags } // If the document id doesn't exist create a unique id for inserting var documentId = data.id ? data.id : uuid.v4(); db.upsert(documentId, jsonObject, function(error, result) { if(error) { callback(error, null); return; } callback(null, {message: "success", data: result}); }); } |
What the above code does is imports appropriate dependencies to interact with the Couchbase database and defines a function to save the data. A JSON body is created with filename, likes, and tags of an image and then the upsert() function is called to save the data. If a document doesn’t have an “id” attribute assigned, the function creates one using uuid.v4().
For retrieval of documents, take a look at the following code snippet:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
/* * Get all documents from Couchbase Server using N1QL */ ImageModel.getAll = function(callback) { var statement = "SELECT META(photos).id, filename, likes, tags " + "FROM `" + config.couchbase.bucket + "` AS photos"; var query = N1qlQuery.fromString(statement).consistency(N1qlQuery.Consistency.REQUEST_PLUS); db.query(query, function(error, result) { if(error) { return callback(error, null); } callback(null, result); }); }; |
The above code snippet returns all images from the bucket using N1QL querying. This comes in handy to list all images from the photogallery bucket. Note, that the Couchbase bucket stores only the filename, one part of Amazon s3 url part is static and the full url is created by appending the filename to the static s3 url of an image.
Static part -> https://s3.amazonaws.com/la-image-tagger-chakrar27/
Dynamic part -> filename
Here, la-image-tagger-chakrar27 is the s3 bucket I created. To create this bucket, upload this cloudformation template to AWS and create a stack. It will create the s3 bucket.
This post is part of the Couchbase Community Writing Program
[…] Part 1 of this blog series, we looked at how we can store and retrieve image metadata to and from a Couchbase bucket. In this […]