I’ve been following cryptocurrency-related subjects such as Bitcoin for a few months now and I’m very fascinated with everything that has been going on.

As a web application developer, one topic that I’ve been particularly interested in learning more about is around cryptocurrency exchanges and how to make them. From the front, these applications seem to be tools for managing accounts, converting Bitcoin to a fiat currency like USD and back, and transferring Bitcoin to other people, but are they more?

We’re going to look at some examples with Node.js and the NoSQL database, Couchbase, that covers topics modeled around cryptocurrency exchanges.

Updates on related topics:

Disclaimer: I am not a cryptocurrency expert, nor have I taken part in any development around financial services or exchanges. I am an enthusiast of the subject material and anything obtained from this article should be properly tested and used at your own risk.

The Take-Away

There are a few things that you will and won’t get from this particular article. For example, let’s start with the things you won’t get out of this article:

  • We won’t configure any banking or credit card services to transfer fiat currencies such as USD.
  • We won’t be broadcasting any signed transactions to the Bitcoin network, finalizing a transfer.

That said, here are some things you can look forward to learning about in this article:

  • We’ll create a hierarchical deterministic (HD) wallet that can generate an unlimited amount of keys for a given seed, each representing a user wallet.
  • We’ll create user accounts each with wallets based on the master seed.
  • We’ll create transactions that represent deposits, withdrawals, and transfers of funds from the exchange, without actually working with a fiat currency.
  • We’ll lookup balances from the Bitcoin network.
  • We’ll create signed transactions to be broadcasted on the Bitcoin network.

There are many things that we’ll see in this article that can be done way better. If you found something that could be improved, definitely share it in the comments. Like I said, I’m not an expert on the topic, just a fan.

The Project Requirements

There are a few requirements that must be met to be successful with this project:

  • You must have Node.js 6+ installed and configured.
  • You must have Couchbase 5.1+ installed and configured with a Bucket and RBAC profile ready to go.

The main point is that I won’t be going through how to get Couchbase up and running. It isn’t a difficult process, but you’ll need a Bucket set up with an application account and an index for querying with N1QL.

Creating a Node.js Application with Dependencies

We’re going to create a new Node.js application and download the dependencies before we start adding any logic. Create a project directory somewhere on your computer and execute the following commands from a CLI within that directory:

I know I could have done all of the dependency installations in a single line, but I wanted to make them clear to read. So what are we doing in the above commands?

First, we’re initializing a new Node.js project by creating a package.json file. Then we’re downloading our dependencies and adding them to the package.json file via the --save flag.

For this example we’ll be using Express Framework. The express, body-parser, and joi packages are all relevant towards accepting and validating request data. Because we’ll be communicating with public Bitcoin nodes, we will be using the request and request-promise promise wrapping package. The very popular bitcore-lib package will allow us to create wallets and sign transactions while the bitcore-mnemonic package will allow us to generate a seed that can be used for our HD wallet keys. Finally, couchbase and uuid will be used for working with our database.

Now we probably want to better structure our project. Add the following directories and files within your project directory if they do not already exist:

All of our API endpoints will be split into categories and placed in each appropriate routing file. We don’t have to do this, but it makes our project just a little bit cleaner. To remove a ton of Bitcoin and database logic out of our routes, we’ll be adding everything that isn’t data validation into our classes/helper.js file. The config.json file will have all of our database information as well as our mnemonic seed. In a realistic scenario, this file should be treated like gold and receive as much protection as it could possibly receive. The app.js file will have all of our configuration and bootstrapping logic towards connecting our routes, connecting to the database, etc.

As a convenience, we’re going to add one more dependency to our project and set it up:

The nodemon package will allow us to hot-reload our project every time we change a file. It is not a requirement, but can save us some time as we’re building.

Open the package.json file and add the following script to make it happen:

We can start the development process of our application at this point.

Developing the Database and Bitcoin Logic

When it comes to developing our application, before we start worrying about API endpoints, we want to create our database and Bitcoin related logic.

We’re going to be spending our time in the project’s classes/helper.js file. Open it and include the following:

We’re going to be passing this class around as a singleton for our application. In the constructor method, we are establishing a connection to our database cluster, opening a Bucket, and authenticating. The open Bucket will be used throughout this helper class.

Let’s knock out the Bitcoin logic before the database logic.

If you’re not familiar with HD wallets, they are essentially a wallet derived of a single seed. Using the seed you can derive children and those children can have children, and so on and so forth.

The master variable in the createKeyPair function represents the top level seed key. Each user account will be a direct child of that key, hence we are deriving a child based on an account value. The account value is a person number and every account created will get an incremental number. However, we’re not going to generate account keys and call it a day. Instead, each account key will have 10,000 possible private and public keys in case they don’t want to use the same key more than once. Once we’ve generated a key at random, we return it.

Similarly, we have a getMasterChangeAddress function like the following:

When we start creating accounts, they will start at one, leaving zero for the exchange or the web application, or whatever you want to call it. We are also allocating 10 possible addresses to this account. These addresses will do two possible things. The first is that they will hold Bitcoin to transfer to other accounts and the second is that they will receive remainder payments, otherwise known as the change. Remember, in a Bitcoin transaction, all unspent transaction output (UTXO) must be spent, even if it is less than the desired amount. This means that the desired amount gets sent to the destination and the remainder gets sent back into one of these 10 addresses.

Are there other ways or better ways to do this? Absolutely, but this one will work for the example.

To get a balance for any address we use or generate using the HD seed, we can use a public Bitcoin explorer:

The above function will take an address and get the balance in decimal format as well as satoshis. Going forward, the satoshi value is the only relevant value to us. If we have X number of addresses for a given account, we can get the total balance using a function like this:

In the above getWalletBalance function we are making a request for each address and when they’ve all completed, we can add the balances and return them.

It takes a little more than just an address balance to be able to transfer cryptocurrency. Instead we need to know the unspent transaction output (UTXO) for a given address. This can be found using the same API from BitPay:

If there is no unspent transaction output, it means there is nothing we can transfer and we should throw an error instead. Having enough to transfer is a different story.

For example, we could do something like this:

In the above function, we are taking a list of addresses and checking to see which one holds an amount greater than the threshold that we provide. If none of them have enough balance, we should probably relay that message.

The final utility related function, is something we’ve kind of already seen:

The above function will get us all master keys, which will be useful for signing and checking value.

Just to reiterate, I’m using a finite value to how many keys are generated. You may or may not want to do the same, it is up to you.

Now let’s dive into some NoSQL logic for storing our application data.

As of right now there is no data in our database. The first logic step might be to create some data. While it isn’t particularly difficult standalone, we can create a function like this:

Essentially, we’re accepting an object and an id to be used as a document key. If a document key isn’t provided, we’ll automatically generate it. When all said and done, we’ll return what was created including the id in the response.

So let’s say we want to create a user account. We can do the following:

Remember, accounts are driven by an auto incrementing numeric value for this example. We can create incrementing values by using a counter in Couchbase. If the counter doesn’t exist, we’ll initialize it at 1 and increment it every next call. Remember, 0 is reserved for application keys.

After we get our counter value, we add it to the object that was passed and call our insert function, which in this case generates a unique id for us.

We haven’t seen it yet because we don’t have any endpoints, but lets assume that when we create an account, it has no address information, only an account identifier. We might want to add an address for the user:

When adding an address, we first get the user by the document id. When the document is retrieved, we get the numeric account value and create a fresh keypair of our 10,000 options. Using a sub-document operation, we can add the keypair to the user document without ever having to download the document or manipulating it.

Something very serious to note about what we’ve just done.

I am storing the unencrypted private key and public address in the user document. This is a big no-no for production. Remember all those stories you read about where people got their keys stolen? In reality, we’d want to encrypt the data before inserting it. We can do that by using the Node.js crypto library, or if we’re using Couchbase Server 5.5, the Node.js SDK for Couchbase offers encryption. We won’t explore it here though.

Okay, we’ve got account data and addresses in the database now. Let’s query for that data:

The above getAddresses function can do one of two things. If an account was provided, we’ll use a N1QL query to get all addresses for that particular account. If no account was provided, we’ll get all addresses for every account in the database. In both scenarios, we’re only getting public addresses, nothing sensitive. Using a parameterized N1QL query, we can return the database results back to the client.

Something to note in our query.

We’re storing our addresses in an array in the user documents. Using an UNNEST operator, we can flatten those addresses and make the response more attractive.

Now let’s say we have an address and we want to get the corresponding private key. We might do the following:

Given a particular account, we create a query similar to what we saw previously. This time, after we UNNEST, we do a WHERE condition to give us results only for the matching address. If we wanted to we could have done an array operation instead. With Couchbase and N1QL, there are numerous ways to solve a problem.

We’re going to change gears a bit here. Up until now we’ve done account oriented operations in our NoSQL database. Another important aspect is transactions. For example, maybe user X deposits some USD currency for BTC and user Y does a withdrawal. We need to store and query for that transaction information.

The API endpoint functions will save the transaction data, but we can still query for it.

Given an account, we want to get an account balance for a particular user.

Wait a second, let’s take a step back because didn’t we already create some account balance functions? Technically we did, but those functions were for checking the wallet balance, not the account balance.

This is where some of my experience turns into gray area. Every time you transfer Bitcoin, there is a fee involved, and sometimes it is quite expensive. When you make a deposit, it isn’t cost effective to transfer money into your wallet because it would be charged a miner fee. Then you would be charged to withdraw and even transfer again. You’ve already lost most of your Bitcoin at that point.

Instead, I believe exchanges have a holding account similar to a stock exchange money market account. There is record of the money that you should have in your account, but it isn’t technically in a wallet. When you want transfer, you’re transferring from the application address, not your user address. When you withdraw, it is just being subtracted.

Again, I don’t know if that is truly how it works, but that is how I’d do it in order to avoid fees everywhere.

Going back to our getAccountBalance function. We’re taking a sum of every transaction. Deposits have a positive value while transfers and withdrawals have a negative value. Aggregating this information together should give you an accurate number, excluding your wallet balance. Later we’ll be getting an account with wallet balance.

Given the little that we know about account balances, we can try to create a transaction from our wallet:

If provided a source address, destination address, and amount, we can create and sign a transaction to be later broadcasted on the Bitcoin network.

First we get the balance for the source address in question. We need to make sure it has enough UTXO to meet the send amount expectation. Note that in this example, we’re doing single address transactions. If you wanted to get complicated, you could send from multiple addresses in a single transaction. We’re not going to do that here. If our single address has enough funds, we get the private key for it and the UTXO data. With the UTXO data we can create a Bitcoin transaction, apply the destination address and a change address, then sign the transaction using our private key. The response can be broadcasted.

Similarly let’s say we wanted to transfer Bitcoin from our holding account:

We’re assuming that our exchange addresses have been loaded with an insane amount of Bitcoin to meet demand.

The first step is to make sure we have funding in our holding account. We can execute that query that sums up each of our transactions to get a valid number. If we have enough, we can get all 10 of our master key pairs and the addresses. We need to check which address has enough funds to send. Remember, single address transactions here, when there could be more.

If an address has enough funds, we get the UTXO data and start making a transaction. This time instead of our wallet as the source address, we use the wallet of the exchange. After we get a signed transaction, we want to create a transaction in our database to subtract the value that we’re transferring.

Before we move onto the API endpoints, I want to re-iterate a few things:

  • I’m assuming that popular exchanges have a holding account to avoid fees imposed on wallet addresses.
  • We are using single address transactions in this example, rather than aggregating what we have.
  • I’m not encrypting the key data in the account documents, when I should be.
  • I’m not broadcasting any transactions, only creating them.

Now let’s focus on our API endpoints, the simple part.

Designing RESTful API Endpoints with Express Framework

Remember, as we configured in the beginning, our endpoints will be split across three files which act as groupings. We’ll start with the smallest and simplest group of endpoints, which are more for utility than anything else.

Open the project’s routes/utility.js file and include the following:

Here we have two endpoints, one for generating mnemonic seeds and the other for getting the fiat value of a Bitcoin balance. Neither are truly necessary, but on first launch, it might be nice to generate a seed value to later save in our config file.

Now open the project’s routes/account.js file so we can handle account information:

Notice that we’re pulling the helper class from the app.js file that we haven’t started yet. Just go with it for now and it will make sense later, although it isn’t anything special.

When it comes to creating accounts, we have the following:

Using Joi we can validate the request body and throw errors if it isn’t correct. Assuming the request body is correct, we can call our createAccount function to save a new account in the database.

With an account created, we can add some addresses:

Using the account id that was passed, we can call our addAddress function to use a subdocument operation on our document.

Not so bad right?

To get all addresses for a particular account, we might have something like the following:

Alternatively, if we don’t provide an id, we can get all addresses from all accounts using the following endpoint function:

Now for probably the trickiest endpoint function. Let’s say we want to get the balance of our account, which includes the holding account as well as each of our wallet addresses. We can do the following:

The above will call both of our functions for obtaining balance, and add the results together to get one massive balance.

The account endpoints weren’t particularly interesting. Creating transactions is a bit more exciting.

Open the project’s routes/transaction.js file and include the following:

We have three different types of transaction. We can deposit fiat currency for Bitcoin, withdraw Bitcoin for fiat currency, and transfer Bitcoin to new wallet addresses.

Let’s take a look at the deposit endpoint:

After we validate the input, we check the current value of Bitcoin in USD with CoinMarketCap. Using the data in the response, we can calculate how many Bitcoin we should get based on the USD amount deposited.

After creating a database transaction, we can save it and since it is a positive number, it will come back as positive balance when querying.

Now let’s say we want to withdraw money from our Bitcoin:

Similar events are happening here. After validating the request body, we get our account balance and make sure the amount we’re withdrawing is less than or equal to our balance. If it is, we can do another conversion based on the current price from CoinMarketCap. We’ll create a transaction using a negative value and save it to the database.

In both these instances we are relying on CoinMarketCap which has had negative controversy in the past. You may want to pick a different resource for conversions.

Finally we have transfers:

If the request contains a source address, we are going to transfer from our own wallet, otherwise we are going to transfer from the wallet the exchange manages.

All of this is based on functions that we had previously created.

With the endpoints out of the way, we can focus on bootstrapping our application and come to a conclusion.

Bootstrapping the Express Framework Application

Right now we have two files that remain untouched by the example. We haven’t added a configuration or driving logic to bootstrap our endpoints.

Open the project’s config.json file and include something like the following:

Remember this file is incredibly sensitive. Consider locking it down or even using a different approach. If the seed is exposed, every private key for all user accounts and the exchange account can be obtained with zero effort.

Now open the project’s app.js file and include the following:

What we’re doing is initializing Express, loading the configuration information, and linking up our routes. The module.exports.helper variable is our singleton that will be used in every other JavaScript file.

Conclusion

You just saw how to get started with building your own cryptocurrency exchange using Node.js and Couchbase as the NoSQL database. We covered a lot, from generating HD wallets to creating endpoints with complex database logic.

I cannot stress this enough though. I am a cryptocurrency enthusiast and have no true experience in the financial space. The things I shared should work, but can be done a whole lot better. Don’t forget to encrypt your keys and keep your seed safe. Test your work and know what you’re getting yourself into.

If you’d like to download this project, check it out on GitHub. If you’d like to share insight, experience, ect., on the topic, please share it in the comments. The community can work towards creating something great!

If you’re a fan of Golang, I created a similar project in a previous tutorial.

Author

Posted by Nic Raboy, Developer Advocate, Couchbase

Nic Raboy is an advocate of modern web and mobile development technologies. He has experience in Java, JavaScript, Golang and a variety of frameworks such as Angular, NativeScript, and Apache Cordova. Nic writes about his development experiences related to making web and mobile development easier to understand.

3 Comments

  1. IMHO bitcoin exchange is fairly sub-optimal choice for article about Couchbase. Anything that touches money needs reliable “D” in ACID (i.e. with 4 or more nines at least). I don’t think Couchbase has it (think node dying and auto-failover happening and people loosing money). Another issue is consistency. Code above has race between checking balance and saving withdrawal transaction, which highlights it pretty clearly.

    Another issue is that checking balance is O(N) in number of account’s transactions. CouchDB materialized views can do account balance fetch in O(log N), but of course this is still very slow log N. AFAIK no niql indexes are capable of that.

    IMHO this article ends up highlighting weaknesses of couchbase. I.e. because couchbase is not suitable as backend of bitcoin exchange. Article like that could be great fit for something like CockroachDB or cloud spanner.

    1. I didn’t demonstrate it in this article, but you can define your own durability requirements via the SDK. If you’re concerned about consistency and losing money, you can change the settings to respond only after it has persisted to disk or only after it has replicated X number of times.

      https://developer.couchbase.com/documentation/server/current/sdk/durability.html

      When it comes to querying you can set the querying consistency. You can wait until the index refreshes if you’d like.

      https://developer.couchbase.com/documentation/server/current/indexes/performance-consistency.html

      I see where you’re coming from, but I don’t think it is as big an issue as you might think.

      Thanks for sharing your two cents though :-)

  2. Sorry about that, I had it private. It is now public.

Leave a reply