If you’re running a Couchbase Server deployment, did you know you can easily migrate to Couchbase Cloud on AWS and instantly gain the scale and efficiency of a managed cloud service model while offloading the work of installations, upgrades and general database infrastructure maintenance? Couchbase Cloud is a fully managed NoSQL Database-as-a-Service (DBaaS) for Couchbase Server. It provides massive scale, allows you to create multi-node clusters in minutes, and supports all the Couchbase features you use today, such as N1QL (SQL for JSON), Full Text Search, Eventing and Analytics. Couchbase Cloud gives you all the power of Couchbase while freeing you up to focus on your application, not your database deployment. What’s best is that Couchbase Cloud is built to connect to any other Couchbase Server for replication, making basic migration a snap. There has never been a better time for migrating from Couchbase Server to Couchbase Cloud on AWS!

Your guide to the cloud

If you’re ready to make the move to Couchbase Cloud, we have a great resource for you! Working with AWS, we recently launched this AWS Prescriptive Guidance pattern, which describes a repeatable process for migrating data and indexes from a self-managed Couchbase Server Enterprise Edition environment to Couchbase Cloud on the Amazon Web Services (AWS) Cloud.

The pattern should be used by any Couchbase customer who is new to Couchbase Cloud and who needs to understand the order of activities for migration.

The basic steps are:

Prepare the migration

  • Review configurations for the nodes and buckets on your self-managed cluster on Couchbase Server.
  • Open the appropriate required inbound TCP ports on the self-managed cluster’s host system to allow Couchbase Cloud to connect for Cross Data Center Replication (XDCR).

Deploy and configure resources on Couchbase Cloud

  • Connect Couchbase Cloud to AWS, configure and deploy the destination cluster on Couchbase Cloud.
    • Use Self-Managed cluster and bucket configurations gathered in step 1 as a general guide for Couchbase Cloud cluster configurations.
  • Configure data access on Couchbase Cloud.
    • Create a database user for the destination destination cluster.

Migrate the data and indexes

  • Set up the connection to the self-managed Cluster connection from the Couchbase Cloud cluster
  • Set up XDCR Replication between the source and destination buckets
  • Migrate indexes using Couchbase Shell from the self-managed cluster to the Couchbase Cloud cluster
  • Verify and validate that the data and indexes are successfully migrated

Since the pattern already covers the individual granular steps for migration, in this post I’ll discuss my experience going through them myself, and I’ll point out a few tips along the way. For the exercise it’s assumed I already have a Couchbase Cloud account, as well as a self-managed Couchbase Server cluster, as such I’ll focus mainly on the actual process of migrating from one environment to the other.

Review The Source and Destination For Migration

A first step for any migration is careful planning, in short, you want to make sure that you create a destination environment that matches the source as closely as possible, which makes for smoother migration. The pattern describes general steps for reviewing your source environment, then using the gathered information to create a similar environment on the destination. Remember that Couchbase Cloud uses multi-dimensional scaling best practices. Services and nodes can only be chosen according to deployment best practices. This could mean that you can’t exactly match your self-managed Couchbase Server cluster’s configurations.

It’s important to note that the pattern provides general guidance for assessing the size of your self-managed Couchbase Server cluster and using it to approximate the required configurations for your Couchbase Cloud destination cluster. For assistance with a more detailed Couchbase Cloud sizing exercise please contact Couchbase Cloud or your Couchbase customer account team.

Once your destination cluster is created, you create your destination buckets. It’s a best practice to create your destination buckets with the same names, memory settings, and conflict resolution methods of the buckets in your self-managed Couchbase Server cluster. 

My Environments

To run through the steps, I first established my source – a three node cluster running Couchbase Server Enterprise Edition 6.6.1 deployed on Microsoft Azure Virtual Machines. Next I established my destination cluster on Couchbase Cloud – using the Control Plane I created a cloud on AWS, then for my cluster I used the “Evaluation Optimized” configuration template, giving me 3 nodes with all services running on each node. With source and destination environments established, I was ready to begin the process. For the exercise I chose to migrate the “travel-sample” sample database that comes bundled with Couchbase Server, since it has a fair amount of data and a lot of indexes, what better way to test the steps?

On Couchbase Server I simply installed the “travel-sample” sample bucket. Then on Couchbase Cloud, I manually created a bucket called “travel-sample” to serve as the destination for migrated data. Note that I did not import the “travel-sample” sample database that comes with Couchbase Cloud, I simply created an empty bucket with the same name as my source bucket on Couchbase Server to serve as the destination for migration.

Connecting Self-Managed Clusters to Couchbase Cloud, what a concept!

To make the connection between my self-managed Couchbase Server cluster and Couchbase Cloud, I leveraged a fantastic feature on Couchbase Cloud, the ability to connect directly to your self-managed cluster, specifically for leveraging XDCR.

The pattern walks you through the process of connecting a self-managed cluster, but I wanted to point out a couple of important things to keep in mind as you go through the steps:

  1. Make sure you’ve opened TCP ports 8091, 8093, 8095 for inbound traffic on your self-managed cluster system – this allows the Couchbase Cloud Control Plane to connect to your system.
  2. Make sure you know the Fully Qualified Domain Name, and make sure you know the IP addresses of every node in your self-managed cluster system.
  3. When entering the self-managed cluster settings, make sure you use the self-managed Couchbase Server cluster’s FQDN in the Host Name property and enter the IP addresses of the destination cluster nodes in the Public/NAT gateway IPs section. The IP addresses must be separated by commas.

self-managed cluster in Couchbase Cloud

Once I had the settings in place, I clicked submit and in just a few seconds I had connected a self-managed cluster in Couchbase Cloud. Now I was ready to migrate data!

XDCR Rocks!

After the self-managed cluster was added to Couchbase Cloud, I was immediately presented with a button to “Create a Replication” for the cluster. I simply clicked the button, selected my destination cluster on Couchbase Cloud, then selected my self-managed “travel-sample” bucket as a source and the newly created empty “travel-sample” bucket that I created on Couchbase Cloud as the destination. Once I saved the replication setting, data began replicating from my self-managed cluster to my Couchbase Cloud cluster almost immediately, it could not have been any easier! And the replication is secure and resilient, allowing for efficient migration of even the largest buckets of data.

Replicating to Couchbase Cloud

Index Migration With cbshell? Shell Yeah!

Next I set out to migrate indexes. Of course it’s very easy to manually migrate indexes by copying the index definition statements from your self-managed cluster and running them in the Query Workbench on Couchbase Cloud, but I had enough indexes that I wanted a more streamlined approach. Couchbase Shell provided the perfect method!

Couchbase Shell, also known as cbshell, is a modern command line shell for Couchbase Server and Couchbase Cloud. The cool thing about cbshell is that you can install it anywhere to perform the migration, all it needs is internet access to both the source and destination clusters, and you can use simple commands in a terminal to migrate the indexes.

On the system where you install cbshell, make sure that Couchbase SDK and OpenSSL are also installed, with both mapped in the system user profile PATH.

Once cbshell was installed, I was able to create a config dotfile in order to connect to my source and destination clusters at the same time, and easily switch context between them. The config file contains connection information for both clusters, as well as root certificates to ensure a secure index migration. The pattern contains a sample config file that you can use by simply copying the sample, editing the properties in the file to add your own cluster endpoints, user/password and root certificates, then saving it to the .cbsh folder.

Once the config file was saved, I launched cbshell in a terminal and set the context to my self-managed cluster:


use cluster On-Prem-Cluster

Then I ran a simple command to migrate the index definitions from my self-managed cluster to my Couchbase Cloud cluster. The granular details are in the pattern, but to give you an idea of how easy the command is, it looks like this (substituting your own bucket name in place of MYBUCKETNAME):

query indexes --definitions | where bucket = "MYBUCKETNAME"| get definition | each { query $it --cluster Cloud-Cluster }

This command effectively took all the indexes from the self-managed cluster “travel-sample” bucket and created them on my destination cluster on Couchbase Cloud.

Then in cbshell I switched context to my Couchbase Cloud cluster:

use cluster Cloud-Cluster

And the final step was to use a simple cbshell command to build the index definitions that were defined as “defer_build”:true.

query "BUILD INDEX ON MYBUCKETNAME ((SELECT RAW name FROM system:indexes WHERE keyspace_id = 'MYBUCKETNAME' AND state = 'deferred'));"

Couchbase Shell Migrating Indexes

And that was it, my indexes were all migrated! Couchbase Shell was so easy, and it was fun to watch the indexes appearing magically in Couchbase Cloud after running the cbshell commands.

To test the migration I ran a few sample N1QL queries on both the source and destination clusters and verified the same results on both environments, SUCCESS!

The experience was straightforward and in the end went very fast after setting up the environments – of course the speed of migration can vary depending on data volume. The pattern is designed as a repeatable process, and is so easy I now use it regularly when moving demo databases from my self-managed Couchbase Server clusters to Couchbase Cloud.

Take advantage of the AWS Workload Migration Program for Couchbase and $ave! 

If you’re ready to migrate to Couchbase Cloud on AWS, you may qualify for the AWS Workload Migration Program for Couchbase, which offers significant savings for eligible customers on the first year AWS infrastructure spend in the form of AWS credits for Couchbase Cloud migrations.

The program supports your migration journey by leveraging best practices and a repeatable process and methodology for migrating to Couchbase on AWS infrastructure. Eligible workloads receive an AWS Credit to offset the initial cost of using the AWS Infrastructure-as-a-Service. It is intended to help customers migrate their database workloads to Couchbase on AWS across a variety of use cases, including from self-managed Couchbase Server deployments to Couchbase Cloud.

Click HERE to learn more and find out if you are eligible for the program!


Posted by Mark Gamble, Dir Product & Solutions Mktg, Couchbase

I am a passionate product marketer with a technical and solution consulting background and 20+ years of experience in Enterprise and Open Source technology. I have launched several database and analytic solutions throughout my career, and have worked with customers across a wide variety of industries including Financial Services, Automotive, Hospitality, High-Tech and Healthcare. I have particular expertise in analytics and AI, love all things data, and am an emphatic supporter of data-for-good initiatives.

Leave a reply