Steps to migrate self-managed Couchbase Server clusters to Couchbase Capella™ Database-as-a-Service (DBaaS)

Couchbase Capella is the easiest and fastest way to begin with Couchbase. This fully managed Database-as-a-Service eliminates your database management efforts and reduces costs while delivering flexibility across all use cases with built-in multi-model capabilities. Its memory-first architecture drives high speed data response at scale, resulting in the best price-performance of any fully managed document database.

Why migrate?

If you’re running a self-managed Couchbase Server implementation, you and your team are probably responsible for handling database maintenance tasks that take time away from developing your applications. As a managed service, Couchbase Capella removes the need to take care of installations, upgrades and general database maintenance – it’s all handled for you. And for those of you using the Community Edition of Couchbase Server, Couchbase Capella can provide a fast and easy way to level up your database with more nodes and Enterprise features. Couchbase Capella supports all Couchbase Server services and capabilities.

This blog provides steps for migrating data and indexes from your Couchbase Server clusters, running either on premises or in the cloud, to Couchbase Capella.

Note: These migration steps apply to migrating from self-managed instances of:

    • Couchbase Server Enterprise Edition v6.6, 7.0.x, 7.1
    • Couchbase Server Community Edition v6.6, 7.0.x, 7.1

Prerequisites 

    • An existing Couchbase Capella account. You can also create and use the Couchbase Capella self-service free trial – NOTE the self service free trial is restricted to a single node and 4 services (for more information on the free trial, please refer to Capella documentation). Follow the instructions in the Getting Started with Couchbase Capella tutorial to get started.
    • An existing self-managed Couchbase Server environment either on premises or deployed on a cloud service provider.

Assumptions

    • Familiarity with administering Couchbase Server and Couchbase Capella
    • Familiarity with running commands in a command line interface (CLI)

Limitations 

This guide is used to migrate data and indexes from Couchbase Server to Couchbase Capella. The guide does not apply to migrating the Couchbase Eventing Service, Couchbase Full-Text Search (FTS) indexes or Couchbase Analytics.

Couchbase versions

    • Couchbase Server (Community Edition or Enterprise Edition) versions 6.6, 7.0.x, 7.1

Utilities

    • Couchbase cbbackupmgr – cbbackupmgr is a Command Line Interface (CLI) tool for managing the backup and restore of Couchbase Server data. It is used for migrating data and indexes from the following versions of self-managed Couchbase Server to Capella:
      • Enterprise Edition version 6.6, version 7x
      • Community Edition version 7x
      • Steps for using cbbackupmgr for migration are detailed in the section of this guide titled “Migrate the data and indexes. Option 1”
    • Couchbase cbbackup and Couchbase cbrestore – cbbackup and cbrestore are CLI tools used for migrating data from the following version of self-managed Couchbase Server to Capella:
      • Community Edition version 6.6
      • Steps for using cbbackup and cbrestore for migration are detailed in the section of this guide titled “Migrate the data and indexes. Option 2”
    • Couchbase XDCR – Couchbase cross data center replication (XDCR) allows data to be replicated across clusters. It is used in this guide for ongoing migration from self-managed Couchbase Server clusters to Capella clusters.
      • Note:
        • XDCR cannot be used with Couchbase Server Community Edition

Important: The Prepare for migrating section of this document provides general guidance for assessing the size of your self-managed Couchbase Server cluster and using it to identify the required configurations for your Couchbase Capella cluster. For help with a more detailed Couchbase Capella sizing exercise, please contact Couchbase.

Prepare for migrating

Important: If you are using the Couchbase Capella free trial, you can migrate data following this guide, but you will not be able to deploy a multi-node Capella environment due to trial configuration restrictions. We recommend converting the free trial to a paid account before beginning a migration in order to configure a full multi-node deployment. To convert your account, open the Billing section of the Couchbase Capella UI and then choose Add Activation ID.

Review your self-managed Couchbase Server cluster:

    • Log in to your self-managed Couchbase Server Web Console and assess your cluster’s nodes and buckets. 
    • Choose the “Servers” tab in the main navigation to show a list of cluster nodes. Record the number of nodes and then choose each node on the list to display its properties. 
    • Record the memory and storage for each individual node.
    • Choose the “Buckets” tab in the main navigation and then choose each bucket in the list to display its properties. 
    • Record the RAM quota and conflict resolution setting for each bucket.
    • You will use your self-managed Couchbase Server cluster configurations as a general guide for sizing and configuring the destination cluster on Couchbase Capella.

NOTE: You do not need to duplicate the exact cluster configuration on Capella. This is especially true if you are using Capella Trial and that comes with a fixed configuration.

Note the Couchbase Service distribution on the self-managed Couchbase Server cluster:

Record the IP address(es) of the self-managed Couchbase Server cluster nodes:

Skip this step if you are using Community Edition

    • Record the IP address for each node in your cluster, this is for whitelisting them on your Couchbase Capella cluster later.
    • TIP: To get the IP of each node in your self managed cluster:
      • SSH into the node
      • Issue this command: “dig +short myip.opendns.com @resolver1.opendns.com

Deploy and configure a cluster on Couchbase Capella

Create and configure a cluster on Couchbase Capella:

    • Log in to the Couchbase Capella UI, choose the “Clusters” tab in the main navigation menu and then choose “Create Cluster.” (In the free trial, you should already have a cluster created for you, and you cannot further customize the sizing).
    • Use the information that you recorded from the review of your self-managed Couchbase Server cluster and choose the cluster template that meets the configuration’s requirements. 
    • If you don’t find an appropriate template, choose “Custom Template” in the “Cluster Sizing” editor. 
    • Choose and configure the nodes to match your self-managed Couchbase Server cluster environment, including number of nodes, services distribution, compute or RAM, and storage.
    • Choose a support zone and support package, and then deploy the cluster. For detailed steps and instructions on creating a cluster, see Create a cluster in the Couchbase Capella documentation.

Important: Couchbase Capella uses multi-dimensional scaling guidelines, and services and nodes can only be chosen according to deployment guidelines.

 

Create a database credentials user:

    • A database credentials user is specific to a cluster, and consists of a username, password, and a set of bucket privileges. This user is required for accessing bucket data.
    • In the Couchbase Capella UI, create a database credential user for the new cluster by following the instructions in Configure database credentials from the Couchbase Capella documentation.

Record Couchbase Capella cluster’s connection endpoint and add IP address as “Allowed IP”.
Save Root Certificates for self-managed Couchbase Server and Couchbase Capella:

    • In the Couchbase Capella UI, choose “Clusters”, and then choose your destination cluster. Choose the “Connect” tab for the cluster and record the connection endpoint for your cluster under “Wide Area Network.”
    • NOTE: If your self-managed Couchbase Server is running version 7.0.x, you will need to copy the hostname url for a DATA SERVICE NODE instead of the Wide Area Network endpoint.
    • In the Couchbase Capella UI, choose the “Nodes” tab for the destination cluster, all nodes in your cluster will be listed. Choose one of the nodes that is running the DATA SERVICE, and record the HOSTNAME for the node.

Add the IP address of the system where you will run the command line tools for migrating data as an Allowed IP. Under “Wide Area Network,” click “Manage Allowed IP,” then click “Add Allowed IP.” Enter the IP address and click “Add IP.” For more information about allowed IPs, see Configure allowed IP addresses in the Couchbase documentation.

Note: If adding an allowed IP fails, in most cases it is related to firewall issues. See Couchbase documentation for information on the default ports used by Couchbase Server: https://docs.couchbase.com/server/current/install/install-ports.html

Download the root certificate for your Capella cluster. Under “Root Certificate,” click “Download.” Save the root certificate as a .pem file extension in a folder on the system that will run Couchbase CLI tools.

Next, log in to your self-managed Couchbase Server Web Console. Copy the root certificate for your self-managed Couchbase Server cluster and save it as a .pem file to the same folder where you saved the root certificate file for your Couchbase Capella cluster. For more information about the root certificate, see Root certificate in the Couchbase Server documentation.

 

Create target buckets on Couchbase Capella:

    • Log in to the Couchbase Capella UI.
    • Create one target bucket in your Couchbase Capella cluster for each source bucket by following the instructions from Create a bucket in the Couchbase Capella documentation.

Important: Bucket names cannot contain an underscore

Migrate using cbbackupmgr

This option applies to the following versions of self-managed Couchbase Server:

    • Enterprise Edition version 6.6, version 7.0.x, version 7.1
    • Community Edition version 7.0.x, version 7.1

Use cbbackupmgr CLI tool:

    • cbbackupmgr is located in the root directory of your self-managed Couchbase Server installation.
    • You use a command line terminal to work with cbbackupmgr.

Depending on the platform, cbbackupmgr is installed with Couchbase Server in the following location:

Linux: /opt/couchbase/bin/cbbackupmgr

Windows (assuming default installation): C:\Program Files\Couchbase\Server\bin\cbbackupmgr

Mac OS X: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/cbbackupmgr

TIP: If you do not have direct access to the install directory for Couchbase Server, you can get the cbbackupmgr command line utility by downloading Couchbase Server and installing it on your local system.

Create a backup repository on the self-managed Couchbase Server system:

    • In the command line terminal, create a directory to serve as a repository for local backups from the self-managed Couchbase Server. The following command examples use Linux syntax, adjust accordingly for your command line terminal and platform
    • The following command will create the directory in your systems root folder:
      mkdir /backups/
    • Run the following cbbackupmgr command to create a repository in the new directory, replacing “<SELF-MANAGED-BUCKET-NAME>” with the actual name of the bucket you wish to migrate:

NOTE:

    • This Repository backs up only the specified bucket
    • Analytics and Eventing are explicitly disabled

Run the following command to examine the repo:

cbbackupmgr info -a /backups -all

Basic information about your repository will be displayed.

Backup the self-managed Couchbase Server bucket:

Run the following command to create the backup:

Note: Make the following substitutions in the command:

    • Replace <SELF-MANAGED-BUCKET-NAME> with the actual name of the bucket you wish to migrate
    • Replace <SELF-MANAGED-SERVER-ADMIN> with the administrator user name on the self-managed Couchbase Server
    • Replace <SELF-MANAGED-SERVER-ADMIN-PWD> with the administrator password on the self-managed Couchbase Server
    • Replace <FULL-PATH-TO-SELF-MANAGED-ROOT-CERT> with the full file path to the self-managed Couchbase Server root certificate (as saved in earlier step)

An example command would look like so:

Output from the command will look like so:

cbbackupmgr output displayed

Examine the backup using the command:

cbbackupmgr info -a /backups -all

Basic information about your backup will be displayed.

Restore the backed up self-managed bucket to Capella:

In the command line terminal, use the following command to restore the backed up self-managed bucket to Capella:

Note: make the following substitutions in the command:

    • Replace <SELF-MANAGED-BUCKET-NAME> with the actual name of the bucket you with to migrate
    • Replace <CAPELLA-ENDPOINT> with the Capella cluster endpoint url recorded earlier.
      • NOTE: If your self-managed Couchbase Server is running version 7.0.x, you will need to use the hostname url for a DATA SERVICE NODE instead of the Wide Area Network endpoint.
      • In the Couchbase Capella UI, choose the “Nodes” tab for the destination cluster, all nodes in your cluster will be listed. Choose one of the nodes that is running the DATA SERVICE, and record the HOSTNAME for the node.
    • Replace <CAPELLA-DATABASE-CREDENTIALS-USER> with the database credentials user on the Capella cluster
    • Replace <CAPELLA-DATABASE-CREDENTIALS-PASSWORD> with the database credentials user password
    • Replace <FULL-PATH-TO-CAPELLA-ROOT-CERT> with the full file path to the Capella cluster root certificate (as saved in earlier step).

The command may take a few mins to run depending on the size of your cluster. Once completed you should see details about the restored backup displayed in the terminal, and a message reading “Restore completed successfully

Examine the migrated documents and indexes:

    • In the Couchbase Capella UI, choose “Tools>Documents” for your target cluster.
    • You should see that all of your self-managed cluster documents have been migrated to Capella.
    • In the Couchbase Capella UI, choose “Tools>Indexes” for your target cluster.
    • You should see that all of your self-managed cluster indexes have been migrated to Capella.

NOTE: Any indexes with definitions set as defer_build:true will be migrated, but will be listed as CREATED. These indexes still need to be built on Capella.

 

Build indexes on Capella:

  • In the Couchbase Capella UI, choose “Tools>Query Workbench” for your target cluster.
  • The following query will create BUILD statements for any indexes that were created but not built during migration. Copy/paste the following query to the Query Workbench in Capella, then click “Execute”:

The query results will return a BUILD statement to build the created indexes. Note there will be a BUILD statement for each scope in your cluster. The results will look something like so:

Copy each BUILD statement (between the double quotes), then paste to the Query Editor and click “Execute” to build the indexes. Repeat for each BUILD statement

Congratulations, you have just migrated data and indexes from your self-managed Couchbase Server cluster to Capella!

 

Migrate using cbbackup and cbrestore

This option applies to the following versions of self-managed Couchbase Server:

  • Community Edition version 6.6

Use the cbbackup CLI tool:

    • cbbackup is located in the root directory of your self-managed Couchbase Server installation.
    • You use a command line terminal to work with cbbackup.
    • Depending on the platform, cbbackup is installed with Couchbase Server in the following location:
      • Linux/opt/couchbase/bin/cbbackup
      • Windows (assuming default installation): C:\Program Files\Couchbase\Server\bin\cbbackup
      • Mac OS X: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/cbbackup

TIP: If you do not have direct access to the install directory for Couchbase Server, you can get the cbbackup and cbrestore command line utilities by downloading Couchbase Server and installing it on your local system.

 

Create a backup on the self-managed Couchbase Server system:

In the command line terminal, create a directory for local backups from the self-managed Couchbase Server. The following command will create the directory in your system root folder:

mkdir /backups/

Run the following cbbackup command to create the backup:

Note: make the following substitutions in the command:

    • Replace <SELF-MANAGED-SERVER-ADMIN> with the administrator user name on the self-managed Couchbase Server
    • Replace <SELF-MANAGED-SERVER-ADMIN-PWD> with the administrator password on the self-managed Couchbase Server
    • Replace <URL-TO-SELF-MANAGED-COUCHBASE-SERVER> with the url to your self-managed Couchbase Server
    • Replace <SELF-MANAGED-BUCKET-NAME> with the name of the bucket you wish to migrate
      • NOTE: This command backs up only the specified bucket

Run the following command to examine the backup:

  • ls /backups

For Windows use:

  • dir /backups

Basic information about your backup will be displayed

Use the cbrestore CLI tool:

  • cbrestore is located in the root directory of your self-managed Couchbase Server installation.
  • You use a command line terminal to work with cbrestore.
  • Depending on the platform, cbrestore is installed with Couchbase Server in the following location:
    • Linux/opt/couchbase/bin/cbrestore
    • Windows (assuming default installation): C:\Program Files\Couchbase\Server\bin\cbrestore
    • Mac OS X/Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/cbrestore

Restore the backup to Capella using cbrestore

Run the following cbrestore command to restore the backup on Capella:

Note: make the following substitutions in the command:

    • Replace <CAPELLA-NODE-HOSTNAME> with the node hostname from your Capella cluster (recorded earlier)
    • Replace <SELF-MANAGED-BUCKET-NAME> with the name of the bucket you with to migrate
    • Replace <CAPELLA-BUCKET-NAME> with the name of the target bucket on Capella
    • Replace <DATE-OF-BACKUP> with the date that you took the backup
      • NOTE: use the date format YYYY-MM-DD
    • Replace <DATE-FOLLOWING-BACKUP> with the date following the date that you took the backup
      • NOTE: use the date format YYYY-MM-DD
    • Replace <CAPELLA-DATABASE-CREDENTIALS-USER> with the database credentials user on the Capella cluster
    • Replace <CAPELLA-DATABASE-CREDENTIALS-PASSWORD> with the database credentials user password
    • Replace <FULL-PATH-TO-CAPELLA-ROOT-CERT> with the full file path to the Capella cluster root certificate (as saved in earlier step).

Here is an example of what the command should look like:

The command may take a few mins to run depending on the size of your cluster. Once completed you should see details about the restored backup displayed in the terminal, and a message reading Done.

Examine the migrated documents and indexes:

    • In the Couchbase Capella UI, choose “Tools>Documents” for your target cluster.
    • You should see that all of your self-managed cluster documents have been migrated to Capella.
    • In the Couchbase Capella UI, choose “Tools>Indexes” for your target cluster.
    • You should see that all of your self-managed cluster indexes have been migrated to Capella.
    • NOTE: Any indexes with definitions set as defer_build:true will be migrated, but will be listed as CREATED but not READY. These indexes still need to be built on Capella.

Build indexes on Capella:

    • In the Couchbase Capella UI, choose “Tools>Query Workbench” in your target cluster.
    • The following query will create BUILD statements for any indexes that were created but not built during migration. 
    • Copy/paste the following query to the Query Workbench in Capella, then click “Execute”:

The query results will return a BUILD statement to build the created indexes. The results will look something like so:

Copy the BUILD statement (between the double quotes), then paste to the Query Editor and click Execute to build the indexes.

Congratulations, you have just migrated data and indexes from your self-managed Couchbase Server cluster to Capella!

 

Ongoing migration via Cross Data Center Replication (XDCR)

This option applies to the following versions of self-managed Couchbase Server:

  • Enterprise Edition version 6.6, 7.0.x, 7.1

This is an optional step. Consider using XDCR if you wish to continue migrating data for a period of time beyond the initial migration using the CLI tools.

 

Connect self-managed Couchbase Server cluster to Couchbase Capella cluster. (Enterprise Edition only):

    • In the Couchbase Capella UI main navigation click Clusters, then click on the target cluster for migration.
    • Click the “Connect” tab, then under “Wide Area Network” click “Manage Allowed IP.”
    • On the Allowed IP screen, click “Add Allowed IP,”
    • On the flyout editor enter the IP address of your self-managed Couchbase Server cluster nodes (or click “+ Add My IP” to automatically paste in your current IP address).
    • Click “Add IP.”
    • Repeat for each cluster node IP address.
    • Click “< BACK” to go to the previous screen.

Note: If whitelisting the IP fails, in most cases it is related to firewall issues. See Couchbase documentation for information on the default ports used by Couchbase Server: https://docs.couchbase.com/server/current/install/install-ports.html

    • Under “Wide Area Network,” copy and save the displayed URL, this is the endpoint connection for your Capella cluster. You will use this URL later to connect your self-managed Couchbase Server cluster to Couchbase Capella.
    • Under “Root Certificate,” click “Copy” (this will capture the certificate to your clipboard for pasting).
    • Log in to your self-managed Couchbase Server Web Console, and in the main navigation click “XDCR.” Click “Add Remote.”

Enter the following settings:

    • Cluster Name – enter a name for the remote cluster connection
    • IP/Hostname – enter the Couchase Capella cluster endpoint URL you copied earlier
    • Username for Remote Cluster – enter the database credentials username for your Capella target cluster
    • Password – enter the password for the database credentials user for your Capella target cluster
    • Enable Secure Connection – Checked
    • Select the “Full” radio button
    • Paste the Capella cluster root certificate you copied earlier
    • Click “Save”

Set up Couchbase XDCR. (Enterprise Edition only):

Couchbase cross data center replication (XDCR) allows data to be replicated across clusters that are located in different data centers. It can be used to migrate data into Couchbase Capella from self-managed Couchbase Server clusters. NOTE: XDCR does not migrate indexes.

In your self-managed Couchbase Server Web Console click “XDCR” in the main navigation, then click “Add Replication.” 

Enter the following settings:

    • Replicate From Bucket – select the source bucket for migration
    • Remote Bucket – enter the remote bucket namett
    • Remote Cluster – select the remote cluster you created earlier
    • Click “Save replication”

The replication process should begin within a few seconds.

Verify the migration

Verify successful data migration:

    • In the Couchbase Capella UI, choose “Clusters” in the main navigation and then click the target cluster in your cluster list. 
    • Choose the “Buckets” tab for your target cluster. Verify that the number of “Items” (documents) in the target bucket match the number of items in the source bucket.
    • In the target cluster, choose “Documents” in the “Tools” dropdown list. Verify that all documents were migrated.

Verify successful index migration:

In the Couchbase Capella UI, choose “Indexes” in the “Tools” dropdown list for your target cluster. Verify that the indexes are migrated and built.

Verify query runtime:

In the Couchbase Capella UI, choose “Query Workbench” in the “Tools” dropdown list for your target cluster. 

Run a sample SQL++ (aka “N1QL”) query or a query used in your application to test that you receive the same results as the query in your self-managed Couchbase Server cluster.

Related resources

Preparing to migrate:

Deploy and configure resources on Couchbase Capella

 

Author

Posted by Couchbase Product Marketing

Leave a reply