With 6.6, Couchbase is introducing the ability to backup directly to AWS S3 or any other S3-compatible Object Store. Using the our backup utility, cbbackupmgr, you can now specify an S3 bucket as a destination for backing up your data. 

Back up to cloud Object Stores (S3)

Object Stores with S3 compatible interface are becoming the defacto standard for backing up large volumes of data. It is often adopted for varied deployment types from cloud-native deployments to hybrid cloud deployments to private cloud deployments. 

Object Storage is a flat storage system, where data is stored as objects instead of a hierarchy of files in a file system or as blocks in a block storage system. Objects consist of data itself, a variable amount of customizable metadata and a unique identifier which is used by the applications to access the object instead of a file name and a file path. These unique identifiers are arranged in a flat address space, which removes the complexity and scalability challenges of a hierarchical file system based on complex file paths. In practice, applications directly manage all of the objects, eliminating the need for a traditional file system. They are accessed using RESTful APIs, which query the object’s metadata.

The biggest advantage of object store is the RESTful access which provides remote accessibility and scalability as it’s based on HTTP/HTTPS. The customizability of the metadata which can be used to query, filter the data, run analytics etc., and the cost are other key factors. It also removes the complexity of having to mount the filesystem for containerized workloads. The performance of the object store can vary widely depending on underlying storage. There are cheap and deep object storages like AWS Glacier, high performance object stores like Nutanix Objects and something in between like AWS S3.

AWS S3 protocol has become the industry standard for object stores. Apart from S3 itself, plenty of other players like Scality, Dell’s ECS, NetApps’ Storage Grid, Nutanix’s Objects and more provide the object storage capability.  

It is often used for disaster recovery, storage and retrieving very large amounts of data like images and media files in big data applications. Since S3 and especially Glacier is extremely cheap customers also prefer to use it as a cheap storage solution for archiving. As the containerized workloads are increasing, adoption for those use cases to alleviate the pain point of mounting files are also catching up.

Now let’s look at how quickly you can backup your data to S3 using cbbackupmgr.

Pre-requisites – The server and the cbbackupmgr tool should be at least 6.6.

Firstly, create an S3 bucket where you want to store backup files

The below prompt appears once the bucket is created

Now, configure an archive in the bucket where you want to create the backup repositories.

To create this, provide the AWS credentials, like region and access keys which you can obtain from permissions tab

Once the archive is configured, now start backing up the files using cbbackupmgr backup command as shown below

Below, you can see how all the files from the source cluster is now backed up in S3 bucket.

Contents in the source cluster

Data backed up to the S3 object store

Following the above mentioned steps, you can backup your files directly to AWS or any other S3 compatible stores seamlessly.

Explore Couchbase Server 6.6 resources

BlogsDocs and TutorialsWebpages and Webinars
What’s New in Couchbase Server 6.6What’s New in Couchbase Server 6.6?New Features in Couchbase Server 6.6: Analytics, Backup, Query, and More
Eventing Improvements (Timers, Handlers, and Statistics)Couchbase Server 6.6 Release NotesCouchbase Analytics Service
Remote Links – Analyze Your Enterprise With Couchbase AnalyticsTry the Couchbase Index Advisor ServiceWhat’s New in Couchbase Server (Product Page)
External Datasets – Extend Your Reach With Couchbase AnalyticsSet Up Analytics Remote and S3 Links Using REST APICompare Editions
Announcing Flex Index With CouchbaseCreate External Datasets Using Data Definition Language (DDL)
Introducing Backing Up to Object Store (S3)Set Up Analytics Remote and S3 Links Using CLI
Import Documents With the Web Admin Console

Posted by Chaitra Ramarao, Sr. Product Manager, Couchbase Inc.

Chaitra Ramarao is a Senior Product Manager at Couchbase, NoSQL database company, leading databases tooling, cross datacenter replication and partner integrations. Her prior gigs include data analytics product management for Kaiser Permanente and software development for Hewlett Packard. She has a Bachelors degree in ECE and a Masters from Carnegie Mellon in Engineering & Technology Innovation Management.

Leave a reply