With 6.6, Couchbase is introducing the ability to backup directly to AWS S3 or any other S3-compatible Object Store. Using the our backup utility, cbbackupmgr, you can now specify an S3 bucket as a destination for backing up your data.
Back up to cloud Object Stores (S3)
Object Stores with S3 compatible interface are becoming the defacto standard for backing up large volumes of data. It is often adopted for varied deployment types from cloud-native deployments to hybrid cloud deployments to private cloud deployments.
Object Storage is a flat storage system, where data is stored as objects instead of a hierarchy of files in a file system or as blocks in a block storage system. Objects consist of data itself, a variable amount of customizable metadata and a unique identifier which is used by the applications to access the object instead of a file name and a file path. These unique identifiers are arranged in a flat address space, which removes the complexity and scalability challenges of a hierarchical file system based on complex file paths. In practice, applications directly manage all of the objects, eliminating the need for a traditional file system. They are accessed using RESTful APIs, which query the object’s metadata.
The biggest advantage of object store is the RESTful access which provides remote accessibility and scalability as it’s based on HTTP/HTTPS. The customizability of the metadata which can be used to query, filter the data, run analytics etc., and the cost are other key factors. It also removes the complexity of having to mount the filesystem for containerized workloads. The performance of the object store can vary widely depending on underlying storage. There are cheap and deep object storages like AWS Glacier, high performance object stores like Nutanix Objects and something in between like AWS S3.
AWS S3 protocol has become the industry standard for object stores. Apart from S3 itself, plenty of other players like Scality, Dell’s ECS, NetApps’ Storage Grid, Nutanix’s Objects and more provide the object storage capability.
It is often used for disaster recovery, storage and retrieving very large amounts of data like images and media files in big data applications. Since S3 and especially Glacier is extremely cheap customers also prefer to use it as a cheap storage solution for archiving. As the containerized workloads are increasing, adoption for those use cases to alleviate the pain point of mounting files are also catching up.
Now let’s look at how quickly you can backup your data to S3 using cbbackupmgr.
Pre-requisites – The server and the cbbackupmgr tool should be at least 6.6.
Firstly, create an S3 bucket where you want to store backup files
The below prompt appears once the bucket is created
Now, configure an archive in the bucket where you want to create the backup repositories.
To create this, provide the AWS credentials, like region and access keys which you can obtain from permissions tab
Once the archive is configured, now start backing up the files using cbbackupmgr backup command as shown below
Below, you can see how all the files from the source cluster is now backed up in S3 bucket.
Contents in the source cluster
Data backed up to the S3 object store
Following the above mentioned steps, you can backup your files directly to AWS or any other S3 compatible stores seamlessly.