Authentication and authorization to the query service in couchbase works in multiple ways – 

  1. Passing credentials through a rest request – curl http://localhost:8093/query/service?pretty=true -d “statement=select * from system:keyspaces” -u Admin:pwd
  2. Passing credentials using the creds named parameter and/or query parameter – curl http://localhost:8093/query/service?pretty=true -d “statement=select * from system:keyspaces&creds=[{user:“Administrator”,”password”:”pass”}]”
  3. Using basic auth in the request 
  4. Request from cbq (similar to 1,2) using the -u -p -creds options and \SET command. 
  5. X.509 Certificates for TLS
  6. Node to Node encryption

With the addition of RBAC, the creds query parameter was made redundant but is still supported for backward compatibility.

The goal of adding X509 Certificate support is to enhance client-server encryption using a certificate that is trusted by the certificate authority.

X.509 certificates enable server authentication and encryption for client-server communications. Couchbase supports both server and client authentication using X509 certificates and you have to be a full Admin or Security Admin to manage certificates. This article talks about server-side X.509 certificate support for authorization in Couchbase. Clients can also verify the identity of Couchbase Server but that will be discussed in another article. 

The most common scenarios for which X509 certificates are used are when clients have to go through the internet, when transferring sensitive data on the wire between application and Couchbase Server, or between data centers (XDCR) or when mandated by compliance regulations. 

What is a X.509 Certificate ?

It is a public key certificate that is used to distribute a public key, signed by a trusted certificate authority verifying the identity of the server. Using it a client is assured that the request is not being sent to an unknown server. These certificates are signed by a third party also known as a Certificate authority. CAs are entities that issue digital certificates. A CA is actually made up of a series of CAs called a CA hierarchy. This CA hierarchy constructs a chain of trust that all node or end entity certificates rely on. The chain does not contain the root CA public key. 

In a hierarchical public key infrastructure (PKI) there are typically 3 kinds of hierarchy. One-tier, Two-tier and N-tier. The CA at the top of this hierarchy is known as the Root-CA. All subsequent CA’s are the Intermediate CA, and the nth (last) CA is known as the Node CA. 

An important point to note is that to trust a certificate used to establish a secure connection it must have been issued by a CA that is included in the trusted store of the device that is connecting.

One-tier / Single-tier CA authority

This consists of a single CA is the simplest form of CA hierarchy but is not usually used in production as a compromise of this root CA results in a compromise of the entire PKI. . Here the root CA is the issuing CA as well and all certificates immediately below the root certificate inherit its trustworthiness.

Two-tier CA authority 

This consists of a Root CA that has issued a certificate for one subordinate known as the Intermediate CA. The difference here is that the issued certificates are trusted as they come from a trusted authority via the intermediate CA. 

Root CA signs-> Intermediate CA signs -> Issuing CA / Cluster CA

N-Tier CA authority

 In most production deployments, a hierarchy has multiple CAs. The root CA issues certificates to the intermediate CAs, which in turn generate intermediate certificates: these are used to sign client certificates, such as a cluster certificate:

  • Trusted root CA > Intermediate CA > Cluster certificate
  • Trusted root CA > Intermediate CA 1 > Intermediate CA 2…. > Intermediate CA n > Cluster certificate

The Two-tier hierarchy is a subtype of the N Tier hierarchy. 

 In all the above cases the certificate chain needs to be verified up unto the root CA. The trust chain contains your certificate, concatenated with all intermediate certificates.

Note here – All intermediate certificates should be installed on your server: otherwise, some clients will assume that the connection is not secure. This results in ‘untrusted’ warnings. 

Setting up X.509 on a couchbase cluster

Some prerequisites before setting up the certificates – 

  • The certificate should be an RSA key certificate in a valid .pem format
  • The certificate should not be invalid – Current time should fall between valid from and valid to as set in the cert. 
  • Use an RSA key-length of 2048 bits or higher. (As computing capabilities increase, longer RSA keys provide increased security.)
  • With a single node cluster you will have 1 directory corresponding to the node certs. With a multi node cluster, create multiple dir corresponding to each node in the cluster – node1, node2 etc. 
  • If you have multiple Intermediate CAs, be sure to stack them in the correct order in the certificate chain. 
  • The private node key and the chain need to be put in ../var/lib/couchbase/inbox relative to your os bin dir from where the service/cluster exes are deployed. 

Naming conventions

ca.pem – Root CA’s public key 

int1.pem – Intermediate CA’s public key (no 1. If you have multiple intermediate CAs, name them appropriately to add to the chain in correct order. This shows which int CA is closest to the node)

node1.pem – Node 1 CA’s public key ( node2.pem – Node 2 CA’s public key and so on )

node1.key – Node 1 CA’s private key

chain.pem – Certificate chain containing the nodes public key and the intermediate public keys that signed the node public key. 

We use the openssl tool to create our certificates. See its documentation for more details on the commands itself. 

Steps to setup X.509 Certs

Step 1 – Create the root private key 

openssl genrsa -out ca.key 2048 2>/dev/null

Generate an RSA private key called ca.key (-out filename) that is the 2048 bits. 

When generating the key, the . or + symbol will be seen. This indicates progress in the key generation. The . represents each number that passes the test and + means a number has passed a single round of the Miller-Rabin primality test. When the newline is seen it means the key was successfully generated and the number (2 prime numbers) has passed all the prime tests. See the documentation of openssl-genrsa for more details. 

Step 2 – Generate the root public key used as the cluster CA

openssl req -new -x509  -days 365 -sha256 -key ca.key -out ca.pem -subj ‘/C=US/O=Couchbase/CN=Couchbase Root CA’ 2>/dev/null

Generate a new self signed root certificate  (-x509 option) request with a sha256 signature (-sha256. This means higher security) that is valid for 1 year (-days 365 in days). The public key ca.pem (-out) is derived from the private key using the -key option to specify the private key.

X.509 certificates have a Subject Distinguished Name (DN) field and can also have multiple names in the Subject Alternative Name extension. It is made of relative DNs.

CN = COmmon Name, O = Organization, C = Country Name

The certificate’s issuer is specified to have the CN (Common Name) of Couchbase Root CA: as this name indicates, the certificate will be the root certificate for the Couchbase 

Step 3 – Generate Intermediate private key (or keys if using an N tier hierarchy as described above )

openssl genrsa -out int1.key 2048 2>/dev/null

Step 4 – Generate the intermediate certificate signing request

openssl req -new -key int1.key -out int1.csr -subj ‘/C=US/O=Couchbase/CN=Couchbase Intermediate CA’ 2>/dev/null

A CSR or certificate signing request is a request sent from an applicant to a CA to apply for a certificate. You can customize: add to or limit the capabilities of the X.509 certificate using an extension file. This information will be used on all the nodes in the cluster. For example – 

cat > v3.ext <<EOF

basicConstraints = CA:FALSE

subjectKeyIdentifier = hash

authorityKeyIdentifier = keyid,issuer:always

EOF

For an extensive list of all the standard extensions see section 4.2 of RFC 5280 on X509 PKI and CRL profile. – https://tools.ietf.org/html/rfc5280 

Step 5 – Create the intermediate Certificate 

openssl x509 -req -in int1.csr -CA ca.pem -CAkey ca.key -CAcreateserial -CAserial rootCA.srl -extfile v3.ext -out int1.pem -days 365 2>/dev/null 

Read from the csr file and pass in the Root CA keys to establish the root certificate’s authority. The root CA encrypted key is used to sign the intermediate CSR.  Before we sign anything, a serial number file needs to be setup for the Root CA. This is so that each certificate can have a unique serial number. This is done using the -CAcreateserial -CAserial <name of file> options. The rootCA.srl is the serial number file. It is a simple text file with ASCII numbers in it. The certificate is customized using the extensions we defined earlier and is valid for a year. When prompts appear, asking for a passphrase for the certificate. Enter an appropriate phrase in response to the prompts. Remember this phrase. 

Finally we have the root and intermediate CA certificates. Now it’s time to set up the node certificate and sign it with the root CA and intermediate key. 

Step 6 – Setup node key and CSR 

openssl genrsa -out node1.key 2048 2>/dev/null

Once the encrypted key for the node is generated, setup the node csr. 

openssl req -new -key node1.key -out node1.csr -subj “/C=US/O=Couchbase/CN=server1_linux” 2>/dev/null

Here the Common name defined in the certificate subject is the node name as defined and mapped in /etc/hosts. 

When setting up the node csr use the nodename(preferable), IP address or URI with a SAN (subject alternative name) certificate.

The usual way to specify the certificate identity is through the Common Name (CN) in the subject DN of the certificate. If you deploy a certificate on a multi-homed host, it is necessary to define a certificate with alternative identities using the subjectAltName certificate-extension.

“subjectAltName = IP:172.23.99.49”

Step 7 – Generate the node certificate using the appropriate extensions

openssl x509 -req -in node1.csr -CA int1.pem -CAkey int1.key -CAcreateserial \

-CAserial intermediateCA.srl -out node1.pem -days 365 

This is similar to the steps above for generating intermediate certificate. 

Step 8Generate the certificate chain 

cat node1.pem int1.pem > chain.pem 

Concatenate all the intermediate and node certificates in the correct order. The root certificate is never included in the chain. This chain will allow the client to verify the intermediate certificate against the root certificate.  

Step 9 – Deploy the certificates 

  1. Copy the node encrypted certificate and chain certificate into the inbox folder within ../var/lib/couchbase/inbox relative to where your binaries are run from for your OS and give them the appropriate permissions using chmod a+x
    • node1.key and chain.pem are copied into ../inbox 
    • chmod a+x node1.key 
    • chmod a+x chain.pem
  2. Upload these to the server
    • curl -X POST –data-binary ca.pem http://Administrator:password@172.23.99.49:8091/controller/uploadClusterCA
    • curl -X POST http://Administrator:password@172.23.99.49:8091/node/controller/reloadCertificate

When you load the cluster certificate into Couchbase, it is first checked to make sure it is a valid X.509 certificate. Next, if the per-node certificates are not signed by the cluster certificate, a warning is shown for each node during configuration. As the per-node certificates are updated, such that they are signed by the cluster certificate, the warning for each node goes away.

Using certs in your N1QL CURL() query or the cbq shell 

In order to verify certificates that have been signed by a third party, the library uses the certificates stored on the local machine. For each query node in a Couchbase cluster, the ..//var/lib/couchbase/n1qlcerts folder contains certs needed by CURL(). If the certs are not found in this folder an error is thrown.  The cacert option is used to pass a certificate name to the function. Passing in a path will cause an error.

select CURL(“https://127.0.0.1:18091/pools”,{“request”:“GET”,”user”:”Bucketuser:password”,“cacert”:“ca.pem”})

 For connecting using the shell use the cacert, cert and the key options.

./cbq –cacert ca.pem –cert chain.pem –key node1.key –engine https://172.23.99.49:18091

With this article, we now know how to setup X509 certificates on our server and use them with the N1QL query and CBQ shell. With the next article we will go in depth into Client side certificates. 

 

Posted by Isha Kandaswamy

Isha Kandaswamy is a Senior Software Engineer at Couchbase. Isha is responsible for the development of designing the different features and tools for the N1QL Query Language -SQL for Json. Also, Designing and implementing features and tools for the N1QL query language.

Leave a reply