Aircraft Accident Recorder

Aircraft flight recorders help in investigating an aircraft accident or incident. Similarly, for databases, if a server suddenly fails abnormally or if you are just troubleshooting the system, error logs can really be handy. Error logs are a gold mine of post-event information because they record all relevant events that happen in the system.

Couchbase Server creates a number of different log files depending on the components of the system that produce the error, and the level and the problem being reported. These individual log files can help you to narrow down to the root cause of the problem much faster. In this blog, we will talk about Couchbase Server logs and how you can use these logs to get more insight into your Couchbase cluster.  Based on your platform (Linux, Windows or Mac OSX), the location of  log files can vary.

Couchbase Server has different log files for different components of the database. For each node, each log file location, contains a list of different log files.

Individual log files are automatically numbered, with a numerical suffix for each new log. There are a maximum of 20 log files per log and each log file has a maximum file size of 10 MB by default. When the maximum number of log files is reached, the oldest log file is overwritten by new file content.

Now, let’s look at two example scenarios where the Couchbase Server logs can be useful –

1. When your javascript map reduce function does not emit any results

When working with views, if you define a view with a field that is not defined, you will not see any results. This may happen particularly in the case when you emit an attribute but the parent does not exist. For example: emit(doc.foo.bar, null) when your documents to not include the attribute “foo”. Here’s a simpler example.

function (doc, meta) {
emit(doc1.foo, null);
}

In figure 1 below, the javascript field “doc1” does not exist. The view can be saved but you will not find any results.

Figure 1 : Example map function in javascript with undefined object doc1 

But to diagnose the problem you can you look at the mapreduce_errors.1 log file. You will notice that the reference error is logged.

Figure 2: Undefined reference error shown in mapreduce_errors file

Currently, if a document gets skipped because of a runtime error as seen above when building a view, it will get skipped from all views in that design document. This is another scenario, where you may want to take a look at the log file closely.

In general, one best practise you should follow when writing views, is testing whether the parent field being dereferenced exists. In our example, we can rewrite the map function as below :

function (doc, meta) {  

 if(doc.foo) 

 { 

   emit(doc1.foo, null);

 }

}

2. XDCR errors when the destination bucket is suddenly deleted 

Cross datacenter replication in Couchbase Server 2.0 supports unidirectional and bidirectional  active-active replication between two Couchbase Server clusters. Let’s assume unidirectional cross datacenter replication is setup from the beer-sample bucket on the source cluster to the testBucket bucket on the destination cluster.

If the testBucket bucket on the destination cluster is destination cluster is suddenly deleted when XDCR is in progress, XDCR cannot continue and an error is logged in the xdcr_error log file on the source cluster node as shown in Figure 4 below. The couchbase server admin console (shown in Figure 5) also shows the last 10 XDCR related errors.

 

Figure 3 : Bucket not found in xdcr_error file

Figure 4 : Last 10 errors shown in Couchbase admin console UI

Found a bug?

Being an open source product, we would love your feedback as you play around with Couchbase Server. If you hit unexpected behavior, log a bug.

Luckily we do have tools that zip up all the needed logs that our dev team needs to investigate. You’ll find the cbcollect_info tool in /opt/couchbase/bin on Linux and /Applications/Couchbase

Server.app/Contents/Resources/couchbase-core/bin on MacOS. This tool collects all the logs for a given Couchbase server node and zips it up into a file.

Don’t forget to attach this to your bug report.

Happy hacking !

Author

Posted by The Couchbase Team

Jennifer Garcia is a Senior Web Manager at Couchbase Inc. As the website manager, Jennifer has overall responsibility for the website properties including design, implementation, content, and performance.

3 Comments

  1. correct example in the first section should be

    function (doc, meta) {
    if(doc.foo)
    {
    emit(doc.foo.bar, null);
    }
    }

  2. The log location link has changed to http://docs.couchbase.com/couc

  3. […] Couchbase is a distributed, high performance, Cache and NoSQL database in a single cluster architecture. Despite a similar name and shared heritage, Couchbase is a very different product than CouchDB or any other NoSQL offering. Being able to monitor and profile Couchbase performance alongside application metrics is critical. Over time, monitoring is the element for a successful deployment of any mission critical system. This is true in general and even more important in distributed computing environments in particular. It’s the only way to ensure long-term success. For this discussion, we want to focus on monitoring but it’s important to note we do provide verbose Couchbase logging to facilitate application troubleshooting. These logs are stored in ‘/opt/couchbase/var/lib/couchbase/logs’. For more insight into these logs please review the Couchbase flight recorder (https://www.couchbase.com/couchbase-server-recorder). […]

Leave a reply