Aaron Benton is an experienced architect who specializes in creative solutions to develop innovative mobile applications. He has over 10 years experience in full stack development, including ColdFusion, SQL, NoSQL, JavaScript, HTML, and CSS. Aaron is currently an Applications Architect for Shop.com in Greensboro, North Carolina and is a Couchbase Community Champion.

Aaron Benton

FakeIt Series 1 of 5: Generating Fake Data

There are countless blog posts on data modeling, key and document patterns. All of these posts give a great introduction into how to structure and model your documents in Couchbase, but none of them tell you what to do next. In this blog series we are going to answer the question, what do you after you’ve defined your data model?

Users Model

For this series we will be working with a greenfield e-commerce application. As with most e-commerce applications, our application is going to have users so this is where we will begin.

We have defined a basic user model to start with.

We’ve done the hardest part, which is defining our model, but now what?

  • How do we represent this model?
  • How do we document this model?
  • Does this model rely on data from other models?
  • How can data be generated from this model?
  • How can we generate fake / test data?

Luckily for us there is a NodeJS project called FakeIt that can answer all of these questions for us. FakeIt is a command-line utility that generates fake data in json, yaml, yml, cson, or csv formats based on models which are defined in yaml. Data can be generated using any combination of FakerJS, ChanceJS, or Custom Functions. The generated data can be output in the following formats and destinations:

  • json
  • yaml
  • cson
  • csv
  • Zip Archive of json, yaml, cson or csv files
  • Couchbase Server
  • Couchbase Sync Gateway Server

We can define a FakeIt model in YAML to represent our JSON model. This provides us a documented and data-typed model that we can communicate how our model should be structure and what the properties are for.

You’re probably saying to yourself, great, I’ve defined my model in YAML but what good does this do me?One of the biggest issues developers face when beginning development is having data to work with. Often times an exorbitant amount of time is spent manually creating documents, writing throw away code to populate a bucket. Additionally you may have a full or partial data dump of your database that has to be imported.

These are time consuming, tedious and in the case of a data dump do not provide any insight or documentation into the available models. We can add a few simple properties to our FakeIt model describing how our model should be generated, and through a single file we can create an endless amount of fake randomized documents.

We have added a data property to each of our models properties describing how that value should be generated. FakeIt supports 5 different ways to generate a value:

  • pre_build: function to initialize the value
  • build: function that builds a value
  • fake: A FakerJS template string i.e. {{internet.userName}}
  • value: A static value to use
  • post_build: a function that runs after every property in the model has been set

These build functions are a JavaScript function body. Each of these functions is passed the following variables that can be used at the time of its execution:

  • documents – An object containing a key for each model whose value is an array of each document that has been generated
  • globals – An object containing any global variables that may have been set by any of the run or build functions
  • inputs – An object containing a key for each input file used whose value is the deserialized version of the files data
  • faker – A reference to FakerJS
  • chance – A reference to ChanceJS
  • document_index – This is a number that represents the currently generated document’s position in the run order
  • require – This is the node require function, it allows you to require your own packages. For better performance require and seth them in the pre_run function.

For example, if we look at the username properties build function it would look like this:

Now that we have defined how our model should be generated, we can start to generate some fake data with it.

With our users model saved to a file models/users.yaml, we can output data directly to the console using the command

blog1 example1

Using this same model we can generate 100 JSON files and save them into a directory named output/ using the command

blog1 example2

Additionally, we can create a zip archive of 1,000 JSON files using the command:


blog1 example3

We can even generate a single CSV file of our model using the following command:


This will create a single CSV file whose name is the name of the model, in this case name: Users with the resulting file being named Users.csv

blog1 example4

Whether you are using JSON files, Zip Archives or CSV files all of these can be imported into Couchbase Server by using the CLI tools cbdocloader (for *.json and *.zip files) or cbimport (for *.json and *.csv files)

While generating static files is beneficial, there is still the extra step of having to import them into Couchbase Server through the available CLI tools. FakeIt also supports Couchbase Server and Sync Gateway as output destinations. We can generate 10,000 JSON documents from our users.yaml model, and output them to a bucket named ecommerce on a Couchbase Server running locally using the command:

blog1 example5

Conclusion

We’ve seen how we can represent a user’s JSON model using YAML to document and describe how a properties value should be generated. That single users.yaml file can be output to the console, JSON files, Zip archive of JSON files, CSV files, and even directly into Couchbase. FakeIt is a fantastic tool to speed up your development and generate larger development datasets. You can save your FakeIt models as part of your codebase for easy repeatable datasets by any developer.

FakeIt is a tool to ease development and testing of your Couchbase deployment. While it can generate large amounts of data, it is not a true load testing tool. There are CLI tools available for load testing and sizing such as cbc-pillowfight and cbworkloadgen

Up Next

  • FakeIt Series 2 of 5: Shared Data and Dependencies
  • FakeIt Series 3 of 5: Lean Models through Definitions
  • FakeIt Series 4 of 5: Working with Existing Data
  • FakeIt Series 5 of 5: Rapid Mobile Development w/ Sync-Gateway

Couchbase Champion

This post is part of the Couchbase Community Writing Program

Posted by Laura Czajkowski, Developer Community Manager, Couchbase

Laura Czajkowski is the Snr. Developer Community Manager at Couchbase overseeing the community, our incentive programs, Experts and Champions group, meetups, and defining our presence at developer events. She’s also responsible for our monthly developer newsletter and engaging with our community in various forms. Laura has been active in Open Source communities since 2000 and has been involved in various activities, including leading and organising conferences on software testing, documentation, and advocacy. Laura is an Open Source advocate and regular conference speaker who is passionate about getting people – everyone from primary school students to technology professionals – involved in Open Source communities both on IRC and in face-to-face discussions, she is easily found online at @czajkowski on twitter and on freenode.

4 Comments

  1. […] FakeIt Series 1 of 5: Generating Fake Data (Laura Czajkowski) […]

  2. […] FakeIt Series 1 of 5: Generating Fake Data we learned that FakeIt can generate a large amount of random data based off a single YAML file and […]

  3. […] far in our FakeIt series we’ve seen how we can Generate Fake Data, Share Data and Dependencies, and use Definitions for smaller models. Today we are going to look […]

Leave a reply