Our Community Guest post comes from Joel Andrews. Joel is a polyglot developer living on the rainy west coast of Canada. He fills a variety of roles at Kashoo including backend developer, database admin, DevOps guy, product owner and occasionally web and Android frontend developer. He has a deep and abiding love for Couchbase (especially Couchbase Mobile) that makes him completely unbiased when discussing the pros and cons of any data storage solution.
When building any system with client and server components, it is critical to validate the data that the client tries to store on the server and to ensure that only those who are authorized can view or modify it. Otherwise a malicious client can take advantage of your system to do all sorts of nasty things. That’s why, when we at Kashoo began using Couchbase Mobile and Sync Gateway to synchronize data between multiple mobile devices and our web application, we built our sync functions to perform comprehensive validation of all document properties and the permissions (i.e. channels) that would be required to read, create, replace and delete them.
The flexibility of Sync Gateway’s sync function API can be both a blessing and a curse, though. We found that, when starting with a simple bucket that had only a few document types, it was not overly difficult to keep things organized and maintainable while still performing a good amount of validation. However, as we added more and more document types, our sync function’s size and complexity quickly ballooned out of control. Writing a sync function that performs comprehensive validation becomes increasingly unmanageable the more document types you add… that is, unless you start to apply some structure to the code.
This led us to break up our Sync Gateway database’s sync function into four primary phases:
-
Figure out the document’s type
-
Verify that the user belongs to one of the channels required to perform the operation
-
Validate each of the document properties
-
Assign channels to the document
This worked well until we introduced a second Sync Gateway database. Because sync functions are defined as a single block of code without the ability to dynamically load external libraries, we duplicated the rough framework that we had created for the first database’s sync function into the second database’s sync function. However, as developers, this duplication of code did not sit well with us because it meant that, as we made improvements to one sync function, we would either have to manually port those changes to the other or they would gradually diverge from each other in terms of functionality.
Thus, we decided to evolve the rough framework further into a standalone tool that can be shared between Sync Gateway databases without requiring code duplication. Because Sync Gateway’s sync function API does not allow the use of external libraries, we took the approach of code generation rather than dynamic linking: you write your document type definitions as JavaScript objects and then use the utility to import those definitions into a pre-generated template to create a cohesive and self-contained sync function.
Now we had a way to define our document types (or “schema”) in a very declarative way that was closer to writing configuration than writing code and where the document type definitions can be maintained completely independently of the code that actually performs authorization, validation and channel assignment. This eliminated much of the code duplication that was previously required. And it worked very well for us. It worked so well, in fact, that we decided to release and support the utility as open source software so that others would not have to duplicate our efforts. The utility is called synctos, and it is available for use now under the highly permissive MIT license.
Since its initial release we have also expanded its capabilities so that it now executes five phases rather than the original four and each phase is significantly more robust:
-
Figure out the document’s type
-
Authorize the user by determining whether
-
the user belongs to one of the channels required to perform the operation
-
the user belongs to one of the roles required to perform the operation
-
the user was explicitly granted permission to perform the operation
-
-
Validate each of the document properties
-
Assign access:
-
add users and/or roles to channels
-
add users to roles
-
-
Assign channels to the document
Here is an example document type definition to give you an idea what synctos is capable of:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 |
{ // A request/requisition for payment of an invoice paymentRequisition: { // Determine whether the document is of this type typeFilter: function(doc, oldDoc) { return doc.type === 'paymentRequisition' || oldDoc.type === 'paymentRequisition'; }, // The channels that the document will be assigned to. // In order for a user to perform each operation, it must belong to at least one of the listed channels for that operation type. channels: { add: [ 'create-payment-requisition' ], replace: [ 'replace-payment-requisition' ], remove: [ 'remove-payment-requisition' ] }, // Define what is expected of each of the properties that belong to this document type propertyValidators: { type: { // The document's type type: 'string', required: true, immutable: true }, businessId: { // The ID of the business with which the payment requisition is associated type: 'integer', required: true, immutable: true, minimumValue: 1 }, invoiceRecordId: { // The ID of the invoice with which the payment requisition is associated type: 'integer', required: true, immutable: true, minimumValue: 1 }, issuedAt: { // When the payment requisition was sent/issued type: 'datetime', immutable: true }, issuedByUserId: { // The ID of the user that issued the payment requisition type: 'string', immutable: true, mustNotBeEmpty: true }, invoiceRecipients: { // Who received the payment requisition type: 'string' } } } } |
It’s easy to get started with synctos using npm (the package manager for Node.js). Execute the following in an empty directory:
|
1 |
npm install synctos |
This will download the latest version of synctos to the local node_modules directory. And, now that synctos is downloaded, if you save the document definitions example above to a file called “example-doc-definitions.js”, you can use synctos to generate a sync function from it:
node_modules/synctos/make-sync-function ./example-doc-definitions.js ./example-generated-sync-function.js
That should have created a fully self-contained sync function by combining synctos’ pre-defined template with the document definitions you provided, as you’ll find if you open “example-generated-sync-function.js” in a text editor. The generated sync function can then be inserted verbatim into a Sync Gateway configuration file.
Following is a simple Sync Gateway example configuration that uses an in-memory instance of the Walrus database engine so that you can try the sync function without having to set up a Couchbase Server bucket. Replace the string “%GENERATED_SYNC_FUNCTION%” with the sync function that was generated in the previous step. Note that the backticks/backquotes (`) that surround the sync function are a custom addition to the JSON format that allows for multiline strings in a Sync Gateway configuration file.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
{ "databases": { "synctos-test": { "server": "walrus:", "users": { "my-user": { "admin_channels": [ "create-payment-requisition", "replace-payment-requisition" ], "password": "password" } }, "sync": `%GENERATED_SYNC_FUNCTION%` } } } |
Save the Sync Gateway configuration above as “example-sync-gateway-config.json”, and then launch Sync Gateway with the new configuration:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
/path/to/sync_gateway ./example-sync-gateway-config.json Now use a tool like curl to create a new instance of the "paymentRequisition" document type. For example: curl -X PUT -u "my-user:password" -H "Content-Type: application/json" -d '{ "type": "paymentRequisition", "businessId": 15, "invoiceRecordId": 79, "issuedAt": "2017-01-11T18:21:18-0800", "issuedByUserId": "my-user", "invoiceRecipients": "foo@example.com,bar@example.com" }' "http://localhost:4984/synctos-test/myDoc1" The response will indicate success: {"id":"myDoc1","ok":true,"rev":"1-ed7b6831f4339188eee412082d44ca32"} |
Creating a valid document is great, but what happens if we try to replace that document with one that violates the document definition? Note that, in the following example, the “businessId” property value has been modified from its original value of 15; this is a problem because the property was marked as “immutable” in the document definition.
|
1 2 3 4 5 6 7 8 |
curl -X PUT -u "my-user:password" -H "Content-Type: application/json" -d '{ "type": "paymentRequisition", "businessId": 99999, "invoiceRecordId": 79, "issuedAt": "2017-01-11T18:21:18-0800", "issuedByUserId": "my-user", "invoiceRecipients": "new-email@example.com" }' "http://localhost:4984/synctos-test/myDoc1?rev=1-ed7b6831f4339188eee412082d44ca32" |
The response will be an error:
|
1 |
{"error":"Forbidden","reason":"Invalid paymentRequisition document: value of item "businessId" may not be modified"} |
As you can see, synctos generates detailed error messages that make it easy to determine what went wrong when trying to write a document revision. This is a simple example with only one validation error, but if the document contained multiple errors, each additional error would be enumerated in the response’s error message.
And what if we try to perform an operation that requires a channel that our user does not belong to? Remember that the user we defined in the Sync Gateway configuration does not belong to the “remove-payment-requisition” channel, so it should not be allowed to remove the document we created earlier.
|
1 2 3 |
curl -X DELETE -u "my-user:password" "http://localhost:4984/synctos-test/myDoc1?rev=1-ed7b6831f4339188eee412082d44ca32" And indeed the response indicates that the user does not belong to the necessary channel: {"error":"Forbidden","reason":"missing channel access"} |
There is much more to synctos (for example, assigning access to users and roles, executing custom actions after each phase of the sync function, and validating generated sync functions using the built in test module), but hopefully this gave you a taste of what it’s capable of and how it can help you to write Sync Gateway sync functions that comprehensively validate your documents. See the project’s documentation on GitHub or npm for plenty more details.
