Using N1QL with Couchbase Eventing Functions.

Now, this is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning. — Winston Churchill

Updating data is usually not the end, but usually a progress of a workflow. Shipping follows ordering; inventory update follows shipping; adjusting credit follows returning;  The next step in the process is required to act to keep the workflow going. The workflows can be simple with few steps or complex with hundreds of steps.  Business Process Management (BPM) is an industry by itself.

Couchbase 5.5 introduced Evening service. Developers can write a Javascript function to execute upon a change to the data. We refer inserts, Updates, Merges, and Deletions together as mutations. Multiple specific use cases have been documented for developing these eventing functions.

JSON data model in Couchbase came from JavaScript.  N1QL is SQL for JSON. The eventing functions are written in Javascript and has integrated N1QL. Using the Eventing functions, writing procedural business logic with instant access to data easy.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

// Simple counting function // After every update to travel sample, keep track of the total number of documents of each type. function OnUpdate(doc, meta) {     try {        // Issue SELECT statement to get the counts.     var ginfo = SELECT type, COUNT(type) typecount FROM `travel-sample` WHERE `type` IS NOT MISSING GROUP BY type;       // loop through the resultset     for (var val of ginfo) {       var ckey = "trcount" + val.type;  // Create the document key-string.       var vtype = val.type; var vtc = val.typecount;  // get the type, count       // do the update.       UPDATE T1 USE KEYS[$ckey] SET  type =  $vtype, typecount =$vtc;     }   } catch(e) { log(e); }     }

Here’s the lifecycle of the functions from a developer perspective. For every mutation, you can define any number of these functions to be executed. It’s the developer responsibility to size the execution times of the functions, depending on the number of mutations.

Here’s the lifecycle of the Eventing functions:

For every insert, update or delete (direct or expiry), you can execute one or more javascript functions.  These functions can read the new data and type of action and then execute the subsequent action. This functionality is well described in Couchbase blogs, and articles.

Statement Type	Eventing Function Invoked
SELECT	None
INSERT	OnUpdate().  The function is invoked once per document inserted.Simple insert inserts a single document. Inserts can have multiple documents using multiple documents in the VALUES clause or can insert multiple documents via INSERT INTO…SELECT statement.
UPDATE	OnUpdate() is invoked once per document updated, except when multiple updates on the same document is deduped into a single update. Update statement can update multiple documents.
UPSERT	OnUpdate().  The behavior is similar to INSERT.
DELETE	OnDelete(). Invoked once per documented deleted.
MERGE	OnUpdate() and/or OnDelete() depending on the insert, update, and delete actions.
CREATE INDEX, DROP INDEX, EXPLAIN, PREPARE, GRANT, REVOKE	No eventing function is invoked.
EXECUTE	Depends on the type of the statement executed.

These functions can also execute N1QL statements.  Rest of the article looks at all aspects of N1QL executed in Eventing Functions.

N1QL Statements in Eventing functions.

Statement	Use cases for N1QL statements in Eventing Functions
CREATE INDEX	Since the schema is flexible, you could potentially inspect the data often/periodically to detect new fields and then create indexes on it.
DELETE	Cascade deletes.
DROP INDEX	A corollary to the CREATE INDEX use case.
INFER	Periodic introspection of the bucket for the structure.  Then take action (validate, create index, update the data model) if necessary.
INSERT	Maintaining referential data (eventually). Updating other documents with references to this data.  E.g Data from a new zip, state, etc. Data copy (fully or partially) to secondary/tertiary documents. Similar to post-trigger action.
MERGE	Keep the secondary data in sync.
SELECT	Fetch any data, run any reports periodically, etc. Check for various things like data quality, data validity, When you do know the target document key, use the built-in direct document references. See examples at: https://docs.couchbase.com/server/5.5/eventing/eventing-examples.html
UPSERT	Keeping secondary/tertiary data in sync. Similar to post-trigger action.
UPDATE	Keeping secondary/tertiary data in sync. Similar to post-trigger action.

Examples:  Let’s try some eventing functions with N1QL.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46

select meta().id, * from execute S1;   [   {     "S1": {       "name": "Joe Smith",       "zip": "94501"     },     "id": "Joe::94040"   },   {     "S1": {       "name": "John Smith",       "zip": "94040"     },     "id": "John::94040"   } ]       select meta().id, * from S2;     [   {     "S2": {       "name": "Joe Smith",       "zip": "94501"     },     "id": "Joe::94040"   },   {     "S2": {       "name": "John Smith",       "zip": "94040"     },     "id": "John::94040"   } ]     select meta().id, * from T1; <No data>

1. Simple N1QL in Functions for logging and cascade delete.

Save away every deleted document in a separate bucket.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54

function OnUpdate(doc, meta) { } function OnDelete(meta) {   try {       var myid = meta.id;       //Insert into the bucket of       INSERT INTO T1 VALUES(UUID(), {"type" : "deleted", "docid":$myid});       //Cascade delete       DELETE FROM S2 USE KEYS [$myid];   } catch(e) { log(e); } }   After creating the function above, the application Runs the following:   DELETE FROM S1 WHERE zip = "94040"; // One document was deleted.   SELECT meta().id, * from S1; // one document just got deleted. There should be only one document remaining. [   {     "S1": {       "name": "Joe Smith",       "zip": "94501"     },     "id": "Joe::94040"   } ]       SELECT meta().id, * from T1; // We should see a log of the deleted [   {     "T1": {       "docid": "John::94040",       "type": "deleted"     },     "id": "2dc9b33d-3cd4-422e-af9c-b0c664c4660f"   } ]   SELECT meta().id, * FROM S2; // We should only see one document due to the effect of cascade delete from the the function [   {     "S2": {       "name": "Joe Smith",       "zip": "94501"     },     "id": "Joe::94040"   } ]

2. OnUpdate() function to keep the aggregate information ready periodically.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

function OnUpdate(doc, meta) {     try {         var stattime = SELECT lastupdate FROM T1 USE KEYS ["trstattime"];         for (var t of stattime) {             var lt = t.lastupdate;             var d = new Date();             var n = d.getTime();             // Every 10 minutes or more             if ((n - lt)  > (1000 * 60 * 10)) {                   UPDATE  T1 USE KEYS["trstattime"]  SET lastupdate = NOW_MILLIS() ;                 var ginfo = SELECT type, count(type) typecount FROM `travel-sample` WHERE `type` IS NOT MISSING GROUP BY type;                 for (var val of ginfo) {                    var ckey = "trcount::" + val.type;                    var vtype = val.type; var vtc = val.typecount;                    UPDATE T1 USE KEYS[$ckey] SET  type =  $vtype, typecount = $vtc;                 }         }      }     } catch(e) { log(e); }     }

References:

1. Couchbase documentation: https://docs.couchbase.com/server/5.5/eventing/eventing-overview.html
2. Couchbase blogs on eventing: https://blog.couchbase.com