Auditing Cosmos DB data using change feed triggers

You may have already heard about Cosmos DB, as it has become very popular and many organizations are using it because of the features it provides.

In this recipe, you will learn to integrate serverless Azure Functions with a serverless NoSQL database in Cosmos DB. You can read more about Cosmos DB at https://docs.microsoft.com/azure/cosmos-db/introduction.

It might often be necessary to keep the change logs of fields, attributes, items, and other aspects for auditing purposes. In the world of relational databases, you might have seen developers using triggers or stored procedures to implement this kind of auditing functionality, where you write code to store data in a separate audit table.

In this recipe, you'll learn how easy it is to audit the changes made to Cosmos DB containers by writing a simple function that gets triggered whenever there is a change to an item in a Cosmos DB container.

Note

In the world of relational databases, a container is the same as a database table and an item is the same as a record.

Getting ready

In order to get started, you need to first do the following:

1. Create a Cosmos DB account.
2. Create a new Cosmos DB container where you can store data in the form of items.

Let's begin by creating a new Cosmos DB account.

Creating a new Cosmos DB account

Navigate to the Azure portal and create a new Cosmos DB account. You will need to provide the following:

• A valid subscription and a resource group.
• A valid account name. This will create an endpoint at <<accountname>>.document.azure.com.
• An API—set this as SQL. This will ensure that you can write queries in SQL. Feel free to try out other APIs.

Creating a new Cosmos DB container

Create a new Cosmos DB container by performing the following steps:

1. Once the account has been created, create a new database and a container. You can create both of them in a single step right from the Azure portal.
2. Navigate to the Overview tab and click on the Add Container button to create a new container:
   

   Figure 3.31: Cosmos DB account—Overview blade

3. You will now be navigated to the Data Explorer tab automatically, where you will be prompted to provide the following details:
   

   Figure 3.32: Cosmos DB—creating a container

   The following are the details that you'll need to add to Figure 3.32:

   

   Figure 3.33: Cosmos DB—creating a container—fields

4. Next, click on the OK button to create the container. If everything went well, you'll see something like the following in the Data Explorer tab of the Cosmos DB account:
   

Figure 3.34: Data Explorer—Cosmos DB database and container

We have successfully created a Cosmos DB database and a container, as shown in Figure 3.34. Let's now go through how to integrate the container with a new Azure function and see how to trigger it whenever there is a change in the Cosmos DB container.

How to do it...

In this section, we'll integrate a Cosmos DB container with Azure Functions by performing the following steps:

1. Navigate to the Cosmos DB account and click on the Add Azure Function menu in the All settings blade of the Cosmos DB account, as shown in Figure 3.35:
   

   Figure 3.35: Cosmos DB—the Add Azure Function menu item

2. You will now be taken to the Add Azure Function blade, where you will choose the Azure function app in which you would like to create a new function (Cosmos DB trigger), as shown in Figure 3.36:
   

   Figure 3.36: Cosmos DB—Azure function integration

3. Once you have reviewed the details, click on the Save button (shown in Figure 3.36) to create the new function, which will be triggered for every change that is made in the container. Let's quickly navigate to the Azure function app (in my case, it is cosmosdbazurefunctions) and see whether the new function with the name productsTrigger has been created. Here is what the function app looks like:
   

   Figure 3.37: Azure Functions—Cosmos DB trigger

4. Replace the default code with the following code of the Azure Functions Cosmos DB trigger, which gets a list of all the items that were updated. The following code just prints the count of items that were updated and the ID of the first item in the Logs console:

   #r "Microsoft.Azure.DocumentDB.Core"

   using System;

   using System.Collections.Generic;

   using Microsoft.Azure.Documents;

   public static void Run(IReadOnlyList<Document> input, ILogger log)

   {

   if (input != null && input.Count > 0)

   {

   log.LogInformation("Items modified " + input.Count); log.LogInformation("First Item Id " + input[0].Id);

   }

   }

5. Now, the integration of the Cosmos DB container and the Azure function is complete. Let's add a new item to the container and see how the trigger gets fired in action. Open a new browser window/tab (leaving the productsTrigger tab open in the browser), navigate to the container, and create a new item by clicking on the New Item button, as shown in Figure 3.38:
   

   Figure 3.38: Data Explorer—creating a new item

6. Once you have replaced the default JSON (which just has an id attribute) with the JSON that has the required attributes, click on the Save button to save the changes and quickly navigate to the other browser tab, where you have the Azure function open, and view the logs to see the output of the function. The following is how my logs look, as I just added a value to the id attribute of the item. It might look different for you, depending on your JSON structure:
   

Figure 3.39: Azure function—the Logs console

In this section, you have learned how to integrate Azure Functions with Cosmos DB. Let's now move on to the next section.

There's more...

While integrating Azure Functions to track Cosmos DB changes, it will automatically create a new container named leases, as shown in Figure 3.40. Be aware that this is an additional cost, as the cost in Cosmos DB is based on the request units (RUs) that are allocated for each container:

Figure 3.40: Cosmos DB—the Containers list

It's important to note that the Cosmos DB trigger wouldn't be triggered (at the time of writing) for any deletes in the container. It is only triggered for creates and updates to items in a container. If it is important for you to track deletes, and then you need to execute soft deletes, which means setting an attribute such as isDeleted to true for records that are deleted by the application and based on the value of the isDeleted attribute, implementing your custom logic in the Cosmos DB trigger.

The integration that we have done between Azure Functions and Cosmos DB uses Cosmos DB change feeds. You can learn more about change feeds here: https://docs.microsoft.com/azure/cosmos-db/change-feed

Don't forget to delete the Cosmos DB account and its associated containers if you think you will no longer use them, because the containers are charged based on the RUs allocated, even if you are not actively using them.

If you are not able to run this Azure function or you get an error saying that Cosmos DB extensions are not installed, then try creating a new Azure Cosmos DB trigger using the templates available, which should then prompt installation.

In this recipe, we first created a Cosmos DB account and created a database and a new container within it. Once the container was created, we integrated it from within the Azure portal by clicking on the Add Azure Function button, which is available at the Cosmos DB account level. We chose the required function app in which we wanted to create a Cosmos DB trigger. Once the integration was complete, we created a sample item in the Cosmos DB container and then verified that the function was triggered automatically for all the changes (all reads and writes but not deletes) that we will make on the container.

Let's now proceed to integrate Azure Functions with Azure Data Factory.