Skip to content

You are viewing documentation for Immuta version 2023.1.

For the latest version, view our documentation for Immuta SaaS or the latest self-hosted version.

cancel

Generate Views with Webhooks

This document details how to externally generate native views for databases that are currently not supported as integrations in Immuta using the Immuta API, equalized projects, and webhooks.

Introduction

Immuta supports several integrations that allow users to interact directly with their systems without going through Immuta's Query Engine, which acts as a proxy. For systems that Immuta does not currently support as an integration, users can generate native views for these systems using

The implementation of generating native views will vary based on internal business requirements; however, the general workflow and interaction with the Immuta API and webhooks are outlined below.

Webhooks

A number of webhooks ensure native views represent the current state of the policies enforced. The table below describes each of these notification types.

Notification Type Description
dataSourceUpdated This notification is triggered whenever a data source is updated, such as when it is enabled, disabled, or renamed; connection information or columns change; or metadata associated with the source changes.
addedToProject This notification is triggered when a data source is added to a project.
removedFromProject This notification is triggered when a data source is removed from a project.
policyUpdated This notification is triggered whenever there is a policy change.

Register for webhook notifications

To receive webhook notifications, you must register for the notifications as a user with the APPLICATION_ADMIN permission to create a global webhook. Unlike a regular webhook, a global webhook ensures that you receive all notifications, even if they don’t pertain to the particular user who registered the hook. Note: You should only register the hook once.

All user-configured webhook integrations must respond within 10 seconds of receiving the webhook request payload. If the webhook integration takes longer to respond, the request will timeout.

Determine if the webhook exists

Before registering a webhook, check if the webhook already exists by comparing it to the names and URLs of existing webhooks. If there is already a webhook with the name and url of the one you want to register, it already exists and you don't need to register it. Completing this process whenever your service starts up will ensure that you never miss notifications.

Use the request and example response provided below for guidance:

Request

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer <TOKEN>" \
    https://demo.immuta.com/webhooks

Example Response

[
    {
        "id": 1,
        "name": "internal-automatic-subscription-webhook",
        "url": "/internal-webhook/automatic-subscription",
        "notificationType": [
            "groupUserAdded",
            "groupUserDeleted",
            "attributeAdded",
            "attributeRemoved",
            "attributeUpdated",
            "userDeleted",
            "userMigrated",
            "policyUpdated",
            "dataSourceUpdated",
            "modelCreated"
        ],
        "global": true,
        "createdBy": 21
    }]
]

Register a webhook

After you have determined that the webhook does not exist, register it with the four notification types outlined in the payload example below.

Request

curl \
    --request POST \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer <TOKEN>" \
    --data @example-payload.json \
    https://demo.immuta.com/webhooks

Example Payload

{
    "webhooks": [{
        "name": "ViewMonitorHook",
        "global": true,
        "notificationType": [
            "dataSourceUpdated",
            "policyUpdated",
            "addedToProject",
            "removedFromProject"
        ],
        "url": "https://external.service.com/processWebhook"
    }]
}

Make API calls when notifications are received

Once a notification is received, additional calls need to be made to the Immuta API to ensure the native views reflect the current state of policies, data sources, and projects. These calls depend on the notification received, so the calls required for each are outlined by notification type throughout this section.

A data source has been updated

Use the tabs below for guidance when you receive the dataSourceUpdated notification.

Workflow

When the dataSourceUpdated notification is triggered, the columns, credentials, or even the backing query of a data source may have changed or the data source may have been disabled or enabled. When a dataSourceUpdated notification is received,

  1. Get the data source metadata to verify that the updated data source belongs to an equalized project. (See the 1. Get Data Source Metadata tab.)

    Note: Notifications are not filtered by blobHandlerType, so you may receive notifications for irrelevant data source types. Whenever you receive this notification, check the returned blobHandlerType when you call the data source endpoint.

  2. Get a list of equalized projects that contain this data source. (See the 2. Get a List of Relevant Projects tab.)

  3. Regenerate the view for each of those projects by calling the view SQL endpoint:
    • Remove the view if the source is disabled.
    • Create the view (if the data source was disabled but has now been enabled).
    • Update the view (if some other update caused the view to change).

dataSourceUpdated Notification

{
    "dataSourceName": "Customer",
    "fields": { "disabled": true, "handler": "Snowflake" },
    "notificationType": "dataSourceUpdated",
    "actionBy": 1,
    "modelType": "datasource",
    "modelId": 3,
    "targetUser": 1,
    "targetGroup": null,
    "actionByName": "John Doe",
    "targetUserName": "John Doe"
}

Get Data Source Metadata Request

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer <TOKEN>" \
    https://demo.immuta.com/dataSource/<DATASOURCE ID>

Get a List of Relevant Projects

  1. Call the project search endpoint and pass the data source id included in the data source metadata response.

    Note: This API call should be made as a user with the APPLICATION_ADMIN permission so that all matching projects are returned regardless of subscription status.

  2. Regenerate the view for each of those projects by calling the view SQL endpoint.

Request

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer <TOKEN>" \
    https://demo.immuta.com/projects?isEqualized=true&dataSourceId=<DATASOURCE ID>

Example Response

{
    "hits": [
        {
            "id": 9,
            "name": "Regulated Clients",
            "status": "open",
            "description": "A project containing data for regulated clients",
            "deleted": false,
            "updatedAt": "2021-01-15T11:10:23.349Z",
            "subscriptionPolicy": null,
            "createdAt": "2021-01-11T15:49:03.146Z",
            "filterId": 29,
            "subscriptionType": "manual",
            "isEqualized": true,
            "acknowledgeRequired": false,
            "subscriptionStatus": "owner",
            "purposeCount": 0,
            "hasDeletedPurposes": false,
            "workspace": null,
            "dataSourceReason": null,
            "dataSourceAddedAt": "2021-01-13T17:04:51.941Z",
            "datasourceAddedBy": "John Doe",
            "allowMaskedJoins": true
        }, {
            "id": 3,
            "name": "All Clients",
            "status": "open",
            "description": "A project containing data for current clients",
            "deleted": false,
            "updatedAt": "2021-01-12T11:05:54.317Z",
            "subscriptionPolicy": null,
            "createdAt": "2021-01-11T09:30:12.843Z",
            "filterId": 13,
            "subscriptionType": "manual",
            "isEqualized": true,
            "acknowledgeRequired": true,
            "subscriptionStatus": "owner",
            "purposeCount": 1,
            "hasDeletedPurposes": false,
            "workspace": null,
            "dataSourceReason": null,
            "dataSourceAddedAt": "2021-01-11T09:30:13.215Z",
            "datasourceAddedBy": "John Doe",
            "allowMaskedJoins": true
        }
    ]
}

A data source has been added to a project

Use the tabs below for guidance when you receive the addedToProject notification.

Workflow

When a data source is added to a project,

  1. Get the data source metadata to verify that the data source is queryable by checking its blobHandlerType. (See the 1. Get Data Source Metadata tab.)
  2. Get the project metadata and confirm that the project is equalized: isEqualized: true. (See the 2. Get Project Metadata tab.)
  3. Call the view SQL endpoint to generate the SQL for the native view.

addedToProject Notification

{
    "dataSourceName": "Case",
    "dataSourceId": 1,
    "projectName": "Demo Project",
    "notificationType": "addedToProject",
    "actionBy": 1,
    "modelType": "project",
    "modelId": 1,
    "targetUser": 1,
    "targetGroup": null,
    "actionByName": "John Doe",
    "targetUserName": "John Doe"
}

Get Data Source Metadata Request

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer <TOKEN>" \
    https://demo.immuta.com/dataSource/<DATASOURCE ID>

Get Project Metadata Request

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer <TOKEN>" \
    https://demo.immuta.com/project/<PROJECT ID>

A data source has been removed from a project

Use the tabs below for guidance when you receive the removedFromProject notification.

Workflow

When a data source is removed from an equalized project, the corresponding native view should be deleted:

  1. Get the project metadata and confirm that the project is equalized: isEqualized: true. (See the 1. Get Project Metadata tab.)
  2. Get the data source metadata to verify that it is a data source that has a corresponding native view. (See the 2. Get Data Source Metadata tab.)
  3. Remove the native view by calling the view SQL endpoint.

removedFromProject Notification

{
    "dataSourceName": "Case",
    "dataSourceId": 1,
    "projectName": "Demo Project",
    "notificationType": "removedFromProject",
    "actionBy": 1,
    "modelType": "project",
    "modelId": 1,
    "targetUser": 1,
    "targetGroup": null,
    "actionByName": "John Doe",
    "targetUserName": "John Doe"
}

Get Project Metadata Request

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer <TOKEN>" \
    https://demo.immuta.com/project/<PROJECT ID>

Get Data Source Metadata Request

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer <TOKEN>" \
    https://demo.immuta.com/dataSource/<DATASOURCE ID>

A policy has been updated

When you receive the policyUpdated notification, determine if the policy has been updated for a project or data source by checking the modelType property in the notification, and then navigate to the corresponding section below for guidance.

modelType: dataSource

Workflow

If the notification is for a data source,

  1. Get the data source metadata and check the blobHandlerType to ensure that the data source is queryable. (See the 1. Get Data Source Metadata tab.)
  2. Get a list of equalized projects that contain the data source. (See the 2. Get a List of Relevant Projects tab.)
  3. Once you have the list of projects, make API calls to the view SQL endpoint for the impacted data source for each of the projects. For example, if the data source has an id of 11 and there were 4 projects with ids 1,4,5, and 9, you would need to make 4 API calls to generate the SQL for the native views.

policyUpdated Notification

{
    "dataSourceName": "Customer",
    "triggeredByGlobal": false,
    "conflictCount": 0,
    "policyType": "data",
    "handlerId": 4,
    "previousHandlerId": null,
    "dataSourceType": "queryable",
    "notificationType": "policyUpdated",
    "actionBy": 1,
    "modelType": "datasource",
    "modelId": 2,
    "notifyInitiator": false,
    "targetUser": 1,
    "targetGroup": null,
    "actionByName": "John Doe",
    "targetUserName": "John Doe"
}

Get Data Source Metadata Request

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer <TOKEN>" \
    https://demo.immuta.com/dataSource/<DATASOURCE ID>

Get a List of Relevant Projects

  1. Call the project search endpoint and pass the data source id from the notification. Note: This API call should be made as a user with the APPLICATION_ADMIN permission so that all matching projects are returned regardless of subscription status.
  2. Make API calls to the view SQL endpoint for the impacted data source for each of the projects to regenerate the views.

Request

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer <TOKEN>" \
    https://demo.immuta.com/projects?isEqualized=true&dataSourceId=<DATASOURCE ID>

Example Response

{
    "hits": [
        {
            "id": 9,
            "name": "Regulated Clients",
            "status": "open",
            "description": "A project containing data for regulated clients",
            "deleted": false,
            "updatedAt": "2021-01-15T11:10:23.349Z",
            "subscriptionPolicy": null,
            "createdAt": "2021-01-11T15:49:03.146Z",
            "filterId": 29,
            "subscriptionType": "manual",
            "isEqualized": true,
            "acknowledgeRequired": false,
            "subscriptionStatus": "owner",
            "purposeCount": 0,
            "hasDeletedPurposes": false,
            "workspace": null,
            "dataSourceReason": null,
            "dataSourceAddedAt": "2021-01-13T17:04:51.941Z",
            "datasourceAddedBy": "John Doe",
            "allowMaskedJoins": true
        }, {
            "id": 3,
            "name": "All Clients",
            "status": "open",
            "description": "A project containing data for current clients",
            "deleted": false,
            "updatedAt": "2021-01-12T11:05:54.317Z",
            "subscriptionPolicy": null,
            "createdAt": "2021-01-11T09:30:12.843Z",
            "filterId": 13,
            "subscriptionType": "manual",
            "isEqualized": true,
            "acknowledgeRequired": true,
            "subscriptionStatus": "owner",
            "purposeCount": 1,
            "hasDeletedPurposes": false,
            "workspace": null,
            "dataSourceReason": null,
            "dataSourceAddedAt": "2021-01-11T09:30:13.215Z",
            "datasourceAddedBy": "John Doe",
            "allowMaskedJoins": true
        }
    ]
}

modelType: project

Workflow

If the notification is for a project,

  1. Get the project metadata and confirm that the project is equalized: isEqualized: true. (See the 1. Get Project Metadata tab.)
  2. Get a list of all data sources contained in the project, and then filter these sources based on the blobHandlerType of each source. (See the 2. Get a List of Relevant Data Sources tab.)
  3. Call the view SQL endpoint to regenerate the native views for these data sources.

policyUpdated Notification

{
    "projectName": "Demo Project",
    "policyType": "subscription",
    "hasWorkspace": false,
    "notificationType": "policyUpdated",
    "actionBy": 1,
    "modelType": "project",
    "modelId": 1,
    "targetUser": 1,
    "targetGroup": null,
    "actionByName": "John Doe",
    "targetUserName": "John Doe"
}

Get Project Metadata Request

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer <TOKEN>" \
    https://demo.immuta.com/project/<PROJECT ID>

Get a List of Relevant Data Sources

  1. Use the project data sources endpoint and specify the project id from the notification. Note: This API call should be made as a user with the APPLICATION ADMIN permission so that all matching projects are returned regardless of subscription status.
  2. Filter the results based on the blobHandlerType value.
  3. Call the view SQL endpoint to regenerate the native views for these data sources.

Request

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer <TOKEN>" \
    https://demo.immuta.com/project/<PROJECT ID>/datasources

Example Response

{
    "count": 2,
    "dataSources": [
        {
            "addedBy": "John Doe",
            "dataSourceName": "Regulated Clients",
            "policyHandlerType": "None",
            "addedOn": "2021-01-15T11:10:23.349Z",
            "dataSourceId": 16,
            "addedByProfile": 1,
            "comment": null,
            "deleted": false,
            "subscriptionType": "manual",
            "subscriptionStatus": "owner",
            "subscriptionPolicy": null,
            "connectionString": "read_only@db.example.com:443/data",
            “"blobHandlerType": "Apache Impala",
            "derivedInThisProject": false
        }, {
            "addedBy": "John Doe",
            "dataSourceName": "Client Contact Information",
            "policyHandlerType": "None",
            "addedOn": "2021-01-15T11:10:23.517Z",
            "dataSourceId": 19,
            "addedByProfile": 1,
            "comment": null,
            "deleted": false,
            "subscriptionType": "manual",
            "subscriptionStatus": "owner",
            "subscriptionPolicy": null,
            "connectionString": "read_only@db.example.com:443/data",
            "blobHandlerType": "Apache Impala",
            "derivedInThisProject": false
        }
    ]
}

Additional notification examples

The tabs below provide additional examples of common notifications.

Data Source Disabled

{
    "dataSourceName": "Customer",
    "fields": { "disabled": true, "handler": "Snowflake" },
    "notificationType": "dataSourceUpdated",
    "actionBy": 1,
    "modelType": "datasource",
    "modelId": 3,
    "targetUser": 1,
    "targetGroup": null,
    "actionByName": "John Doe",
    "targetUserName": "John Doe"
}

Data Source Enabled After Being Disabled

{
    "action": "update",
    "dataSourceName": "Customer",
    "fields": { "disabled": false, "handler": "Snowflake" },
    "notificationType": "dataSourceUpdated",
    "actionBy": 1,
    "modelType": "datasource",
    "modelId": 3,
    "targetUser": 1,
    "targetGroup": null,
    "actionByName": "John Doe",
    "targetUserName": "John Doe"
}

Data Source Added To Project

{
    "dataSourceName": "Case",
    "dataSourceId": 1,
    "projectName": "Demo Project",
    "notificationType": "addedToProject",
    "actionBy": 1,
    "modelType": "project",
    "modelId": 1,
    "targetUser": 1,
    "targetGroup": null,
    "actionByName": "John Doe",
    "targetUserName": "John Doe"
}

Data Source Removed From Project

{
    "dataSourceName": "Case",
    "dataSourceId": 1,
    "projectName": "Demo Project",
    "notificationType": "removedFromProject",
    "actionBy": 1,
    "modelType": "project",
    "modelId": 1,
    "targetUser": 1,
    "targetGroup": null,
    "actionByName": "John Doe",
    "targetUserName": "John Doe"
}

Project Created

{
    "projectName": "Demo Project",
    "notificationType": "modelCreated",
    "actionBy": 1,
    "modelType": "project",
    "modelId": 1,
    "targetUser": 1,
    "targetGroup": null,
    "actionByName": "John Doe",
    "targetUserName": "John Doe"
}

Global Data Policy Applied (will receive one per affected data source)

{
    "dataSourceName": "Customer",
    "triggeredByGlobal": true,
    "conflictCount": 1,
    "policyType": "data",
    "handlerId": 1,
    "previousHandlerId": null,
    "dataSourceType": "queryable",
    "applied": [ "Redact It All" ],
    "notificationType": "policyUpdated",
    "actionBy": 1,
    "modelType": "datasource",
    "modelId": 2,
    "notifyInitiator": true,
    "targetUser": 1,
    "targetGroup": null,
    "actionByName": "John Doe",
    "targetUserName": "John Doe"
}

Global Data Policy Removed (will receive one per affected data source)

{
    "dataSourceName": "Date Dim",
    "triggeredByGlobal": true,
    "conflictCount": 0,
    "policyType": "data",
    "handlerId": null,
    "previousHandlerId": 2,
    "dataSourceType": "queryable",
    "removed": [ "Redact It All" ],
    "notificationType": "policyUpdated",
    "actionBy": 1,
    "modelType": "datasource",
    "modelId": 3,
    "notifyInitiator": true,
    "targetUser": 1,
    "targetGroup": null,
    "actionByName": "John Doe",
    "targetUserName": "John Doe"
}

Local Data Policy Applied

{
    "dataSourceName": "Customer",
    "triggeredByGlobal": false,
    "conflictCount": 0,
    "policyType": "data",
    "handlerId": 4,
    "previousHandlerId": null,
    "dataSourceType": "queryable",
    "notificationType": "policyUpdated",
    "actionBy": 1,
    "modelType": "datasource",
    "modelId": 2,
    "notifyInitiator": false,
    "targetUser": 1,
    "targetGroup": null,
    "actionByName": "John Doe",
    "targetUserName": "John Doe"
}

Local Data Policy Removed

{
    "dataSourceName": "Customer",
    "triggeredByGlobal": false,
    "conflictCount": 0,
    "policyType": "data",
    "handlerId": null,
    "previousHandlerId": 4,
    "dataSourceType": "queryable",
    "notificationType": "policyUpdated",
    "actionBy": 1,
    "modelType": "datasource",
    "modelId": 2,
    "notifyInitiator": false,
    "targetUser": 1,
    "targetGroup": null,
    "actionByName": "John Doe",
    "targetUserName": "John Doe"
}

Project Policies Updated

{
    "projectName": "Demo Project",
    "policyType": "subscription",
    "hasWorkspace": false,
    "notificationType": "policyUpdated",
    "actionBy": 1,
    "modelType": "project",
    "modelId": 1,
    "targetUser": 1,
    "targetGroup": null,
    "actionByName": "John Doe",
    "targetUserName": "John Doe"
}