Skip to content

You are viewing documentation for Immuta version 2023.1.

For the latest version, view our documentation for Immuta SaaS or the latest self-hosted version.

cancel

Schema Monitoring

Schema monitoring allows organizations to monitor their data environments. When it is enabled, Immuta monitors the organization's servers to detect when new tables or columns are created or deleted, and automatically registers (or disables) those tables in Immuta. These newly updated data sources will then have any global policies and tags that are set in Immuta applied to them. The Immuta data dictionary will be updated with any column changes, and the Immuta environment will be in sync with the organization's data environment. This automated process helps organization keep compliant without the need to manually keep data sources up to date.

Schema monitoring is enabled while creating or editing a data source. It runs every night by default but can be configured to a different frequency. Data Owners or Governors can edit the naming convention for newly detected data sources and the Schema Detection Owner from the schema project page after it has been enabled.

See Create a Data Source for instructions on enabling schema monitoring or Manage Schema Monitoring for instructions on editing the schema monitoring settings.

Column Detection

Column detection is a part of schema monitoring, but can also be enabled on its own to detect the column changes of a select group of tables. Column detection monitors when columns are added or removed from a table and when column types are changed and updates those changes in the appropriate Immuta data source's data dictionary.

See Create a Data Source for instructions on enabling column detection.

Tracking New Data Sources and Columns

When new data sources and columns are detected and added to Immuta, they will automatically be tagged with the New tag. This allows Governors to use the seeded New Column Added Global Policy to mask the data sources and columns, since they could contain sensitive data. Data Owners can then review and approve these changes from the Requests tab of their profile page. Approving column changes removes the New tags from the data source.

The New Column Added Global Policy is active by default.

See Clone, Activate, or Stage a Global Policy to stage this seeded Global Policy if you do not want new columns automatically masked.

Workflow

  1. Immuta user creates a data source with Schema Monitoring enabled.
  2. Every 24 hours, at 12:30 a.m. UTC by default, Immuta checks the servers for any changes to tables and columns.

  3. If Immuta detects a change, it will update the appropriate Immuta data source or column:

    1. If Immuta detects a new table, then Immuta creates an Immuta data source for that table and tags it "New".
    2. If Immuta detects a table has been deleted, then Immuta disables that table's data source.
    3. If Immuta detects a previously deleted table has been re-created, then Immuta restores that table's data source and tags it "New".
    4. If Immuta detects a new column within a table, then Immuta adds that column to the data dictionary and tags it "New".
    5. If Immuta detects a column has been deleted, then Immuta deletes that column from the data dictionary.
    6. If Immuta detects a column type has changed, then Immuta updates the column type in the data dictionary.
    7. Data sources and columns tagged "New" will be masked by the seeded New Column Added Global Policy until a Governor or Data Owner approves the changes.

To run schema monitoring or column detection manually, see the Manually Run Jobs page.