Skip to content

You are viewing documentation for Immuta version 2023.1.

For the latest version, view our documentation for Immuta SaaS or the latest self-hosted version.

cancel

Native Schema Monitoring for Snowflake (Private Preview)

Immuta can monitor your data environment, detect when new tables or columns are created or deleted in Snowflake, and automatically register (or disable) those tables in Immuta for you. Those newly updated data sources will then have any global policies and tags that you have set up applied to them. The Immuta data dictionary will be updated with any new columns, and your Immuta environment will be in sync with your Snowflake tables. This automated process helps with scaling and keeping your organization compliant without the need to manually keep your data sources up to date.

Architecture

Once enabled on a data source, Immuta calls to Snowflake every 24 hours by default to find when each table within the registered schema was last altered. If the timestamp is after the last time native schema monitoring was run, then Immuta will update the table or columns that have been altered. This process works well when monitoring a large number of data sources because it only updates the recently altered tables and cuts down the amount of Snowflake computing required to run column detection, which specifically updates the columns of registered data sources.

If necessary, an Immuta admin can also manually run native schema monitoring through the API to run globally on all data sources.

If you have an Immuta environment with data sources other than Snowflake, the legacy schema monitoring feature will run on all non-Snowflake data sources. The native schema monitoring feature only works with Snowflake integrations and Snowflake data sources.

Automatic workflow

  1. Immuta user creates a data source with schema monitoring enabled.
  2. Every 24 hours, at 12:30 a.m. UTC by default, Immuta sends a query to Snowflake for the information_schema view asking for when each data source’s table was last altered.
  3. If the table was altered after the last time native schema detection ran, Immuta updates the data source, columns, and data dictionary.
  4. Immuta tags new data sources and columns with the tag “New” so that you can use the templated "New Column Added" global policy to mask all new data until it has been reviewed.

Limitations

  • This feature only works with Snowflake data sources. Any non-Snowflake data sources will run with the legacy schema monitoring.
  • Your organization will not see performance improvements if it is making changes to all tables consistently. This feature is intended to improve performance for organizations with a large number of tables and relatively few changes made within the ecosystem comparatively.

Migration

There is no migration required with this feature. Once enabled, native schema monitoring will start running on all Snowflake data sources with legacy schema monitoring previously enabled and can be applied to all new Snowflake data sources.

Configuration

See the configuration page for instructions on enabling this feature.