Issue with Unity Catalog Lineage Not Appearing in Microsoft Purview

Hi Sooriya E Thanks for the detailed info, from what you’ve described, it sounds like you’re pretty close to having everything configured correctly.

Here are a few key things to double-check that usually cause this kind of issue with Unity Catalog lineage not showing up in Microsoft Purview:

System Tables and Runtime Version Make sure your Databricks workspace is running on Databricks Runtime 13.2 or later because that’s when system tables and lineage capture fully kicked in. Also confirm the system tables like system.access.table_lineage and system.access.column_lineage are enabled.

• https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/lineage

Permissions for Purview the Purview Managed Identity (or service principal) that runs the scan needs at least SELECT permissions on those system tables, plus USE CATALOG permission on the Unity Catalog. Without those, Purview can’t read the lineage metadata.

• https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/data-lineage

Unity Catalog–Managed Tables Only Lineage only works for tables registered inside Unity Catalog if you’re writing data to mounted ADLS paths or unmanaged files, lineage won’t be tracked. Make sure your silver and export datasets are registered Delta tables within Unity Catalog.

Purview Scan Settings Verify that your Purview scan is running a full scan with lineage extraction enabled for your Databricks source. Sometimes lineage doesn’t appear until after a full scan completes.

Validate Lineage Inside Databricks Try running a query like this in Databricks to see if lineage records actually exist for your tables:

SELECT * FROM system.access.table_lineage WHERE target_table_full_name = 'catalog.schema.table_name';

If you don’t get any results here, the problem is on the Databricks side (lineage not being captured) rather than Purview.

Finally:

• Transformations run using Pandas or Python UDFs might not generate lineage.
• Make sure there aren’t any network or firewall rules blocking Purview’s access to your Databricks workspace.
• If your transformations are complex or you need more robust lineage, you could consider using the Purview ADB Lineage Solution Accelerator which uses OpenLineage standards for richer tracking.

Hope this helps. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Hi Venkat,

We have enabled System.access.table_lineage and column_lineage. I am able to view those within the lineage tab of Unity catalog in Databricks. I am also able to query the system.access.table_lineage and column_lineage tables in the sql editor of Databricks. The Purview managed identity has been given data reader permission so it can select, browse and use catalog permissions are included in it. Can we please schedule a call so that we can resolve this ASAP.

Thanks,

Sooriya E