How to automate data lineage in purview from Databricks Unity catalog table?

Hi Aparna S
Thanks for sharing the detailed context and confirming that:

• Lineage is visible in Unity Catalog
• The Purview scanner is configured with lineage extraction enabled
• The service principal has access to system.access.table_lineage and system.access.column_lineage
• Yet lineage is not appearing in Purview, as shown in your screenshot

Here's what could be happening and how to troubleshoot:

Lineage export from Unity Catalog vs. ingestion by Purview

While lineage exists inside Databricks, Purview relies on the REST-based Databricks Unity Catalog APIs to extract it. This means:

• The scanner does not infer lineage automatically just because it exists in Databricks
• It needs the correct Databricks lineage API endpoints to respond successfully

Check this first:

• Make sure the Databricks account has lineage tracking enabled at the workspace and catalog level
• Ensure the service principal has Unity Catalog "Data Lineage Consumer" role in addition to object/table-level access

Scanner settings

Even if you enabled lineage extraction in the scanner, there are additional config requirements:

• Use the Microsoft Purview Unity Catalog (Databricks) scanner, not the generic metastore or Hive scanner
• Scanner must target the Unity Catalog metastore directly, not just the workspace
• The scan must complete successfully and report lineage extraction: succeeded in the status

Known limitations

As of now (mid-2025), Purview only supports table-level lineage extraction from Unity Catalog-not notebook or SQL function-level traces.

Also, lineage won’t appear unless a valid job context (e.g., Databricks Job or Delta Live Table pipeline) has generated lineage metadata. Interactive queries alone might not always trigger exportable lineage.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

I couldn't find the role - Data Lineage Consumer but validated remaining settings. In my scenario, the table is created from data in ADLS (external data in UC). I suspect that would potentially be the reason why I can see the lineage in UC and not in Purview.

Hi Aparna S

Thanks for validating the settings and for highlighting that the table is built over external ADLS data in Unity Catalog.

You're absolutely right in your suspicion - currently, Purview doesn't support lineage extraction for external tables in Unity Catalog (i.e., tables where the data source is ADLS and not Delta managed by UC). This is a known limitation. While Unity Catalog internally displays lineage for such tables, the Databricks-Purview scanner can only extract lineage for managed Delta tables where job-level metadata (from pipelines, jobs, or notebooks) is recorded and accessible via the REST lineage APIs.

Here’s what you can consider:

• Lineage extraction in Purview works best with managed Delta tables created and modified via tracked jobs or pipelines.
• External tables (e.g., defined over ADLS paths) often lack sufficient metadata exposure for Purview to ingest lineage, even if visible within UC.

Suggested Workaround:

If lineage tracking is business critical for these external tables, you could:

• Use Delta Live Tables (DLT) or notebook jobs to materialize the transformation - this will register lineage metadata that the scanner can ingest.
• Or switch to managed Delta format for the target table, if feasible.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the answer if it helps - that may also benefit other users with similar issues.

Aparna S We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Sorry for the delay in response. I am a bit confused because I am trying to fetch lineage for a managed delta table. In my scenario, I have a source file residing in ADLS and I am using a databricks notebook to create a managed delta table and load my csv file to that table. I am able to see the lineage information in UC but when connecting to Purview and doing a scan, lineage is not available.

As a workaround I am trying to fetch lineage from Databricks and post it to Azure Purview using REST APIs