Purview Data Quality Scan - DataBricks Source. DataType TimeStamp not supported

Question

Purview Data Quality Scan - DataBricks Source. DataType TimeStamp not supported

Adam Crawford 20

Hi All,

In Microsoft Purview. I scan a data source - Databricks Unity Catalog into Purview Data map and some columns are shown in the schema as TimeStamp

User's image

I assign the data bricks table as an asset in a Data Product and then connect to the same data bricks source to run Data Quality rules. I profile the same table and the same columns are now shown as Unsupported, meaning I cannot design rules which use these columns

User's image

I connect using a service principle added just for this purpose, not the Purview Account Service Principle. I can read and use all other column data types in the rules.

How do I make the TimeStamp datatype columns supportable in Purview data quality rules.? I am not wanting to ask data engineers to separate out the columns into Year, Month, Date as a work around , please. At this time it seems like a service limitation / bug.

Thanks in advance.

Smaran Thoomu 28,225 Reputation points Microsoft External Staff Moderator

2025-07-28T07:02:03.6333333+00:00
Hi Adam Crawford
Thanks for sharing the detailed scenario and screenshots - they make the issue very clear.

What you're observing with the TIMESTAMP columns being marked as "Unsupported" during Purview Data Quality (DQ) profiling is currently a known limitation of the DQ engine, particularly when working with Databricks Unity Catalog sources.

During the Data Map ingestion, TIMESTAMP fields are recognized and correctly registered in the schema.

However, when authoring Data Quality rules or profiles, these same fields appear as unsupported, making them unavailable for validation or transformation logic.

Possible Causes:

Purview Data Quality (DQ) doesn’t yet fully support all complex data types, including TIMESTAMP, from external sources like Unity Catalog - especially when accessed via a custom service principal instead of the default Purview MSI.

There may be internal metadata mapping gaps between Databricks and Purview's profiling engine for datetime types.

Suggested Workarounds:

Test with Purview’s default MSI (Managed Identity) to see if support changes.

If immediate rule authoring is critical, consider creating derived columns (e.g., casted to string or date) in a staging view within Databricks - just for DQ profiling purposes.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues
Adam Crawford 20 Reputation points

2025-08-11T14:39:17.38+00:00

We are having to cast timestamps as date columns.
Smaran Thoomu 28,225 Reputation points Microsoft External Staff Moderator

2025-08-11T19:03:44.77+00:00

Adam Crawford Thanks for confirming that you’ve had to cast the TIMESTAMP columns as DATE to proceed with Purview Data Quality rules. As mentioned earlier, this behavior is due to a current service limitation in the DQ engine for TIMESTAMP fields from Databricks Unity Catalog sources.

The product team is already evaluating this gap, but at present the supported approach is either:

Using the default Purview MSI to check if the behavior changes for your environment, or

Casting/converting these columns to a supported type before DQ profiling (as you’ve done).

Since this is a common scenario, I recommend logging your use case in the Azure Feedback Portal so it can be prioritized. If you share the link here, we can help route it internally for more visibility.

Hope this clarifies the current state and next steps.

1 answer

Your answer

Smaran Thoomu 28,225 Reputation points Microsoft External Staff Moderator

2025-07-28T07:02:03.6333333+00:00

Hi Adam Crawford
Thanks for sharing the detailed scenario and screenshots - they make the issue very clear.

What you're observing with the TIMESTAMP columns being marked as "Unsupported" during Purview Data Quality (DQ) profiling is currently a known limitation of the DQ engine, particularly when working with Databricks Unity Catalog sources.

During the Data Map ingestion, TIMESTAMP fields are recognized and correctly registered in the schema.

However, when authoring Data Quality rules or profiles, these same fields appear as unsupported, making them unavailable for validation or transformation logic.

Possible Causes:

Purview Data Quality (DQ) doesn’t yet fully support all complex data types, including TIMESTAMP, from external sources like Unity Catalog - especially when accessed via a custom service principal instead of the default Purview MSI.

There may be internal metadata mapping gaps between Databricks and Purview's profiling engine for datetime types.

Suggested Workarounds:

Test with Purview’s default MSI (Managed Identity) to see if support changes.

If immediate rule authoring is critical, consider creating derived columns (e.g., casted to string or date) in a staging view within Databricks - just for DQ profiling purposes.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues
Adam Crawford 20 Reputation points

2025-08-11T14:39:17.38+00:00

We are having to cast timestamps as date columns.
Smaran Thoomu 28,225 Reputation points Microsoft External Staff Moderator

2025-08-11T19:03:44.77+00:00

Adam Crawford Thanks for confirming that you’ve had to cast the TIMESTAMP columns as DATE to proceed with Purview Data Quality rules. As mentioned earlier, this behavior is due to a current service limitation in the DQ engine for TIMESTAMP fields from Databricks Unity Catalog sources.

The product team is already evaluating this gap, but at present the supported approach is either:

Using the default Purview MSI to check if the behavior changes for your environment, or

Casting/converting these columns to a supported type before DQ profiling (as you’ve done).

Since this is a common scenario, I recommend logging your use case in the Azure Feedback Portal so it can be prioritized. If you share the link here, we can help route it internally for more visibility.

Hope this clarifies the current state and next steps.

Answer 1

Austin Lowes 0

Workaround sounds viable but timestamps are commonplace and hardly new. Surely if this is a known limitation there would be an MS workaround planned in the pipeline, for example to do the casting within Purview? There are bound to be others out there that will experience this too...

Smaran Thoomu 28,225 Reputation points Microsoft External Staff Moderator

2025-07-30T18:13:18.6266667+00:00

Hi Austin Lowes
Thanks again for the thoughtful follow-up - and you're absolutely right, TIMESTAMP support is essential in many scenarios, and we agree the current limitation is not ideal.

While this is already being evaluated internally by the product team, we highly encourage you to submit this requirement as formal product feedback via the Azure Feedback Portal. Having it raised and upvoted directly by users like you significantly helps with visibility and prioritization across engineering backlogs.

Please feel free to include your specific Databricks + Purview scenario when submitting, as real-world context adds weight to the request.

Let us know once done and we can internally route it to the appropriate team for added visibility.
I hope this information helps. Please do let us know if you have any further queries.

Share via

Purview Data Quality Scan - DataBricks Source. DataType TimeStamp not supported

1 answer

Your answer