Share via

in dlt pipeline , after dlt.except_all("rules") , the dlt.table can not filter the column("_dlt_valid")

dgeorgewnxu 0 Reputation points
2025-06-26T07:17:55.77+00:00

in my DLT pipeline , I want load the bronze table data into silver table after validation , other non-valid data load into reject table . in silver silver use the function dlt.except_all("rules") , but in silver layer part , the pipeline can not filter the column "_dlt_valid" (the column "_dlt_valid" should be create by function except_all) .

Developer technologies | Windows Forms
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Omkara Varshitha Kunapalli (INFOSYS LIMITED) 235 Reputation points Microsoft External Staff
    2025-08-08T08:43:39.4866667+00:00

    Thanks for sharing the issue you're facing with your DLT pipeline. You're trying to load validated data from a bronze table into a silver table using dlt.except_all("rules"), and route invalid data to a reject table.

    In DLT, when using , _dlt.except_all("rules") _the dlt_valid column is generated but not always accessible in the silver layer for filtering. To fix this, use _.withColumn("dlt_valid", . . .) or _selectExpr("*", "dlt_valid") after applying expect_all. This ensures _dlt__valid is retained in the schema and usable for filtering or routing to reject tables.

     

    If you have any further questions or need additional help, feel free to reach out. Have a great day!

     

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.