ADF Pipeline failed with error "An insufficient number of arguments were supplied for the procedure or function cdc.fn_cdc_get_net_changes"

Question

ADF Pipeline failed with error "An insufficient number of arguments were supplied for the procedure or function cdc.fn_cdc_get_net_changes"

Aaron Wong 40

I have a pipeline setup to copy set of tables from a source Azure DB into another Azure SQL DB. The pipeline has CDC enabled dataflow. The tables that I am trying to copy/sync have the CDC enabled. I have a setup/metadatatable where I have columns - Sourceschema, sourcetablename, targetschema, targettablename, checkpoint , keycolumn and pipleine_name.

Inside the pipeline I am using a Lookup activity to query my setup table and get the columns into different parameters/variables. I have the checkpoint column as "Intex2_DataCopy_CLO_US_Collat_Stats_CK1" and I am passing this to the pipeline. I am not maintaining the lsn values anywhere, I guess Azure SQL CDC internally maintains it somewhere to identify which rows need to be synced.

I am using a similar setup for another client and the pipelines are working properly on a trigger to run daily and do the incremental changes.

The new pipeline worked fine when I triggered it manually the first time, the tables data got copied successfully into my sink database. However, when I trying to trigger it again manually, the pipeline is failing with the following error. All the dataflow activities within the pipeline failed with similar error. Please let me know what could potentially be causing this error to happen and how to fix it or avoid it in future -

"Job failed due to reason: at Source 'IntexSource': Failed while creating PreparedStatement due to org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0) (vm-52b34269 executor 1): shaded.msdataflow.com.microsoft.sqlserver.jdbc.SQLServerException: An insufficient number of arguments were supplied for the procedure or function cdc.fn_cdc_get_net_changes_ ... .

at shaded.msdataflow.com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:265)

at shaded.msdataflow.com.microsoft.sqlserver.jdbc.SQLServerResultSet$FetchBuffer.nextRow(SQLServerResultSet.java:5477)

at shaded.msdataflow.com.microsoft.sqlserver.jdbc.SQLServerResultSet.fetchBufferNext(SQLServerResultSet.java:1790)

at shaded.msdataflow.com.microsoft.sqlserver.jdbc.SQLServerResultSet.next(SQLServerResultSet.java:1048)

at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anon$1.getNext(JdbcUtils.scala:357)"

Smaran Thoomu 28,225 Reputation points Microsoft External Staff Moderator

2025-07-31T18:38:14.1633333+00:00
Hi Aaron Wong

Thanks for sharing the detailed setup and error message - it’s very helpful.

The error you're seeing:

"An insufficient number of arguments were supplied for the procedure or function cdc.fn_cdc_get_net_changes..."

usually indicates that the required parameters for the cdc.fn_cdc_get_net_changes_<capture_instance> function are not being passed correctly during the second run. This typically happens if:

Possible Causes:

Checkpoint (LSN) value is missing or invalid – The CDC function requires both @from_lsn and @to_lsn values. If the pipeline doesn't pass valid LSN values during re-run, the SQL engine throws this error.

First run doesn’t persist LSNs – If you're not maintaining or capturing the start_lsn from the previous run, the pipeline may not know where to resume.

Metadata table or Lookup doesn’t return values as expected – Ensure your checkpoint field is populated and that it's returning correct values to the pipeline parameters.

Capture instance missing or invalid – If the CDC capture instance was dropped or renamed, it would cause failures in resolution.

Suggested Steps:

Check the Lookup output – Validate that the checkpoint value (LSN or marker) is correctly returned and passed into your data flow.

Verify CDC metadata – Use SQL to run the function manually and ensure the capture instance exists and LSN values are valid:

SELECT * FROM cdc.fn_cdc_get_net_changes_<capture_instance>(@from_lsn, @to_lsn, 'all');

Persist the last successful LSN – Store the end_lsn of each successful run and pass it as the new start_lsn for the next run.

Initial full load special case – If it's the first run, consider using cdc.fn_cdc_get_all_changes_ instead.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the answer if it helps - that may also benefit other users with similar issues.
Smaran Thoomu 28,225 Reputation points Microsoft External Staff Moderator

2025-08-04T14:45:44.01+00:00

Aaron Wong We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Aaron Wong 40 Reputation points

2025-08-04T15:17:45.19+00:00

Smaran Thoomu - Unfortunately, I still haven't received a reason why ADF would fail to get the lsn values sometimes, while it works perfectly fine the rest of the time. As I mentioned, I am only maintaining a checkpoint key in my control table/metadata table for each table that I want to sync in the pipeline and I have verified that it is unique and is being read in the dataflow correctly. I also ran the query you provided manually and I can see the last lsn values being returned. I am not maintaining the lsn values in the control table. I believe ADF/Azure SQL DB is doing it automatically internally as the same kind of pipeline with the control table has been working fine for over a year. My question is why would ADF fail to retrieve the lsn values sometimes and how to fix the issue from happening again? I have attached a screenshot where you can see that only one table failed with the error while others are successful. Thanks!
Smaran Thoomu 28,225 Reputation points Microsoft External Staff Moderator

2025-08-04T20:05:50.07+00:00

Aaron Wong I kindly request you to provide the details mentioned in the private message so we can identify the root cause and address the issue effectively.
Smaran Thoomu 28,225 Reputation points Microsoft External Staff Moderator

2025-08-08T11:16:11.4366667+00:00

Aaron Wong Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Accepted answer

0 additional answers

Your answer

Smaran Thoomu 28,225 Reputation points Microsoft External Staff Moderator

2025-08-04T14:45:44.01+00:00

Aaron Wong We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Aaron Wong 40 Reputation points

2025-08-04T15:17:45.19+00:00

Smaran Thoomu - Unfortunately, I still haven't received a reason why ADF would fail to get the lsn values sometimes, while it works perfectly fine the rest of the time. As I mentioned, I am only maintaining a checkpoint key in my control table/metadata table for each table that I want to sync in the pipeline and I have verified that it is unique and is being read in the dataflow correctly. I also ran the query you provided manually and I can see the last lsn values being returned. I am not maintaining the lsn values in the control table. I believe ADF/Azure SQL DB is doing it automatically internally as the same kind of pipeline with the control table has been working fine for over a year. My question is why would ADF fail to retrieve the lsn values sometimes and how to fix the issue from happening again? I have attached a screenshot where you can see that only one table failed with the error while others are successful. Thanks!
Smaran Thoomu 28,225 Reputation points Microsoft External Staff Moderator

2025-08-04T20:05:50.07+00:00

Aaron Wong I kindly request you to provide the details mentioned in the private message so we can identify the root cause and address the issue effectively.
Smaran Thoomu 28,225 Reputation points Microsoft External Staff Moderator

2025-08-08T11:16:11.4366667+00:00

Aaron Wong Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

Hi Aaron Wong
We have internally reviewed the case and also connected with you to investigate further. Based on our discussion and testing, the issue was resolved by updating the checkpoint value used in your metadata/control table. After adjusting the checkpoint, the pipelines started running successfully without triggering the cdc.fn_cdc_get_net_changes error.

This indicates that the issue was likely due to the checkpoint pointing to an invalid or expired LSN range, which caused the CDC function to receive incomplete parameters during execution. Once the checkpoint was corrected, ADF was able to fetch the valid LSN window and resume CDC operations as expected.

Changing the checkpoint value fixed the issue, and the pipeline is now working consistently. The customer confirmed there are no further issues.

As a best practice, we recommend tracking and persisting the start_lsn and end_lsn values for each successful run, so that recovery is smoother in case of unexpected failures or state mismatches in future runs.

Hope this helps. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

ADF Pipeline failed with error "An insufficient number of arguments were supplied for the procedure or function cdc.fn_cdc_get_net_changes"

0 additional answers

Your answer