How to resubmit an existing pipeline job for retraining by REST API?.

Question

How to resubmit an existing pipeline job for retraining by REST API?.

Zhuoling Li 41 Microsoft Employee

Not sure if this is an appropriate ask. We are trying to create a training pipeline job in use of az cli and then trigger it by ADF (Azure Data Factory) for retraining.

The reason why we are using ADF is that we want to schedule to preprocess data before retraining. It seems I can interact with pipeline by only REST API. Otherwise, I need to publish pipeline (which is v1 and I don't want to use) to use "Machine Learning Execute Pipeline".

So here come three questions.

Is REST API indeed the only choise for me to interact with Machine Learning pipeline through ADF?
What's the best practice to perform retraining? Do I need to create a pipeline component even though it is no need to be shared?
How to resubmit a pipeline job by REST API? I found this https://learn.microsoft.com/en-us/rest/api/azureml/jobs/create-or-update?view=rest-azureml-2025-06-01&tabs=HTTP#pipelinejob but only got error "JobSubmissionFailure: Failed to submit job due to Invalid pipeline job since step jobs does not exist." My url and payload is like shown bellow.
PUT https://management.azure.com/subscriptions/xxx/resourcegroups/xxx/providers/Microsoft.MachineLearningServices/workspaces/xxx/jobs/bruno_test_3?api-version=2025-06-01
```
   {
     "properties": {
       "displayName": "bruno_test",
       "jobType": "Pipeline",
       "sourceJobId": "/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.MachineLearningServices/workspaces/xxx/jobs/ashy_atemoya_618d7hj06l"
     }
   }
```

Accepted answer

0 additional answers

Your answer

Answer 1

Hi Zhuoling Li,
Thank you for your questions. Let me address your points one by one regarding resubmitting pipeline jobs for retraining via REST API and integration with Azure Data Factory (ADF):

Is REST API the only way to interact with Azure Machine Learning pipelines from ADF?

Yes, if you are using Azure Machine Learning v2 pipelines (which do not require publishing), the only supported way to trigger them from Azure Data Factory (ADF) is by using the Azure ML REST API or a custom Azure Function/Logic App. The built-in ADF activity “Machine Learning Execute Pipeline” only supports v1 published pipelines and is not compatible with v2. While the Azure ML SDK and CLI are useful for local development and CI/CD workflows, they cannot be invoked directly from ADF. Therefore, using the REST API or creating a callable endpoint (e.g., via Azure Functions) is the recommended approach when integrating ADF with v2 pipelines for dynamic orchestration and scheduling.

What is the best practice for retraining and should a pipeline component be created even if it’s not shared?
Yes, it is considered a best practice to encapsulate your retraining logic inside a pipeline component even if you don’t intend to share or reuse it. Pipeline components provide a structured, trackable, and auditable framework for managing machine learning workflows. This is especially important for retraining scenarios, as it improves maintainability, supports better monitoring and lineage tracking, and allows for seamless integration with orchestrators like ADF or Azure ML schedules. Even for single-use or internal components, defining them explicitly helps maintain a clean and scalable architecture.

How to resubmit a pipeline job by REST API?

To rerun a previous pipeline job using the Azure ML REST API, using sourceJobId alone is not sufficient. The error (e.g., "Invalid pipeline job since step jobs do not exist") typically means the referenced job no longer contains its full step definitions. The sourceJobId field does not clone the original job definition. Instead, you must retrieve the full job payload (including all step configurations) from the original run, modify it as needed (such as updating input data or parameters), and submit it as a new job via the REST API.

For reference: Data ingestion with Azure Data Factory
Tutorial: Create production machine learning pipelines

Share via

How to resubmit an existing pipeline job for retraining by REST API?.

0 additional answers

Your answer