Automatically cancelling a copy activity when no data is being read.

hi aaron! thanks for posting this tricky question ))

so u wanna stop copy activities when they're stuck but keep them running if they're actually working... got it! here's how we can tackle this with some azure magic )

u can use a combination of azure functions and data factory's monitoring api. set up a function that checks the copy activity's progress while it runs. the function can peek at metrics like 'rows read' and 'data read' through the api. if nothing moves for X minutes, boom – it cancels the activity https://docs.microsoft.com/en-us/rest/api/datafactory/pipeline-runs/get

and here's a little code snippet to get u started

var response = await client.PipelineRuns.GetWithHttpMessagesAsync( resourceGroupName, factoryName, runId); var metrics = response.Body.RunOutput;

worth looking into azure monitor alerts too. u can create custom alerts based on activity metrics. if 'rows read' stays at zero for too long, trigger an action ))

sometimes the issue might be with the data source itself. maybe its throttling connections or just being slow. u could add a pre-copy check that tests the connection speed first.

when dealing with flaky data sources, always have a plan B. maybe add retry logic with exponential backoff. the azure sdk makes this pretty easy to implement )

and dont forget – u can use data factory's built-in retry settings. tweak those before going full custom solution. sometimes simple fixes work best %)

let me know if u hit any snags

Alex

and "yes" if you would follow me at Q&A - personaly thx.
P.S. If my answer help to you, please Accept my answer

https://ctrlaltdel.blog/

hi aaron ))

so u got this tricky situation where ur copy activity just hangs there like a lazy cat, no data moving, just time to first byte creeping up... ugh, frustrating )) good news is, we can fix this with some clever tricks, for azure data factory specifically u can use a pipeline watchdog approach.

1. add a until loop right after ur copy activity. inside it, place a web activity that checks the copy activity's status using the ADF monitoring API.

make the web activity call this endpoint https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelineruns/{runId}/queryActivityruns?api-version=2018-06-01. yeah, long url, but it works.

parse the response to check rowsRead and timeToFirstByte. if no rows read and time keeps climbing, u can bail out early.

quick json snippet for the web activity body

{ "lastUpdatedAfter": "@{addminutes(utcnow(), -5)}", "lastUpdatedBefore": "@{utcnow()}" }

full docs here https://docs.microsoft.com/en-us/rest/api/datafactory/activity-runs/query-by-pipeline-run

ever tried azure monitor alerts? u can set metrics to trigger when data flow stalls. not perfect, but better than nothing.

logic apps can monitor this, but yeah, only after completion... sad. still, worth keeping in mind for other workflows.

if u use custom logging, u could track bytes read per minute. then auto-kill if it flatlines. bit hacky, but hey, it works ))

if u wanna get fancy, u could wrap this whole thing in an azure function. it'd poll the ADF api every minute, then force-cancel if nothing moves. overkill? maybe. cool? absolutely :))

let me know if u hit snags

Alex

and "yes" if you would follow me at Q&A - personaly thx.
P.S. If my answer help to you, please Accept my answer

https://ctrlaltdel.blog/