[Azure Cosmos DB][Multi-Write Setup] Does Cosmos DB SDK support cross region retries for transient error codes like 429/410

Question

[Azure Cosmos DB][Multi-Write Setup] Does Cosmos DB SDK support cross region retries for transient error codes like 429/410

Soumya Prateek Raul 70 Microsoft Employee

We have a multi-write region setup.

Wanted to know if Cosmos SDK inherently a cross region retries for transient error codes ?

Does the cross region retry kick-in after a single failure or multiple failures ?

Is the cross region retry configurable at the SDK level ?

Mahesh Kurva 6,850 Reputation points Microsoft External Staff Moderator

2025-07-28T22:46:50.4766667+00:00

Hi Soumya Prateek Raul,

Greetings!!

Yes, the Azure Cosmos DB SDK does support cross-region retries in a multi-write region setup, particularly for transient errors like HTTP 429 (Too Many Requests) and 410 (Gone). However, there are some key details and caveats to how and when these retries happen:

Wanted to know if Cosmos SDK inherently a cross region retries for transient error codes ?

Yes, the Cosmos DB SDK does support cross-region retries, particularly in multi-write region configurations. Transient error codes like:

429 (Too Many Requests)

410 (Gone – e.g., due to partition movement)

408, 503, and other transient issues

are handled through automatic retries within the SDK.

For more information, please refer the document: https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/conceptual-resilient-sdk-applications

Does the cross region retry kick-in after a single failure or multiple failures ?

Initial retries are attempted within the same region.

If failures persist (e.g., the region is degraded or unavailable), and the account has multiple write regions enabled, the SDK may retry the request in another write region.

This behavior leverages internal features like endpoint discovery, partition-level circuit breakers, and cross-region hedging (in newer SDKs like .NET v3+).

For more information, please refer the document: https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/troubleshoot-sdk-availability

Is the cross region retry configurable at the SDK level ?

Yes, SDKs allow configuration of:

Preferred regions: Controls retry order across regions.

Endpoint discovery: Enables the SDK to discover healthy regions dynamically.

Retry policies: Customize retry count, interval, and timeout behavior.

For more information, please refer the document: https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/performance-tips-dotnet-sdk-v3

I hope this information helps. Please do let us know if you have any further queries.
Soumya Prateek Raul 70 Reputation points Microsoft Employee

2025-07-29T04:48:00.31+00:00

Needed some more clarifications on the above answer. I needed some specific information so this can be implemented in production.

[ Ex - In your response you mention 408 error code is also retried, but in MSFT docs it's mentioned that 408 error code is not retried as these operations are idempotent].

[If failures persist (e.g., the region is degraded or unavailable), and the account has multiple write regions enabled, the SDK may retry the request in another write region] =>

Here, you mention SDK may retry the request**,** do we know decisively does it or does it not?

[ Initial retries are attempted within the same region. If failures persist (e.g., the region is degraded or unavailable), and the account has multiple write regions enabled, the SDK may retry the request in another write region ] =>

Do we know how many retries are attempted in same region before going cross region and any way to configure this count ?
Mahesh Kurva 6,850 Reputation points Microsoft External Staff Moderator

2025-07-30T16:20:49.8633333+00:00

Hi Soumya Prateek Raul,Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Accepted answer

0 additional answers

Your answer

Soumya Prateek Raul 70 Reputation points Microsoft Employee

2025-07-29T04:48:00.31+00:00

Needed some more clarifications on the above answer. I needed some specific information so this can be implemented in production.

[ Ex - In your response you mention 408 error code is also retried, but in MSFT docs it's mentioned that 408 error code is not retried as these operations are idempotent].

[If failures persist (e.g., the region is degraded or unavailable), and the account has multiple write regions enabled, the SDK may retry the request in another write region] =>

Here, you mention SDK may retry the request**,** do we know decisively does it or does it not?

[ Initial retries are attempted within the same region. If failures persist (e.g., the region is degraded or unavailable), and the account has multiple write regions enabled, the SDK may retry the request in another write region ] =>

Do we know how many retries are attempted in same region before going cross region and any way to configure this count ?
Mahesh Kurva 6,850 Reputation points Microsoft External Staff Moderator

2025-07-30T16:20:49.8633333+00:00

Hi Soumya Prateek Raul,Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

Mahesh Kurva 6,850 Microsoft External Staff Moderator

Hi Soumya Prateek Raul,

Thanks for the response.

In your response you mention 408 error code is also retried, but in MSFT docs it's mentioned that 408 error code is not retried as these operations are idempotent].

You're absolutely correct, 408 is not retried for write operations. This is because writes are not always idempotent and retrying them could lead to duplicates.

User's image

Reference: https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/conceptual-resilient-sdk-applications#timeouts-and-connectivity-related-failures-http-408503

The SDK retries within the current region first. If persistent failures occur (like the region being unhealthy), the SDK then fails over and retries in another write region, if configured. So, the cross-region retry behavior is conditional on the health of the current region and is an automatic part of the SDK’s design for multi-write setups.

Azure Cosmos DB SDKs in multiregional environments.

Do we know how many retries are attempted in same region before going cross region and any way to configure this count ?

The default max retry count for throttled requests is 9 retries with exponential backoff.

The SDK also uses a partition-level circuit breaker. By default, if a partition experiences 5 consecutive failures, the SDK marks it as unavailable and begins retrying in another region.

https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/performance-tips-dotnet-sdk-v3?tabs=trace-net-core#configuration-options

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Mahesh Kurva 6,850 Reputation points Microsoft External Staff Moderator

2025-07-31T17:00:56.82+00:00

Hi Soumya Prateek Raul,

Following up to see if the above answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Soumya Prateek Raul 70 Reputation points Microsoft Employee

2025-07-31T17:41:50.36+00:00

Yes, it does answer. Before we resolve the thread Just a follow-up, I read that in case of multi-write region one write region is always the hub (main region) and another write region is satellite. So a write is always confirmed when it has been written to hub region.

I wanted some clarification on this statement

a) if i write to satellite write region, does this mean, reads will start reflecting the write only after it has been written to hub region?

b) if i write to satellite and the data couldnt be replicated to hub due to issue with a particular partition , will the partition stop accepting any further writes ? what would be the behavior
Mahesh Kurva 6,850 Reputation points Microsoft External Staff Moderator

2025-08-01T05:01:09.8566667+00:00

Hi Soumya Prateek Raul,

If I write to a satellite write region, will reads reflect the write only after it is written to the hub region?

When you write to a satellite region, the data is saved locally right away. If you read from the same region, you will usually see the data immediately. But if you read from another region (like the main "hub" region), you might not see the data right away it has to be replicated and confirmed there first.

https://learn.microsoft.com/en-us/azure/cosmos-db/multi-region-writes?utm_source=chatgpt.com

If the satellite can't replicate to the hub due to a partition issue, will the partition stop accepting writes?

No, writes will still be accepted in the satellite region. The data is kept locally and marked as "tentative" until it can be sent to the hub. Once the issue is fixed, Cosmos DB will automatically catch up and sync the data. Your app might experience temporary delays in data being available across all regions.

https://learn.microsoft.com/en-us/azure/reliability/reliability-cosmos-db-nosql

I hope this information helps. Please do let us know if you have any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Mahesh Kurva 6,850 Reputation points Microsoft External Staff Moderator

2025-08-01T17:00:57.5733333+00:00

Hi Soumya Prateek Raul,

Following up to see if the above answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

[Azure Cosmos DB][Multi-Write Setup] Does Cosmos DB SDK support cross region retries for transient error codes like 429/410

0 additional answers

Your answer