Significant drop in RPS after migrating to Azure Application Gateway V2 – Potential MS backbone issue?

Hi @Niket Kumar Singh

It sounds like you’re facing quite a challenge after migrating your workloads from GCP to Azure using the Application Gateway Standard V2. Dropping from ~150K RPS to around 40K-50K RPS is definitely concerning. Here are some steps and checks you can take or consider:

SNAT Port Limitations: The SNAT port limits can significantly affect the number of concurrent connections to your backend. Make sure you're not hitting this limit:

• If using public IPs for the backend, each requires a separate SNAT port.
• To mitigate this, you can increase the number of Application Gateway instances, scale out your backends (more IPs), or consider moving backends into the same virtual network using private IPs.

If Application Gateway reaches the SNAT port limit, it affects the requests per second (RPS). For example, Application Gateway can't open a new connection to the back end, and the request fails.

Reference: Architecture best practices for Azure Application Gateway v2

How many backend VMs are actually configured in your Application Gateway backend pool? If you have, say, 50 VMs, but the healthy host count consistently shows 6, then only 6 VMs are truly being utilized or perceived as healthy for routing, which would drastically limit your RPS even if the AGW itself has capacity.

1. Is your backend pool configured correctly to include all intended VMs? Double-check the backend pool settings in the Azure portal or via PowerShell/CLI.
2. Are there any NSGs or UDRs on the backend VMs themselves that might be preventing the Application Gateway from establishing connections to all of them, even if the health probes are succeeding?

Backend Pool HTTP Settings:

Keep-Alive: Ensure HTTP Keep-Alive is enabled and configured appropriately in your backend application and Application Gateway HTTP settings. This reduces the overhead of establishing new connections for every request.

Request Timeout: Check the "Request timeout (seconds)" setting in your HTTP settings. If it's too low, it could lead to 499 errors if the backend is slow. If it's too high, it could keep connections open unnecessarily.

Host Header: Is the host header being correctly passed to the backend? Misconfigurations here can lead to issues.

Application Gateway Capacity: Make sure that the capacity is optimized. You mentioned 125 instances, but it’s worth scaling beyond this if you observe high loads or consider using larger instance sizes if applicable.

Traffic Distribution: Check if the traffic distribution is properly configured. Make sure that your DNS records are updated and not still pointing to the GCP load balancer. While you think DNS caching isn't likely, make sure that any clients using static DNS data refresh appropriately.

Investigate Front Door and Routing: Since you suspect an Azure network or routing issue, validate:

• Azure's service health to see if there are ongoing issues.
• If you are using Azure Front Door or have any specific routing rules that might be inhibiting traffic flow to your Application Gateway.

Monitor Logging and Analytics: Continue to leverage Azure Monitor and KQL queries to gain insights. Pay particular attention to network-related logs that might provide hints of packet drops or connectivity issues.

Azure Network Watcher:

• Connection Troubleshoot: Use Azure Network Watcher's Connection Troubleshoot to test connectivity from a source VM (e.g., a test VM in a different VNet, or even from one of your backend VMs to the Application Gateway's frontend IP, if it allows it for diagnostic purposes).
• IP Flow Verify: Verify if traffic is allowed to and from the Application Gateway's public IP.
• Next Hop: Use Next Hop to see the next hop for traffic destined to your AGW's public IP from a test VM.

Action: Set up Network Watcher in your region and use its diagnostic tools.

Kindly let us know if the above helps or if you need any further assistance on the issue.

Hi Sai Prasanna Sinde

I’d like to provide additional information based on our setup and investigations so far:

Backend Configuration: We have verified that all backend pool targets are using private IP addresses in the 10.x.x.x range So, SNAT port limitations should not be a factor in our case.

We are using manual scaling, and our Application Gateway Standard V2 is configured at the maximum instance count of 125.

Despite having 125 instances and a large backend pool, we are seeing a drastic drop in RPS — from ~150K RPS on GCP to ~40K-50K RPS on Azure.

We have checked and confirmed DNS resolution, backend health probes, request/connection metrics, and no hits are being seen on the old GCP load balancer, so DNS caching or client misrouting is unlikely.

Could you please guide us on what additional checks we can perform? Does this scenario point towards any known limitations or potential issues on the Azure network side that could explain this behavior?

@Niket Kumar Singh

Thank you for reaching out.

Based on your question above as RPS dropped 40K-50K I wonder if you are hitting the limitation documented here for Application Gateway V2 : Application Gateway V2 only allows 62500 Max. connections per second (Estimated based on using an RSA 2048-bit key TLS certificate.)

Does the 150K requests represents new connections to the Application Gateway?
If your traffic requirement needs more than 125 instances, you can use Azure Traffic Manager or Azure Front Door in front of your Application Gateway. For more information, please see Connect Azure Front Door Premium to an Azure Application Gateway with Private Link and **Use Azure App Gateway with Azure Traffic Manager.
You can also refer this Load-balancing options guide for best practices in Azure https://learn.microsoft.com/en-us/azure/architecture/guide/technology-choices/load-balancing-overview

Please let me know if you have any questions or concerns. Thank you!

Hi ChaitanyaNaykodi-MSFT

Thank you for your detailed response and the guidance provided.

We would like to seek further clarification on the Application Gateway V2 limitation you mentioned regarding 62500 max connections per second (based on RSA 2048-bit TLS certificate).

Our specific question: Does this limit of ~62.5K connections per second apply per Application Gateway deployment regardless of how many instances we provision (e.g., 125 instances)? In other words, even though we have scaled our Application Gateway Standard V2 to the maximum instance count (125 instances in our case), does the total connection handling capacity remain capped at 62.5K new connections per second for the entire deployment?

If so, could you confirm if the only way to handle beyond this limit would be to:

• Deploy multiple Application Gateways behind Azure Front Door or Traffic Manager
• Or consider alternative architectures suitable for this scale

We would like to confirm the maximum concurrent connections that can be handled:

Per instance

Per deployment

As per our understanding (based on documentation and Capacity Unit description):

Each compute unit supports approximately a single Capacity Unit consists of the following parameters:

2500 Persistent connections, 2.22-Mbps throughput, 1 Compute Unit

Since we have manually scaled our Application Gateway Standard V2 to 125 instances, we assume:

Max concurrent connections = 2,500 × 125 = 312,500 persistent connections

Could you please validate if this interpretation is correct

We appreciate your assistance in helping us understand this limit clearly so we can plan our next steps accordingly.

@Niket Kumar Singh

Thank you for getting back and apologies for the delay.>

does the total connection handling capacity remain capped at 62.5K new connections per second for the entire deployment?

Yes, this is correct.
Connections per second per compute unit is 50 for Standard V2. One application gateway instance can handle a minimum of 10 compute units (Currently documented here). For 125 instances
125x10x50 = 62500 Max. connections per second.

If so, could you confirm if the only way to handle beyond this limit would be to:

• Deploy multiple Application Gateways behind Azure Front Door or Traffic Manager
• Or consider alternative architectures suitable for this scale

Yes, I think using Front Door Or Traffic manager will be recommended. For Azure Front Door Maximum requests per second per profile is 100,000 although you can increase this limit by creating a quota support request. You can find more details here (There's currently a 5,000 requests per second per POP limit for each Front Door profile. Beyond this limit, the POP location will drop connections. If requests are concentrated in one of more regions and exceed this limit, you can request a higher POP limit by submitting an Azure support request.)

Max concurrent connections = 2,500 × 125 = 312,500 persistent connections Could you please validate if this interpretation is correct

This will be higher as one application gateway instance can handle a minimum of 10 compute units.
This value will 2500 x 10 x 125 = 312,5000 Persistent new connections for 125 instances.

Apologies for the delay once again and hope this helps. Please let me know if you have any questions. Thank you!