Azure Kubernetes Service
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
5XX Errors During Rolling Updates in AKS with AGIC (App Gateway Ingress Controller)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 80%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYG%3C/text%3E%3C/svg%3E)
Yervand Galoyan
0
Reputation points
We are experiencing transient 5XX errors (502/503) during rolling deployments of services in our AKS cluster. The internal rollout behaves correctly, but end users receive errors externally during pod recreation.
Our AKS cluster uses AGIC (Azure Application Gateway Ingress Controller) to expose services via Azure Application Gateway.
During deployments using Kubernetes RollingUpdate strategy:
Pods terminate and new ones start successfully.
kubectl rollout status shows no issues.
Readiness and liveness probes are correctly configured.
However, external users receive 5XX errors (502/503) exactly when old pods are terminating and new ones are starting.
We suspect that App Gateway is routing traffic to terminating or not-yet-ready pods, possibly because AGIC has not yet removed or updated the backend pool members.
Steps to Reproduce:
Trigger a rolling update via kubectl apply or Helm.
Monitor pod status and application logs.
- Observe 502/503 errors from external clients during pod transitions. Detailed Description: Our AKS cluster uses AGIC (Azure Application Gateway Ingress Controller) to expose services via Azure Application Gateway. During deployments using Kubernetes
RollingUpdatestrategy:- Pods terminate and new ones start successfully.
-
kubectl rollout statusshows no issues. - Readiness and liveness probes are correctly configured.
- However, external users receive 5XX errors (502/503) exactly when old pods are terminating and new ones are starting.
Azure Kubernetes Service
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 64%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDR%3C/text%3E%3C/svg%3E)
Durga Reshma Malthi 9,355 Reputation points Microsoft External Staff Moderator2025-07-28T14:46:39.1733333+00:00 Could you please refer to this github document - https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/docs/how-tos/minimize-downtime-during-deployments.md
Consider increasing the
terminationGracePeriodSecondsin your pod specifications.terminationGracePeriodSeconds: 101Change the Pod and/or Deployment specs by adding preStop container life-cycle hooks, with a delay (sleep) of at least 90 seconds.
lifecycle: preStop: exec: command: ["sleep", "90"]Hope this helps!
Please Let me know if you have any queries.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 8%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHS%3C/text%3E%3C/svg%3E)
Himanshu Shekhar 160 Reputation points Microsoft External Staff Moderator2025-07-31T09:25:08.2+00:00 Hi Yervand Galoyan - Could you please provide an update on this post?
Kindly let us know if the suggested steps helps or you need further assistance on this issue.
-
Himanshu Shekhar 160 Reputation points Microsoft External Staff Moderator
2025-08-04T18:08:42.2166667+00:00 Hello Yervand Galoyan
I wanted to check if the comment provided helped. If you still need assistance, please let us know, we are here to help whenever you need support.
Sign in to comment
Sign in to answer