Pods Not Being Evicted From AKS Cluster

Question

Pods Not Being Evicted From AKS Cluster

SEAN MCGURK 0

I have an AKS cluster that has pods scheduled on it by means of the following helmsman command:

helmsman --keep-untracked-releases --debug --target release-name -f ./state_definition.toml

Once this completes, the application is deployed successfully to the cluster and 2 new pods are created but the existing pods for the application are not evicted by the scheduler

kubernetes version 1.31.1

Can anyone suggest a good starting point for beginning to look at this problem?

5 answers

Your answer

Answer 1

Akram Kathimi 1,831 Microsoft Employee

Hi @SEAN MCGURK ,

Thank you for posting this question.

Start by verifying how the application is deployed. Check the workload type Inspect whether the application is managed by a Deployment, StatefulSet, or DaemonSet.
If you find multiple Deployments for the same app, the upgrade may have created a new Deployment instead of updating the existing one. This usually happens if the name or release labels changed.
If there’s only one Deployment, check the ReplicaSets. If you see multiple ReplicaSets with active pods, the old pods belong to an older ReplicaSet that hasn’t been scaled down. Check the events to see why the old pods are not being removed.

Please Accept the answer if the information helped you. This will help us and others in the community as well.

SEAN MCGURK 0

Thanks for your answer!

So the application is being deployed by means of a deployment

But the deployment is creating multiple replicasets, as can be seen below:

❯ kubectl get all | grep sso-svc
pod/sso-svc-deployment-6d548674b8-cbtxp       2/2     Running            0                   3m39s
pod/sso-svc-deployment-6d548674b8-qd8k8       2/2     Running            0                   3m58s
pod/sso-svc-deployment-74b8cb4ffb-csj6r       2/2     Running            0                   3m21s
pod/sso-svc-deployment-74b8cb4ffb-t7trx       2/2     Running            0                   3m1s
service/sso-svc-svc                           ClusterIP      10.x.xxx.xxx   <none>                                                 80/TCP                                                                                                                                   37h
deployment.apps/sso-svc-deployment                           2/2     2            2           19h
replicaset.apps/sso-svc-deployment-6d548674b8                           2         2         2       19h
replicaset.apps/sso-svc-deployment-74b8cb4ffb                           2         2         2       19h

Akram Kathimi 1,831 Microsoft Employee

can you please share the output of:

kubectl describe deployment.apps/sso-svc-deployment  
kubectl describe replicaset.apps/sso-svc-deployment-6d548674b8
kubectl describe replicaset.apps/sso-svc-deployment-74b8cb4ffb

SEAN MCGURK 0

@Akram Kathimi Thanks for your help so far!

kubectl describe deployment.apps/sso-svc-deployment:

Name:                   sso-svc-deployment
Namespace:              default
CreationTimestamp:      Tue, 12 Aug 2025 15:15:19 +0100
Labels:                 app.kubernetes.io/instance=sso-svc
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=sso-svc
                        app.kubernetes.io/part-of=sso-svc
                        app.kubernetes.io/version=0.6089.1-FTR.3
                        helm.sh/chart:sso-svc-0.6089.1-FTR.3
Annotations:            deployment.kubernetes.io/revision: 1
                        meta.helm.sh/release-name: sso-svc
                        meta.helm.sh/release-namespace: default
                        version: 0.6089.1-FTR.3
Selector:               app.kubernetes.io/instance=sso-svc,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=sso-svc,app.kubernetes.io/part-of=sso-svc
Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:       app.kubernetes.io/instance=sso-svc
                app.kubernetes.io/managed-by=Helm
                app.kubernetes.io/name=sso-svc
                app.kubernetes.io/part-of=sso-svc
                app.kubernetes.io/version=0.6089.1-FTR.3
                helm.sh/chart=sso-svc-0.6089.1-FTR.3
  Annotations:  checksum/config: aa81bc110554b0f5da0675e336ceb6a32b319eb385eb130ece1db65e1a040c32
                version: 0.6089.1-FTR.3
  Containers:
   sso-svc:
    Image:      sso-svc:0.6089.1-FTR.3
    Port:       8080/TCP
    Host Port:  0/TCP
    Limits:
      cpu:     1
      memory:  1G
    Requests:
      cpu:      100m
      memory:   400Mi
    Liveness:   http-get http://:8080/v1/svc-internal/checkz/ready delay=20s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:8080/v1/svc-internal/checkz/ready delay=10s timeout=10s period=30s #success=1 #failure=3
    Environment Variables from:
      sso-svc-sso-svc-config  ConfigMap  Optional: false
    Environment:
      LOG_DIR:              /data/logs/sso/svc
      SECRET_DIR:           /etc/secrets
      CONFIG_DIR:           /etc/config
      NODE_NAME:             (v1:spec.nodeName)
      POD_NAME:              (v1:metadata.name)
      POD_NAMESPACE:         (v1:metadata.namespace)
      POD_IP:                (v1:status.podIP)
      POD_SERVICE_ACCOUNT:   (v1:spec.serviceAccountName)
    Mounts:
      /data/logs/sso/svc from log-volume (rw)
      /etc/config/appsettings.Production.json from config-volume (ro,path="appsettings.Production.json")
      /etc/config/psdk-environment-settings.json from psdk-config-volume (ro,path="psdk-environment-settings.json")
      /etc/secrets from secret-config-volume (ro)
  Volumes:
   config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sso-svc-config
    Optional:  false
   configspa-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sso-svcspa-config
    Optional:  false
   psdk-config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      psdk-platformsdk-config
    Optional:  true
   log-volume:
    Type:          HostPath (bare host directory volume)
    Path:          /data/logs/sso/svc
    HostPathType:
   secret-config-volume:
    Type:          Secret (a volume populated by a Secret)
    SecretName:    sso-svc-secret
    Optional:      false
  Node-Selectors:  <none>
  Tolerations:     <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   sso-svc-deployment-74b8cb4ffb (2/2 replicas created)
Events:          <none>

kubectl describe replicaset.apps/sso-svc-deployment-6d548674b8:

Name:           sso-svc-deployment-6d548674b8
Namespace:      default
Selector:       app.kubernetes.io/instance=sso-svc,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=sso-svc,app.kubernetes.io/part-of=sso-svc,pod-template-hash=6d548674b8
Labels:         app.kubernetes.io/instance=sso-svc
                app.kubernetes.io/managed-by=Helm
                app.kubernetes.io/name=sso-svc
                app.kubernetes.io/part-of=sso-svc
                app.kubernetes.io/version=0.6089.1-FTR.3
                helm.sh/chart=sso-svc-0.6089.1-FTR.3
                pod-template-hash=6d548674b8
Annotations:    deployment.kubernetes.io/desired-replicas: 2
                deployment.kubernetes.io/max-replicas: 3
                deployment.kubernetes.io/revision: 1
                meta.helm.sh/release-name: sso-svc
                meta.helm.sh/release-namespace: default
                version: 0.6089.1-FTR.3
Controlled By:  Deployment/sso-svc-deployment
Replicas:       2 current / 2 desired
Pods Status:    2 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:       app.kubernetes.io/instance=sso-svc
                app.kubernetes.io/managed-by=Helm
                app.kubernetes.io/name=sso-svc
                app.kubernetes.io/part-of=sso-svc
                app.kubernetes.io/version=0.6089.1-FTR.3
                helm.sh/chart=sso-svc-0.6089.1-FTR.3
                pod-template-hash=6d548674b8
  Annotations:  checksum/config: 959e91d3d72d60f81da19115e5f1307cb4f92f8d304eb09df907118cb2ffd9d1
                version: 0.6089.1-FTR.3
  Containers:
   sso-svc:
    Image:      sso-svc:0.6089.1-FTR.3
    Port:       8080/TCP
    Host Port:  0/TCP
    Limits:
      cpu:     1
      memory:  1G
    Requests:
      cpu:      100m
      memory:   400Mi
    Liveness:   http-get http://:8080/v1/sso-svc-internal/checkz/ready delay=20s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:8080/v1/sso-svc-internal/checkz/ready delay=10s timeout=10s period=30s #success=1 #failure=3
    Environment Variables from:
      sso-svc-config  ConfigMap  Optional: false
    Environment:
      LOG_DIR:              /data/logs/sso/svc
      SECRET_DIR:           /etc/secrets
      CONFIG_DIR:           /etc/config
      NODE_NAME:             (v1:spec.nodeName)
      POD_NAME:              (v1:metadata.name)
      POD_NAMESPACE:         (v1:metadata.namespace)
      POD_IP:                (v1:status.podIP)
      POD_SERVICE_ACCOUNT:   (v1:spec.serviceAccountName)
    Mounts:
      /data/logs/sso/svc from log-volume (rw)
      /etc/config/appsettings.Production.json from config-volume (ro,path="appsettings.Production.json")
      /etc/config/psdk-environment-settings.json from psdk-config-volume (ro,path="psdk-environment-settings.json")
      /etc/secrets from secret-config-volume (ro)
  Volumes:
   config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sso-svc-config
    Optional:  false
   configspa-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sso-svcspa-config
    Optional:  false
   psdk-config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      psdk-platformsdk-config
    Optional:  true
   log-volume:
    Type:          HostPath (bare host directory volume)
    Path:          /data/logs/sso/svc
    HostPathType:
   secret-config-volume:
    Type:          Secret (a volume populated by a Secret)
    SecretName:    sso-svc-secret
    Optional:      false
  Node-Selectors:  <none>
  Tolerations:     <none>
Events:
  Type    Reason            Age   From                   Message
  ----    ------            ----  ----                   -------
  Normal  SuccessfulCreate  34m   replicaset-controller  Created pod: sso-svc-deployment-6d548674b8-qd8k8
  Normal  SuccessfulCreate  34m   replicaset-controller  Created pod: sso-svc-deployment-6d548674b8-cbtxp

kubectl describe replicaset.apps/sso-svc-deployment-74b8cb4ffb:

Name:           sso-svc-deployment-74b8cb4ffb
Namespace:      default
Selector:       app.kubernetes.io/instance=sso-svc,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=sso-svc,app.kubernetes.io/part-of=sso-svc,pod-template-hash=74b8cb4ffb
Labels:         app.kubernetes.io/instance=sso-svc
                app.kubernetes.io/managed-by=Helm
                app.kubernetes.io/name=sso-svc
                app.kubernetes.io/part-of=sso-svc
                app.kubernetes.io/version=0.6089.1-FTR.3
                helm.sh/chart=sso-svc-0.6089.1-FTR.3
                pod-template-hash=74b8cb4ffb
Annotations:    deployment.kubernetes.io/desired-replicas: 2
                deployment.kubernetes.io/max-replicas: 3
                deployment.kubernetes.io/revision: 1
                meta.helm.sh/release-name: sso-svc
                meta.helm.sh/release-namespace: default
                version: 0.6089.1-FTR.3
Controlled By:  Deployment/sso-svc-deployment
Replicas:       2 current / 2 desired
Pods Status:    2 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:       app.kubernetes.io/instance=sso-svc
                app.kubernetes.io/managed-by=Helm
                app.kubernetes.io/name=sso-svc
                app.kubernetes.io/part-of=sso-svc
                app.kubernetes.io/version=0.6089.1-FTR.3
                helm.sh/chart=sso-svc-0.6089.1-FTR.3
                pod-template-hash=74b8cb4ffb
  Annotations:  checksum/config: aa81bc110554b0f5da0675e336ceb6a32b319eb385eb130ece1db65e1a040c32
                version: 0.6089.1-FTR.3
  Containers:
   sso-svc:
    Image:      sso-svc:0.6089.1-FTR.3
    Port:       8080/TCP
    Host Port:  0/TCP
    Limits:
      cpu:     1
      memory:  1G
    Requests:
      cpu:      100m
      memory:   400Mi
    Liveness:   http-get http://:8080/v1/sso-svc-internal/checkz/ready delay=20s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:8080/v1/sso-svc-internal/checkz/ready delay=10s timeout=10s period=30s #success=1 #failure=3
    Environment Variables from:
      sso-svc-config  ConfigMap  Optional: false
    Environment:
      LOG_DIR:              /data/logs/sso/svc
      SECRET_DIR:           /etc/secrets
      CONFIG_DIR:           /etc/config
      NODE_NAME:             (v1:spec.nodeName)
      POD_NAME:              (v1:metadata.name)
      POD_NAMESPACE:         (v1:metadata.namespace)
      POD_IP:                (v1:status.podIP)
      POD_SERVICE_ACCOUNT:   (v1:spec.serviceAccountName)
    Mounts:
      /data/logs/sso/svc from log-volume (rw)
      /etc/config/appsettings.Production.json from config-volume (ro,path="appsettings.Production.json")
      /etc/config/psdk-environment-settings.json from psdk-config-volume (ro,path="psdk-environment-settings.json")
      /etc/secrets from secret-config-volume (ro)
  Volumes:
   config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sso-svc-config
    Optional:  false
   configspa-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sso-svcspa-config
    Optional:  false
   psdk-config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      psdk-platformsdk-config
    Optional:  true
   log-volume:
    Type:          HostPath (bare host directory volume)
    Path:          /data/logs/sso/svc
    HostPathType:
   secret-config-volume:
    Type:          Secret (a volume populated by a Secret)
    SecretName:   sso-svc-secret
    Optional:      false
  Node-Selectors:  <none>
  Tolerations:     <none>
Events:
  Type    Reason            Age   From                   Message
  ----    ------            ----  ----                   -------
  Normal  SuccessfulCreate  52m   replicaset-controller  Created pod: sso-svc-deployment-74b8cb4ffb-csj6r
  Normal  SuccessfulCreate  52m   replicaset-controller  Created pod: sso-svc-deployment-74b8cb4ffb-t7trx

Answer 2

Hi @SEAN MCGURK ,

From the describe deplyment output, we can see that the deployemnt does recognize the older replicaset as one of its own:

Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   sso-svc-deployment-74b8cb4ffb (2/2 replicas created)
Events:          <none>

The same thing is seen when checking the annotations: deployment.kubernetes.io/revision: 1

This indicates that the deployment is only on its 1st revision, and didnt get any updates.

It looks like the older replicaset belongs to an older deployment with the same name.

To resolve the issue, delete the older replicaset sso-svc-deployment-6d548674b8.

After that, try to redeploy the helm chart to see if the same issue occurs again, if it does, you would need to troubleshoot why this behavior is happening.

Answer 3

So I have:

Deleted the replicaset(s)
Re-deployed the application (re-created the deployment)

And the second replicaset has now gone away but more pods have been added

If I describe one of the (old) pods I get:

Name:             sso-svc-deployment-6d548674b8-qd8k8


...


Status:           Running
IP:               10.xxx.xxx.xx 
IPs:   
  IP:           10.xxx.xxx.xx 
Controlled By:  ReplicaSet/sso-svc-deployment-6d548674b8

But this replicaset no longer exists on the cluster - it has been replaced by a newer replicaset so these look like 'orphaned' pods that are still running on the cluster

Answer 4

Akram Kathimi 1,831 Microsoft Employee

@SEAN MCGURK this looks like an issue that occurs when pods are not being terminated. Checking the deployment, this might be caused because of the host path mounted volume.

Is the issue occurring with other deployments in the cluster? is it possible to remove this mount from the deployment as a test?

btw, to force delete the pod you can use kubectl delete pod <> -n <> --force --grace-period=0

Answer 5

SEAN MCGURK 0

It seems like this is a known issue with Helm/Kubernetes.

See: https://github.com/helm/helm/issues/2096

The solution is therefore modify your deployment so that the chart name and chart version are no longer included as a label on the deployment

Share via

Pods Not Being Evicted From AKS Cluster

5 answers

Your answer