10250 I/O timeouts error when running kubectl log command

TCP timeouts can be caused by blockages of internal traffic that runs between nodes. To investigate TCP time-outs, verify that this traffic isn't being blocked, for example, by network security groups (NSGs) on the subnet for your cluster nodes.

Connect to the cluster

First, connect to your Azure Kubernetes Service (AKS) cluster by running the following command:

export RESOURCE_GROUP=<your-resource-group>
export CLUSTER_NAME=<your-cluster-name>

az aks get-credentials --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME

Symptoms

Tunnel functionalities, such as kubectl logs and code execution, work only for pods that are hosted on nodes on which tunnel service pods are deployed. Pods on other nodes that have no tunnel service pods cannot reach to the tunnel. When viewing the logs of these pods, you receive the following error message:

kubectl logs $POD_NAME

Results:

Error from server: Get "https://aks-agentpool-xxxxxxxxx-vmssxxxxxxxxx:10250/containerLogs/vsm-mba-prod/mba-api-app-xxxxxxxxxx/technosvc": dial tcp <IP-Address>:10250: i/o timeout

Solution

To resolve this issue, allow traffic on port 10250 as described in this article.

Contact us for help

If you have questions or need help, create a support request, or ask Azure community support. You can also submit product feedback to Azure feedback community.