Debugging Kubernetes Workloads with kubectl

Something is broken in your cluster. A pod is not starting, a service is not reachable, or a deployment is stuck. Kubernetes provides extensive diagnostic information, but you need to know where to look. This guide walks through a systematic debugging workflow using kubectl.

The Debugging Workflow

Most Kubernetes issues follow a predictable pattern. Start broad and narrow down:

1. Check the resource status (is it running, pending, or failed?)
2. Read events (what happened recently?)
3. Examine logs (what did the application say?)
4. Inspect the environment (is the configuration correct?)
5. Test connectivity (can the pod reach its dependencies?)

Step 1: Check Resource Status

Start with an overview:

# Pods in a namespace
kubectl get pods -n myapp

# More detail with wide output
kubectl get pods -n myapp -o wide

# All resources in a namespace
kubectl get all -n myapp

Look at the STATUS and RESTARTS columns. Common statuses and what they mean:

Status	Meaning
Running	Container is executing
Pending	Pod is waiting for scheduling or image pull
CrashLoopBackOff	Container starts, crashes, restarts repeatedly
ImagePullBackOff	Cannot pull the container image
ErrImagePull	Image pull failed (first attempt)
CreateContainerConfigError	Bad config (missing secret, configmap, etc.)
Init:0/1	Init container has not completed
Terminating	Pod is shutting down

Step 2: Describe the Resource

kubectl describe shows the full story of a resource, including events:

kubectl describe pod myapp-7d9f8b6c5-x2k4n -n myapp

Focus on these sections:

Conditions tell you the pod’s current state:

Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True

Events show what happened in chronological order:

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  2m    default-scheduler  Successfully assigned myapp/myapp-xxx
  Normal   Pulling    2m    kubelet            Pulling image "myapp:v2"
  Warning  Failed     1m    kubelet            Failed to pull image "myapp:v2": not found
  Warning  BackOff    30s   kubelet            Back-off pulling image "myapp:v2"

This immediately shows the problem: the image tag v2 does not exist in the registry.

Step 3: Read Container Logs

# Current container logs
kubectl logs myapp-7d9f8b6c5-x2k4n -n myapp

# Previous container logs (after a crash)
kubectl logs myapp-7d9f8b6c5-x2k4n -n myapp --previous

# Follow logs in real time
kubectl logs -f myapp-7d9f8b6c5-x2k4n -n myapp

# Logs from a specific container in a multi-container pod
kubectl logs myapp-7d9f8b6c5-x2k4n -n myapp -c sidecar

# Last 50 lines
kubectl logs myapp-7d9f8b6c5-x2k4n -n myapp --tail=50

# Logs from all pods matching a label
kubectl logs -l app=myapp -n myapp --all-containers

The --previous flag is essential for CrashLoopBackOff. The current container has just started and has no useful output. The previous container has the crash logs.

Step 4: Exec into a Container

When logs are not enough, get a shell inside the container:

kubectl exec -it myapp-7d9f8b6c5-x2k4n -n myapp -- /bin/sh

If the container uses a minimal base image without a shell (like distroless), use an ephemeral debug container:

kubectl debug -it myapp-7d9f8b6c5-x2k4n -n myapp \
  --image=busybox:latest \
  --target=myapp

This attaches a busybox container that shares the process namespace with your application container. You can inspect the filesystem, check network connectivity, and run diagnostic tools without modifying the original image.

Step 5: Debug Cluster Events

View events across the entire namespace for a broader picture:

# All events in a namespace, sorted by time
kubectl get events -n myapp --sort-by='.lastTimestamp'

# Watch events in real time
kubectl get events -n myapp --watch

# Filter warning events
kubectl get events -n myapp --field-selector type=Warning

Events expire after about an hour by default. If you need longer retention, forward events to a logging system.

Debugging Common Failures

CrashLoopBackOff

The container starts, exits, and Kubernetes restarts it with increasing backoff delays.

# Check why it crashed
kubectl logs myapp-xxx -n myapp --previous

# Check exit code
kubectl describe pod myapp-xxx -n myapp | grep -A5 "Last State"

Common causes: application error on startup, missing environment variable, wrong command, database not reachable.

Pending Pods

The pod is not being scheduled.

kubectl describe pod myapp-xxx -n myapp | grep -A10 Events

Common causes:

• Insufficient resources: No node has enough CPU/memory. Check kubectl describe nodes for allocatable resources.
• Unmatched node selector or affinity: The pod requires a label no node has.
• Unmatched tolerations: The target nodes have taints the pod does not tolerate.
• PVC not bound: The pod uses a PVC that cannot be satisfied.

ImagePullBackOff

kubectl describe pod myapp-xxx -n myapp | grep -A5 "Warning.*pull"

Common causes: wrong image name/tag, private registry without imagePullSecret, registry rate limiting.

Fix for private registries:

kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=user \
  --docker-password=pass \
  -n myapp

Then add imagePullSecrets to the pod spec.

Debugging Services

When a service is not reachable:

# Check service exists and has endpoints
kubectl get svc myapp-svc -n myapp
kubectl get endpoints myapp-svc -n myapp

If endpoints show <none>, the service’s selector does not match any running pods. Compare labels:

# Service selector
kubectl get svc myapp-svc -n myapp -o jsonpath='{.spec.selector}'

# Pod labels
kubectl get pods -n myapp --show-labels

DNS Resolution

Test DNS from inside the cluster:

kubectl run dns-test --rm -it --image=busybox:latest --restart=Never -- \
  nslookup myapp-svc.myapp.svc.cluster.local

If DNS fails, check CoreDNS:

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns

Port Forwarding for Local Testing

Bypass the service and connect directly to a pod:

# Forward local port 8080 to pod port 8080
kubectl port-forward pod/myapp-xxx 8080:8080 -n myapp

# Forward through a service
kubectl port-forward svc/myapp-svc 8080:80 -n myapp

This helps determine whether the issue is in the application or in the service/ingress configuration.

Debugging Resource Pressure

Check node resource usage:

# Node-level resource usage (requires metrics-server)
kubectl top nodes

# Pod-level resource usage
kubectl top pods -n myapp --sort-by=memory

# Check resource requests vs allocatable on a node
kubectl describe node worker-1 | grep -A10 "Allocated resources"

If pods are being OOMKilled (out of memory), you will see OOMKilled in the container’s last termination reason:

kubectl describe pod myapp-xxx -n myapp | grep -i oom

Increase the memory limit or fix the memory leak in the application.

Useful Debugging Shortcuts

# Get pod YAML to inspect the full spec
kubectl get pod myapp-xxx -n myapp -o yaml

# JSONPath for specific fields
kubectl get pod myapp-xxx -n myapp -o jsonpath='{.status.containerStatuses[0].state}'

# Watch pods change state
kubectl get pods -n myapp --watch

# Delete a pod to trigger a fresh restart
kubectl delete pod myapp-xxx -n myapp

Practical Recommendations

Always start with kubectl describe and events before diving into logs. Use --previous for CrashLoopBackOff logs. For networking issues, check endpoints first since a service with no endpoints is the most common cause of unreachable services. Use ephemeral debug containers instead of adding debugging tools to production images. Set up metrics-server so kubectl top works for resource debugging. Practice these commands before an incident so they are muscle memory when it matters.