Kubernetes and Azure Kubernetes Service

Kubernetes manages containerized workloads by continuously trying to make actual cluster state match the state declared in Kubernetes objects. Azure Kubernetes Service (AKS) provides a managed Kubernetes control plane and Azure integrations, while customers remain responsible for workloads, access, configuration, networking choices, and operational readiness.

Who this guide is for

Use this guide to understand:

• Kubernetes architecture and core objects.
• Application deployment and service exposure.
• Configuration, secrets, health probes, and resource controls.
• AKS access and day-two operations.
• Security and production practices.
• Common pod, networking, scheduling, and rollout failures.

Architecture

Control plane

The control plane exposes the Kubernetes API, stores desired state, schedules pods, and runs controllers that reconcile resources. In AKS, Microsoft manages the control plane service, but you still manage cluster configuration and application objects.

Node

A node is a worker machine that runs pods. The kubelet communicates with the API server and makes sure assigned containers run. A container runtime starts and stops containers, while the network implementation connects workloads.

Pod

A pod is the smallest deployable Kubernetes unit. One pod usually contains one main application container, although tightly coupled helper containers may share the pod.

Pods are replaceable. Do not rely on an individual pod name, IP address, or local filesystem for durable application state.

Deployment

A Deployment manages stateless replicas and rolling updates. It creates ReplicaSets, which in turn maintain the requested number of pods.

Service

A Service gives a stable network identity to a changing set of pods selected by labels. Common types include ClusterIP, NodePort, and LoadBalancer.

Ingress and Gateway

Ingress or Gateway API resources define HTTP routing, but they require a compatible controller. Creating an Ingress object alone does not create a working ingress implementation.

Install kubectl and connect to AKS

Install Azure CLI and kubectl, authenticate, and select the correct subscription:

az login
az account set --subscription <subscription-id>
az account show --output table

Retrieve user credentials for a cluster:

az aks get-credentials \
  --resource-group <resource-group> \
  --name <aks-cluster>

Verify access:

kubectl cluster-info
kubectl get nodes
kubectl auth can-i get pods --all-namespaces

Avoid using administrative credentials for routine work. Use Microsoft Entra integration, Azure RBAC, or Kubernetes RBAC with least privilege.

Deploy an application

Deployment manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudforge-api
  labels:
    app.kubernetes.io/name: cloudforge-api
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: cloudforge-api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: cloudforge-api
    spec:
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: api
          image: <registry>/cloudforge-api:<immutable-tag>
          ports:
            - name: http
              containerPort: 8000
          envFrom:
            - configMapRef:
                name: cloudforge-api-config
          readinessProbe:
            httpGet:
              path: /health/ready
              port: http
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health/live
              port: http
            initialDelaySeconds: 20
            periodSeconds: 20
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 512Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            readOnlyRootFilesystem: true

Use an immutable image tag or digest so a deployment can be audited and rolled back. Make sure the application supports a read-only root filesystem before enabling that control.

Internal Service

apiVersion: v1
kind: Service
metadata:
  name: cloudforge-api
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: cloudforge-api
  ports:
    - name: http
      port: 80
      targetPort: http

Apply and verify:

kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
kubectl rollout status deployment/cloudforge-api --timeout=180s
kubectl get pods,service -l app.kubernetes.io/name=cloudforge-api

Configuration and secrets

Use a ConfigMap for non-sensitive settings:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cloudforge-api-config
data:
  LOG_LEVEL: "INFO"
  ENVIRONMENT: "production"

A Kubernetes Secret is encoded, not automatically encrypted in every context. Protect access with RBAC, enable platform encryption controls, and prefer integration with an external secret store such as Azure Key Vault when appropriate.

Never commit real secrets to Git, manifests, container images, or public documentation.

Health probes

Readiness probe

Readiness determines whether the pod should receive Service traffic. Use it for dependencies required to serve requests, but avoid fragile checks that make every pod unavailable during a temporary downstream issue.

Liveness probe

Liveness determines whether Kubernetes should restart a container. It must detect an unrecoverable process condition, not general dependency failure.

Startup probe

Use a startup probe for applications with slow or unpredictable initialization. Until it succeeds, Kubernetes delays liveness and readiness evaluation.

Poor probe design can cause cascading restarts or route traffic to an application before it is ready.

Resource requests and limits

The scheduler uses requests to place pods. CPU and memory limits constrain consumption. Missing requests can produce unpredictable scheduling; unrealistic limits can cause throttling or out-of-memory termination.

Start with measurements from load testing and production telemetry, then adjust. Avoid copying arbitrary values between different applications.

Rollouts and rollback

Inspect a rollout:

kubectl rollout status deployment/cloudforge-api
kubectl rollout history deployment/cloudforge-api

Undo the most recent Deployment rollout:

kubectl rollout undo deployment/cloudforge-api

Rollback restores the previous pod template; it does not automatically reverse database migrations or external changes. Design database releases for backward compatibility.

Scaling

Manual scaling:

kubectl scale deployment cloudforge-api --replicas=4

Horizontal autoscaling requires resource metrics and correctly configured requests. Validate that the application itself can scale horizontally and does not depend on local session state.

AKS cluster autoscaling changes node capacity; a HorizontalPodAutoscaler changes workload replicas. They solve different problems and may be used together.

Security baseline

• Use Entra integration and least-privilege RBAC.
• Separate workloads with namespaces where that boundary is useful.
• Restrict pod-to-pod traffic with NetworkPolicy when supported by the network implementation.
• Run containers as non-root.
• Drop Linux capabilities that are not required.
• Prevent privilege escalation.
• Use trusted, scanned, minimal images.
• Pin production images to immutable versions or digests.
• Keep Kubernetes and node images within supported versions.
• Restrict public API server access where possible.
• Use workload identity instead of embedding cloud credentials.
• Apply Pod Security standards or equivalent admission policy.
• Back up application data and required cluster configuration.

Essential diagnostic workflow

Start broad, then narrow the failure boundary:

kubectl get pods -n <namespace> -o wide
kubectl get deployment,replicaset,service,endpoints -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --all-containers
kubectl logs <pod> -n <namespace> --previous

Check desired replicas, pod status, restarts, node placement, warning events, container logs, and previous container logs. Do not begin by deleting the failing pod; deletion may destroy evidence while the controller simply creates another identical failure.

Troubleshooting common states

CrashLoopBackOff

The container repeatedly starts and exits. Investigate:

1. Current and previous container logs.
2. Exit code and termination reason in kubectl describe pod.
3. Missing configuration or secrets.
4. Application startup command.
5. Dependency or certificate failures.
6. Liveness probe failures.
7. Memory limits and OOMKilled status.

ImagePullBackOff

Check the image name and tag, registry reachability, AKS-to-ACR permissions, image pull secrets, private DNS, and firewall rules.

kubectl describe pod <pod> -n <namespace>
az acr repository show-tags --name <registry-name> --repository <repository> --output table

Pending

Read scheduling events:

kubectl describe pod <pod> -n <namespace>
kubectl get nodes
kubectl top nodes

Common causes include insufficient CPU or memory, incompatible node selectors, taints without tolerations, unavailable persistent volumes, quota limits, and cluster autoscaler constraints.

OOMKilled

The container exceeded its memory limit or the node experienced memory pressure. Confirm the termination reason, compare working-set memory with the limit, investigate leaks or spikes, and set a measured request and limit.

Service has no endpoints

Compare the Service selector with pod labels:

kubectl get service <service> -n <namespace> -o yaml
kubectl get pods -n <namespace> --show-labels
kubectl get endpoints <service> -n <namespace>

Pods must match the selector and be ready before they normally appear as usable endpoints.

Ingress returns 502 or 503

Check the ingress controller, backend Service, endpoints, pod readiness, service targetPort, application listening address, and network policies. Review controller logs and events. A healthy pod alone does not prove that the full routing path works.

Forbidden

Identify the current identity and test the precise action:

kubectl auth whoami
kubectl auth can-i get pods -n <namespace>
kubectl auth can-i create deployments -n <namespace>

Grant the minimum Role or ClusterRole binding required. Do not solve routine authorization failures by granting cluster-admin.

DNS resolution fails

Test from a temporary diagnostic pod approved for the environment, inspect CoreDNS pods and logs, validate NetworkPolicy, and confirm the requested Service name and namespace.

AKS upgrade preparation

Before an upgrade:

1. Review supported Kubernetes versions and upgrade paths.
2. Check API deprecations used by workloads.
3. Confirm PodDisruptionBudgets permit safe eviction.
4. Validate capacity for surge nodes.
5. Test the target version outside production.
6. Confirm backups and recovery procedures.
7. Plan monitoring and rollback or remediation.

An AKS control-plane or node upgrade does not validate application compatibility automatically.

Namespaces, labels, and annotations

Namespaces organize namespaced resources and can support access, policy, and quota boundaries. They are not automatically hard security or network-isolation boundaries.

Labels identify and select objects. Use a consistent scheme such as the recommended app.kubernetes.io/* labels so Deployments, Services, dashboards, and cost reports describe workloads consistently.

Annotations store non-identifying metadata such as tool configuration, ownership references, and change information. Do not put secrets in labels or annotations because they are commonly readable and exported.

Useful inspection commands:

kubectl get pods -n <namespace> --show-labels
kubectl get deployment <deployment> -n <namespace> -o jsonpath='{.metadata.annotations}'
kubectl get all -n <namespace> -l app.kubernetes.io/name=<application>

Scheduling controls

The scheduler chooses a node that satisfies resource and placement requirements.

Node selectors and affinity

Use node selectors for simple hard placement requirements. Affinity supports richer required and preferred rules. Overly strict rules can leave pods Pending even when the cluster has unused capacity.

Taints and tolerations

Taints repel pods; tolerations allow a pod to be scheduled onto a tainted node but do not guarantee placement there. Use them to reserve specialized pools or protect system workloads.

Topology spread

Topology-spread constraints distribute replicas across zones or nodes. Confirm there are enough eligible nodes and that the constraints remain satisfiable during upgrades and failures.

Persistent storage

Pods are replaceable, so durable state belongs on persistent services or volumes designed for the workload.

• A PersistentVolume represents storage capacity.
• A PersistentVolumeClaim requests capacity and access characteristics.
• A StorageClass describes dynamic provisioning behavior.
• A StatefulSet provides stable identities and ordered behavior for suitable stateful applications.

Before choosing an AKS storage option, evaluate access mode, zone behavior, latency, throughput, snapshots, encryption, backup, restore, and failure recovery.

Inspect storage:

kubectl get persistentvolume
kubectl get persistentvolumeclaim -n <namespace>
kubectl describe persistentvolumeclaim <claim> -n <namespace>
kubectl get storageclass

A Bound claim proves Kubernetes attached a matching volume; it does not prove the application can mount, read, write, or recover its data.

Kubernetes networking model

Each pod receives a network identity from the cluster networking implementation. Services provide stable discovery and traffic distribution. NetworkPolicy can restrict traffic when enforced by the selected data plane.

Troubleshoot networking layer by layer:

1. Is the application process listening on the expected address and port?
2. Is the container port documented correctly?
3. Is the readiness probe passing?
4. Does the Service selector match pod labels?
5. Does targetPort resolve to the correct container port?
6. Are EndpointSlices populated?
7. Do NetworkPolicies permit the traffic?
8. Does DNS resolve the Service name?
9. Is the ingress or gateway controller healthy?
10. Are the load balancer, NSG, route, DNS, and certificate correct outside the cluster?

This sequence prevents changing an external load balancer when the actual failure is an unready pod.

Ingress and TLS

An ingress path typically includes public DNS, an Azure load balancer or Application Gateway, an ingress controller, a Kubernetes Service, and ready pods. Record the ownership and health check at every hop.

For TLS:

• Use certificates issued by an approved authority.
• Monitor expiration and renewal.
• Keep private keys in an approved secret system.
• Confirm the certificate name matches the requested hostname.
• Test the complete chain and modern protocol support.
• Understand whether TLS terminates at the edge, ingress, or application.

Do not diagnose every 502 as an ingress problem. A backend startup failure, wrong target port, failed readiness probe, or empty endpoint set can produce the same external symptom.

PodDisruptionBudget and maintenance

A PodDisruptionBudget limits voluntary disruption to a workload. It does not protect against every node failure and cannot create missing capacity.

Set budgets using actual replica counts and availability needs. An impossible budget can block node drains and upgrades. Test:

kubectl get poddisruptionbudget -A
kubectl describe poddisruptionbudget <name> -n <namespace>

Combine disruption budgets with multiple replicas, topology distribution, readiness probes, and sufficient surge capacity.

Autoscaling layers

Horizontal Pod Autoscaler

The HPA changes pod replicas based on observed metrics. CPU-based scaling requires resource requests because utilization is calculated against the request.

Cluster autoscaler

The cluster autoscaler changes node-pool capacity when pods cannot schedule or nodes remain underused under its rules.

Vertical Pod Autoscaler

VPA recommends or changes pod requests depending on mode and configuration. Automatic changes may restart pods.

Avoid independent scaling policies that fight one another. Define minimum capacity for availability and test behavior during traffic spikes, deployments, and node upgrades.

Observability

Collect signals from the application, Kubernetes, nodes, and Azure platform:

• Request rate, errors, duration, and saturation.
• Pod readiness, restarts, and termination reasons.
• Deployment availability and rollout duration.
• Node CPU, memory, disk, and pressure conditions.
• Kubernetes events.
• Control-plane and audit information available through the platform.
• Ingress requests, backend health, and TLS failures.
• Dependency health and application traces.

Use correlation identifiers and immutable deployment versions so an incident can be connected to a release.

Avoid relying only on kubectl logs; container logs may disappear after pod deletion or node loss unless they are collected centrally.

Backup and recovery

Separate these recovery concerns:

• Recreating the cluster and node pools.
• Reapplying Kubernetes manifests or GitOps state.
• Restoring persistent application data.
• Restoring certificates and external secret references.
• Reestablishing DNS, identity, and network integrations.
• Recovering workloads in another region when required.

Test recovery. A repository of manifests does not restore database or volume contents, and a volume snapshot does not recreate the application platform.

Production deployment checklist

• Image is trusted, scanned, and immutable.
• Requests and limits are evidence-based.
• Readiness, liveness, and startup probes have distinct purposes.
• At least two replicas are used where availability requires them.
• Disruption and topology behavior are tested.
• Service selectors and ports are verified.
• Ingress, DNS, and TLS are monitored.
• Workload identity replaces embedded Azure credentials.
• RBAC and NetworkPolicy follow least privilege.
• Logs, metrics, traces, and alerts identify the release.
• Rollback and data-migration compatibility are understood.
• Backup and restoration are tested.
• Supported Kubernetes and node-image versions are maintained.

Troubleshooting decision table

Command quick reference

kubectl config current-context
kubectl config get-contexts
kubectl get all -n <namespace>
kubectl get pods -n <namespace> -w
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --tail=200
kubectl exec -it <pod> -n <namespace> -- <command>
kubectl rollout status deployment/<name> -n <namespace>
kubectl rollout restart deployment/<name> -n <namespace>
kubectl top pods -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Use kubectl exec only when required and permitted. A minimal production image may intentionally contain no shell or diagnostic tools.

Frequently asked questions

Is AKS the same as Kubernetes?

AKS is Microsoft's managed Kubernetes service. It provides Kubernetes APIs plus Azure-managed control-plane and platform integrations.

Should application secrets be stored directly in YAML?

No. Do not commit secret values to manifests. Use an approved secret-management workflow and workload identity where possible.

Why does deleting a pod not fix the problem?

Controllers recreate pods from the same template. If the image, command, configuration, probe, or permissions are wrong, the new pod will usually fail in the same way.

What is the first command to run during an incident?

Start with kubectl get pods -n <namespace> -o wide, then review events, description, and logs while preserving evidence.

Official references

• Kubernetes documentation
• Kubernetes concepts
• kubectl reference
• Kubernetes security
• Azure Kubernetes Service documentation
• AKS best-practices guidance