CKAD concept notes: probes and the Pod lifecycle

Based on: Kubernetes v1.35 (commands and YAML checked on kind kindest/node:v1.35.8)

Contents

• Pod phase
• Pod conditions
• Container state
• restartPolicy
• Probe types
• Probe handlers
• Timing fields
• Examples
• Probes vs. init containers
• Exam tips
• References

Pod phase

status.phase is a summary of where the Pod is in its lifecycle. The official docs are explicit that it isn’t a state machine covering every container and Pod state.

CrashLoopBackOff and Terminating in the STATUS column of kubectl get pod aren’t phases. They’re display values kubectl shows; the phase is a field in the Pod API.

Pod conditions

status.conditions is the list of conditions the Pod has or hasn’t passed yet. The ones the kubelet manages that tie directly into probes:

Each condition carries status (True/False/Unknown), lastTransitionTime, reason, message, and so on. If you add custom conditions with readinessGates, the Pod is only ready when all containers are ready and every gate condition is True.

Container state

Each container is in one of three states: Waiting, Running, Terminated. Check with kubectl describe pod <name>.

• Waiting: neither Running nor Terminated. It’s doing work needed to start, like pulling the image or applying Secrets. A Reason field is shown with it
• Running: executing without issues. If there was a postStart hook, it has already finished
• Terminated: it started and then either completed or failed. Shows the reason, exit code, and start/finish times. A preStop hook runs before a container enters this state

restartPolicy

Pod spec.restartPolicy. Values are Always (default), OnFailure, Never. Applies to app containers and regular init containers.

• restart delays use exponential backoff: 10s, 20s, 40s … capped at 300s (5 minutes). After running cleanly for 10 minutes the backoff resets
• while the backoff is in effect, kubectl shows CrashLoopBackOff
• liveness/startup probe failures can also lead to CrashLoopBackOff
• sidecar containers (init containers with a per-container restartPolicy: Always) ignore the Pod restartPolicy and always restart
• workload constraints: Deployments only allow Always, Jobs only allow OnFailure or Never

Probe types

What changes when there’s a startup probe

Liveness and readiness probes don’t run until the startup probe succeeds. Their initialDelaySeconds also counts from the moment the startup probe succeeds.

The official docs recommend a startup probe when startup takes longer than initialDelaySeconds + failureThreshold × periodSeconds. Point it at the same endpoint as the liveness probe and give it a generous failureThreshold. Leave the liveness defaults alone.

Does the liveness probe wait for readiness?

No. The two probes don’t depend on each other succeeding. If you want liveness to start later, use initialDelaySeconds or a startup probe.

What if a probe isn’t defined?

That probe’s result is always treated as Success. The exception: a defined readiness probe reports Failure until its initial delay passes. So a Pod with a readiness probe starts without traffic and only gets traffic once the probe succeeds.

When a liveness probe is set up wrong

This is something the official docs call out with a caution box. A liveness probe should only fail when the container is truly unrecoverable (a deadlock, for example). If liveness fails under heavy load, containers restart one after another, the remaining Pods take on even more load, and it turns into a cascading failure.

If the process already dies on its own when something goes wrong, the kubelet handles it through restartPolicy without any liveness probe. A common pattern is to use the same lightweight HTTP endpoint as readiness but give liveness a higher failureThreshold. That way the Pod stays not-ready for a while before it gets killed.

Probe handlers

Each probe specifies exactly one of these four.

The result is Success, Failure, or Unknown. Unknown means the diagnostic itself failed, so nothing happens and it moves on to the next check.

named ports

You give a container port a name, then refer to it by that name instead of the number elsewhere. port in httpGet and tcpSocket accepts it.

ports:
  - name: http # name port 8080 "http"
    containerPort: 8080
livenessProbe:
  httpGet:
    path: /healthz
    port: http # refer to it by name, not number

• the port number lives only in containerPort. If 8080 becomes 9090, the probes stay as they are
• the same name can be referenced from a Service targetPort and a NetworkPolicy ports.port. A Service looks up the port with that name in each Pod, so Pods using different numbers still sit behind one Service
• naming rules: at most 15 characters, lowercase letters, digits, and -, at least one letter, can’t start or end with -. Must be unique within a Pod
• the grpc handler doesn’t accept named ports. Numbers only

Timing fields

The maximum startup time a startup probe allows is failureThreshold × periodSeconds. With failureThreshold: 30 and periodSeconds: 10, that’s 300 seconds.

Examples

A Pod with all three probes

apiVersion: v1
kind: Pod
metadata:
  name: probe-demo
spec:
  containers:
    - name: app
      image: registry.k8s.io/e2e-test-images/agnhost:2.40
      args: ["liveness"]
      ports:
        - name: http
          containerPort: 8080
      startupProbe:
        httpGet:
          path: /healthz
          port: http
        failureThreshold: 30
        periodSeconds: 10
      livenessProbe:
        httpGet:
          path: /healthz
          port: http
        periodSeconds: 10
        failureThreshold: 3
      readinessProbe:
        tcpSocket:
          port: 8080
        periodSeconds: 5

• for up to 300 seconds after start, the startup probe watches /healthz. Once it succeeds, liveness and readiness take over
• the image and args come from the HTTP liveness example in the official docs. This image’s /healthz returns 200 for the first 10 seconds and 500 after that, so it’s expected that this Pod keeps restarting on probe failures. Running it on a v1.35 kind cluster, the first liveness-failure restart shows up after about 40 seconds, and after the restart the startup probe misses the first 10 seconds, so Startup probe failed ... statuscode: 500 events pile up

exec, grpc

livenessProbe:
  exec:
    command: ["cat", "/tmp/healthy"]
  initialDelaySeconds: 5
  periodSeconds: 5

livenessProbe:
  grpc:
    port: 2379
  initialDelaySeconds: 10

Commands to check

kubectl describe pod probe-demo      # Unhealthy / Killing and "Liveness probe failed" in Events
kubectl get pod probe-demo           # READY 0/1, whether RESTARTS goes up
kubectl get pod probe-demo -o jsonpath='{.status.phase}'
kubectl get pod probe-demo -o jsonpath='{.status.conditions}'
kubectl get pod probe-demo -o jsonpath='{.status.containerStatuses[0].state}'
kubectl explain pod.spec.containers.livenessProbe

RESTARTS goes up the moment a failed container is back in the running state.

Probes vs. init containers

The requirement from the intro post, “check the DB connection before the app starts”, is confusing because it looks like either one could handle it. They differ in when they run and what happens on failure.

• if the app process shouldn’t even start until the dependency is ready: init container
• if the app stays up but you want to pull it out of traffic when the dependency drops: readiness probe. The official docs also suggest that apps with a strong backend dependency use liveness for the app’s own health and readiness to also cover backend availability
• sidecar containers live in initContainers but keep running and do support probes

Exam tips

• probes have no imperative flags. Build a skeleton with kubectl run app --image=nginx --dry-run=client -o yaml > pod.yaml and add the probe in YAML
• the field lives at spec.containers[].livenessProbe, not directly under spec
• field names are camelCase: livenessProbe, readinessProbe, startupProbe. Handlers are httpGet, tcpSocket, exec, grpc
• successThreshold for liveness and startup must be 1
• a readiness failure never causes a restart. If RESTARTS is climbing, look at liveness or startup
• if liveness fails in a Pod with restartPolicy: Never, the container is killed and doesn’t come back
• CrashLoopBackOff in the STATUS column isn’t a phase. If the question asks for the phase, check with -o jsonpath='{.status.phase}'

kubectl explain

kubectl explain pod.spec.containers.livenessProbe          # common fields: periodSeconds, failureThreshold, etc.
kubectl explain pod.spec.containers.livenessProbe.httpGet  # path, port, httpHeaders
kubectl explain pod.spec.containers.startupProbe
kubectl explain pod.spec.containers.lifecycle              # postStart, preStop
kubectl explain pod.spec.terminationGracePeriodSeconds
kubectl explain pod.spec.restartPolicy                     # Always / OnFailure / Never

References

• Pod Lifecycle
• Liveness, Readiness, and Startup Probes
• Configure Liveness, Readiness and Startup Probes
• Init Containers
• Sidecar Containers
• CKAD Curriculum v1.35