CKAD concept notes: multi-container Pod patterns

Based on: Kubernetes v1.35 (commands and YAML checked on kind kindest/node:v1.35.8)

Contents

• Container types inside a Pod
• init containers
• sidecars
• ambassador and adapter
• ephemeral containers
• Example
• Easily confused
• Exam tips
• References

Container types inside a Pod

What they share

• network: one Pod IP shared by all. Containers talk to each other over localhost:<port>, so ports must not collide
• storage: each container mounts volumes defined in spec.volumes via volumeMounts. Files passed between containers usually go through an emptyDir
• IPC: SystemV semaphores and POSIX shared memory also work
• the process namespace is separate by default. Turn on process namespace sharing if you need it

init containers

Containers that do preparation work before the Pod starts. The docs give examples like waiting for a Service to exist, cloning a Git repository into a volume, or rendering a config file from a template.

• they use a separate image from the app, so tools like sed or dig don’t have to live in the app image
• they can be given access to Secrets the app containers can’t see
• used to hold back the app until some condition is met

How it works

1. Once networking and storage are ready, the kubelet runs the init containers in the order they appear in the spec
2. Each init container has to succeed (exit 0) before the next one starts
3. When they’re all done, the app containers start

On failure:

• Pod restartPolicy is Never: the whole Pod is marked failed
• OnFailure: that init container is retried
• Always: init containers get OnFailure behavior

Other rules:

• until init finishes, the Pod phase is Pending and the Initialized condition is false. kubectl get shows STATUS as Init:0/2
• regular init containers can’t use lifecycle, livenessProbe, readinessProbe, or startupProbe. Adding one gets the Pod rejected by the API server with Forbidden: may not be set for init containers without restartPolicy=Always
• they can run more than once, so write them to be idempotent. A file you write into an emptyDir may already be there
• on a running Pod, only an init container’s image can be changed, and changing it doesn’t restart the Pod
• container names must be unique across init and app containers
• ports on init containers aren’t tied to Services

Resource accounting

• the largest request/limit among the init containers = the effective init request/limit
• the Pod’s effective request/limit = max(sum of app and sidecar containers, effective init value) + pod overhead
• scheduling uses these effective values, so a big init container reserves resources that are only used during initialization

sidecars

A container running next to the main app to handle supporting work like log shipping, monitoring, or data sync. The point is to add functionality without touching the app code.

native sidecars

The approach introduced by the SidecarContainers feature gate (enabled by default since v1.29). Put restartPolicy: Always on an initContainers entry and it becomes a sidecar.

• ordering is guaranteed like init containers. Once a sidecar reaches started, the next init container begins. If it has a startupProbe, that probe has to pass before it counts as started
• after starting, it keeps running until the Pod ends. If it exits, it’s restarted regardless of the Pod restartPolicy
• probes are supported. The readinessProbe result feeds into the Pod’s ready status
• on Pod shutdown: sidecars only get TERM after the main containers have fully stopped, and they stop in reverse order of definition
• in a Job, once the main container finishes the Job completes even if the sidecar is still running
• changing the image restarts just that container, not the Pod
• if the main containers use up the whole grace period during shutdown, sidecars can get SIGKILL right away. A non-zero exit code in that case can be treated as normal

  • grace period: how long the Pod gets between the delete request and a forced kill. Set with spec.terminationGracePeriodSeconds, 30 seconds by default. Any container still running when it runs out gets SIGKILL

  • it’s one timer shared by the whole Pod, not one per container. The clock keeps running while the main container handles its SIGTERM, so the longer the main container takes, the less time is left for sidecars

  • example: 30-second grace period, the main container needs 28 seconds to clean up, and a log-shipping sidecar needs 5 seconds to flush its remaining logs

    t=0s   Pod delete requested. SIGTERM to the main container
    t=28s  main container exits (exit 0). Only now does the sidecar get SIGTERM
    t=30s  deadline reached. SIGKILL to the sidecar mid-cleanup -> exit 137 (128+9)

    if the main container uses all 30 seconds, the sidecar never gets SIGTERM at all and is SIGKILLed along with the main container

  • a sidecar only exists to support the main container, so if the main container shut down cleanly, the Pod’s shutdown did its job. The official docs also say external tooling should generally ignore a sidecar’s non-zero exit code here. Make sure monitoring doesn’t flag a sidecar’s 137 as a failure

classic sidecars

Add another app container under containers. This works on older versions without per-container restartPolicy, and it’s still valid today. Since it’s just a regular app container, though:

• there’s no guarantee it starts before the app
• it’s a problem in Jobs. A Pod only becomes Succeeded when every container has exited, so a sidecar that keeps running means it never finishes

The official docs also say that if start and stop order don’t matter and every container is needed for the Pod to work, this approach is good enough.

ambassador and adapter

Patterns introduced alongside the sidecar in the 2015 Kubernetes blog post “Patterns for Composite Containers”. The current concepts docs have no separate entry for them, and there’s no dedicated field either; you just add containers.

The ambassador relies on the shared network namespace; the adapter relies on a shared volume or localhost. All three are implemented with containers or native sidecars.

ephemeral containers

A temporary debugging container attached to a running Pod. Stable since v1.25.

• you can’t add containers to a Pod after it’s created, but ephemeral containers are added through the ephemeralcontainers subresource. That’s also why kubectl edit can’t add one
• once added, they can’t be changed or removed
• no automatic restarts, no resource guarantees
• ports, livenessProbe, readinessProbe, and resources aren’t allowed
• not available for static Pods
• when to use one: the container has crashed, or the image has no shell (distroless, for example) so kubectl exec doesn’t work

kubectl debug -it <pod> --image=busybox:1.28 --target=<container>

--target joins the process namespace of the given container so you can see its processes. The container runtime has to support it; if it doesn’t, the debug container may come up with a separate process namespace.

Example

A Pod where an init container writes a config file and a native sidecar reads the app log.

apiVersion: v1
kind: Pod
metadata:
  name: web
spec:
  initContainers:
    - name: init-config
      image: busybox:1.36
      command: ["sh", "-c", 'echo "server_name=web" > /config/app.conf']
      volumeMounts:
        - name: config
          mountPath: /config
    - name: log-shipper
      image: busybox:1.36
      restartPolicy: Always
      command: ["sh", "-c", "touch /logs/app.log; tail -F /logs/app.log"]
      volumeMounts:
        - name: logs
          mountPath: /logs
  containers:
    - name: app
      image: busybox:1.36
      command:
        [
          "sh",
          "-c",
          "while true; do cat /config/app.conf >> /logs/app.log; sleep 5; done",
        ]
      volumeMounts:
        - name: config
          mountPath: /config
        - name: logs
          mountPath: /logs
  volumes:
    - name: config
      emptyDir: {}
    - name: logs
      emptyDir: {}

Order: init-config completes → log-shipper starts (started) → app starts. log-shipper keeps running alongside app.

To use the same sidecar the classic way, move it under containers and drop restartPolicy. log-shipper and app then start together, in no particular order.

kubectl apply -f web.yaml
kubectl get pod web                      # STATUS: Init:0/2 → Running
kubectl logs web -c init-config          # init container logs
kubectl logs web -c log-shipper -f       # sidecar logs
kubectl logs web --all-containers
kubectl exec -it web -c app -- sh
kubectl describe pod web                 # State/Reason under Init Containers

Easily confused

init container, native sidecar, classic sidecar

kubectl exec vs kubectl debug

Exam tips

Commands

There’s no imperative command that builds a multi-container Pod in one go. Generate a skeleton, then edit the YAML.

kubectl run web --image=busybox:1.36 --dry-run=client -o yaml \
  --command -- sh -c 'sleep 3600' > web.yaml
# write initContainers, the extra containers, and volumes by hand

kubectl logs <pod> -c <container>
kubectl exec -it <pod> -c <container> -- sh
kubectl debug -it <pod> --image=busybox:1.28 --target=<container>

kubectl explain

kubectl explain pod.spec.initContainers
kubectl explain pod.spec.initContainers.restartPolicy  # Always = native sidecar
kubectl explain pod.spec.shareProcessNamespace         # sharing processes between containers
kubectl explain pod.spec.volumes.emptyDir              # sharing files between containers
kubectl explain pod.spec.containers.volumeMounts

References

• Init Containers | Kubernetes
• Sidecar Containers | Kubernetes
• Ephemeral Containers | Kubernetes
• Pods: Pods with multiple containers | Kubernetes
• Pod Lifecycle: Pod phase, Restart policy, Pod shutdown and sidecar containers | Kubernetes
• Debug Running Pods: ephemeral container | Kubernetes
• Share Process Namespace between Containers in a Pod | Kubernetes
• The Distributed System ToolKit: Patterns for Composite Containers | Kubernetes Blog (2015)
• kubectl debug
• CNCF CKAD curriculum