Deep Dive Into Kubernetes Init Containers

Deep Dive Into Kubernetes Init Containers

What is an init container?

An init container is a type of container that has a few modified operational behavior and rules. One of the most dominant features is that init containers are started and terminated before application containers, and they must run to completion with success. They specifically exist for initializing the workload environment.

Motivations for using init containers

Using distroless container images as base images almost ensures using init containers if the application requires tasks to be run in initialization. As we can’t install tooling into the application container image, we need to create a new image for just initialization to be used by the new init container.

  • Waiting for required resources to be ready:
  • Kubernetes Services(+end points)
  • Database
  • Cache
  • Queue
  • API
  • Etc.
  • Database migrations/applying seeds
  • Reducing secret/credential exposure by moving privileged tasks to init containers.
  • Fetching assets from a git repository or cloud storage which require credentials.
  • Using different security contexts for deploy-time and runtime.
  • Generating configuration files using runtime properties, for example, pod IP, assigned external IP, hostname, external ingress IP, etc.
  • Generating configuration files using key/value stores.

Lifecycle of init containers

Here is an overview of the lifecycle from applying manifests via kubectl to pod receiving traffic.

Pod Lifecycle

Init containers’ features and behaviors

Init container is a container type with specific differences geared to the initialization of a pod. The kubelet executes init containers before any other containers in the pod after ensuring storage and network services are up. If there is more than one init container, they are run sequentially in the order they appear on the configuration.

Running sequentially

Each init container must be completed successfully with exit code zero to run the next init container if any is defined. If any of the init containers return a non-zero exit code, kubelet restarts it. Once all init containers are run to completion successfully (zero exit code), kubelet starts the regular containers in parallel.

Restart policy

A non-zero exit code triggers a restart of the init-container depending on the pod.spec.restartPolicy. If pod’s restartPolicy is set to “Always,” init containers are restarted as if the restartPolicy was set to “OnFailure.” If rather a pod’s restartPolicy is set to “Never,” then init containers aren’t restarted on failure, so the pod is deemed to have failed permanently.

Updating init container images

A change in the image used for a init container doesn’t warrant a restart or change in the pod. Kubelet uses the updated image in the next cycle of the pod.

Fields that aren’t supported

Although init containers and regular containers share the exact spec of container object, some fields aren’t allowed to be set for init containers.

Requests and limits calculation

Resource management is one of the most important jobs of a cluster. Init containers require resources to run to completion, so setting requests and limits for resources makes sense.

Separation of concerns

Using the init container approach means that we would have at least two different container images; an image for the init container, which includes all the tooling for setup, and the other is the application image. Having separate images allows us to distribute ownership of the images to different teams.

Security

Using init containers can bring some security benefits. Let’s assume that the app requires some assets from a git repository or cloud storage. Instead of giving access to the secrets/credentials to the app container, where it might get accessed by an unauthorized party, we can only provide access to the init container. Reducing access means that exposure of the secrets/credentials is short-lived and less accessible.

Configuring init containers

Init containers are defined in the pod.spec.initContainers array, whereas regular containers are defined under the pod. spec.containers array. Both hold Container objects.

// File: https://github.com/kubernetes/kubernetes/blob/e6c093d87ea4cbb530a7b2ae91e54c0842d8308a/pkg/apis/core/types.go#L2813
...
// PodSpec is a description of a pod
type PodSpec struct {
Volumes []Volume
// List of initialization containers belonging to the pod.
InitContainers []Container
// List of containers belonging to the pod.
Containers []Container
...

Debugging and troubleshooting of init containers

We can gather information regarding init containers directly and indirectly. Before we dive into the individual init container statuses, let’s look at pod-wide indicators.

Pod status

During the pod’s initialization, some “init container” related statuses are displayed on pod status. These valuable outputs are shown with a prefix of “init:”

kubectl get pods

NAME READY STATUS RESTARTS AGE
...
k8s-init-containers-668b46c54d-kg4qm 0/1 Init:1/2 1 8s
kubectl get pods

NAME READY STATUS RESTARTS AGE
...
k8s-init-containers-668b46c54d-kg4qm 0/1 Init:CrashLoopBackOff 5 4m12s
kubectl get pods --watch

NAME READY STATUS RESTARTS AGE
k8s-init-containers-64f984c8d7-tdjrg 0/1 Pending 0 0s
k8s-init-containers-64f984c8d7-tdjrg 0/1 Pending 0 4s
k8s-init-containers-64f984c8d7-tdjrg 0/1 Init:0/4 0 4s
k8s-init-containers-64f984c8d7-tdjrg 0/1 Init:0/4 0 6s
k8s-init-containers-64f984c8d7-tdjrg 0/1 Init:Error 0 7s
k8s-init-containers-64f984c8d7-tdjrg 0/1 Init:1/4 1 8s
k8s-init-containers-64f984c8d7-tdjrg 0/1 Init:1/4 1 9s
k8s-init-containers-64f984c8d7-tdjrg 0/1 Init:2/4 1 43s
k8s-init-containers-64f984c8d7-tdjrg 0/1 Init:3/4 1 44s
k8s-init-containers-64f984c8d7-tdjrg 0/1 PodInitializing 0 45s
k8s-init-containers-64f984c8d7-tdjrg 1/1 Running 0 46s

Pod conditions

The pod conditions are high-level status views of the pod regarding certain statuses, which is a good place to start. The kubelet populates Pod.status.Conditions array with PodCondition objects.

  • Initialized
  • Ready
  • ContainersReady
  • PodScheduled
...
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
...
...
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
...
kubectl get pods -o=jsonpath='{"Pod Name, Condition Initialized"}{"
"}{range .items[*]}{.metadata.name},{@.status.conditions[?(@.type=="Initialized")].status}{"
"}{end}'

Pod Name, Condition Initialized
buildkit-builder0-76784c68c7-c52nw,True
k8s-init-containers-b4ddb8ffc-2sh4m,True
postgresql-postgresql-0,True
redis-master-0,True

ContainerStatus object (in pod.status.initContainerStatuses[])

The status of the init containers can be found under the pod.status.initContainerStatuses array, which holds a ContainerStatus object for each init container.

kubectl get pods k8s-init-containers-64f984c8d7-tdjrg -o yaml
...
initContainerStatuses:
- containerID: containerd://b60870084a2065d…
image: docker.io/loftsh/app-init:FLiBwNW
imageID: docker.io/loftsh/app-init@sha256:e46e2…
lastState: {}
name: init-fetch-files
ready: true
restartCount: 2
state:
terminated:
containerID: containerd://b60870084a2065def2d289f7…
exitCode: 0
finishedAt: "2022-02-27T02:09:37Z"
reason: Completed
startedAt: "2022-02-27T02:09:37Z"
...

Container state field

The state field we’ve seen in ContainerStatus is available in the kubectl describe pods <pod-name> command output as well. Let's examine a pod that has more init containers in it. We use a pod with two init containers and an app container for this example.

kubectl describe pod k8s-init-containers-668b46c54d-kg4qm

Init Containers:
init-fetch-files:
Container ID: containerd://31ac3…
Image: loftsh/app-init:rQyZJBK
Image ID: docker.io/loftsh/app-init@sha256:79ce…
Port: <none>
Host Port: <none>
Command:
fetchFiles.sh
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 15 Mar 2022 13:54:39 +0000
Finished: Tue, 15 Mar 2022 13:54:39 +0000
Ready: True
Restart Count: 0
Environment:
PUBLIC_ROOT: /data/public
Mounts:
/data from app-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-44vbg (ro)

Init Containers:

init-check-services:
Container ID: containerd://eb1dbae93999a63fed7…
Image: loftsh/app-init:kxPrHjW
Image ID: docker.io/loftsh/app-init@sha256:5…
Port: <none>
Host Port: <none>
Command:
checkServices.sh
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 15 Mar 2022 14:11:23 +0000
Finished: Tue, 15 Mar 2022 14:11:23 +0000
Ready: False
Restart Count: 223
Environment:
APP_NAME: app
SERVICES: redis-master,postgresql
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ftmj6 (ro)

Containers:
app:
Container ID:
Image: loftsh/app:sNKfQGb
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
APP_NAME: app
Mounts:
/data from app-data (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-44vbg (ro)

Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
...
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 04 Mar 2022 07:16:21 +0000
Finished: Fri, 04 Mar 2022 07:16:21 +0000
...

Container logs

You can access the init containers logs the same way you do with a regular container.

kubectl logs k8s-init-containers-668b46c54d-kg4qm -c init-check-services

At least one (app-redis) of the services (app-redis app-postgresql app-mongodb) is not available yet.

Events

Kubernetes events are also a good source of information.

kubectl get events

LAST SEEN TYPE REASON OBJECT MESSAGE
81s Warning BackOff pod/k8s-init-containers-5c694cd678-gr8zg Back-off restarting the failed container

Conclusion

Init containers bring a different mindset to the initialization phase of an app or service. As we’ve seen in the earlier sections, there are many benefits of using init containers ranging from enhanced security, separation of ownership, ability to keep the app containers lean as possible, and more. Configuration, troubleshooting, and monitoring init containers aren’t much different from regular containers, apart from the few behavioral differences you need to consider.

References and further reading

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Loft Labs

Loft Labs

>> www.loft.sh << Build Your Internal Kubernetes Platform With Virtual Clusters, Namespace Self-Service & Secure Multi-Tenancy