Using Kubernetes Ephemeral Containers for Troubleshooting
by Levent Ogut
Containers and the ecosystem around them changed how engineers deploy, maintain, and troubleshoot workloads. But debugging an application on a Kubernetes cluster can be daunting at times, as you might not find the tools you need in the container. Many engineers build containers with distroless images based on slimmed-to-the-bone distributions where there’s not even a package manager or a shell. In the deep end, some teams use scratch as a base image and only add the files the application needs to run. Some reasons why this is a common practice is:
- To have a smaller attack vector area.
- To have faster-scanning performance.
- Reduced image size.
- To have a faster build and CD/CI cycle.
- To have fewer dependencies.
These stripped-down base images don’t include the tools you would use to troubleshoot an application or its dependencies. This is the perfect opportunity for the Kubernetes ephemeral containers feature to shine. Ephemeral containers allow you to create a container image that includes all the debugging tools you might need. Once there is a need for debugging, you would deploy the ephemeral container into the running pod of your choice.
You can not add a container to a deployed pod; you need to update the spec, and resources are re-created. However, an ephemeral container can be added to an existing pod to allow you to troubleshoot a live issue.
Ephemeral containers is an alpha feature in Kubernetes 1.22, so the official recommendation is not to use it in production environments.
Configuration of Ephemeral Containers
Ephemeral containers share the same spec as regular containers. However, some fields are disabled, and some behaviors are changed. Some of the significant changes are listed below; check the ephemeral container spec for a complete list.
- They are not to be restarted.
- Resources definition is not allowed.
- Ports are not allowed.
- Startup, liveness, and readiness probes are not allowed.
Enabling Ephemeral Containers in Your Cluster
As this feature is in the alpha state, it needs to be explicitly enabled using feature gates. Change to feature gates flag might not be allowed in most if not all managed cloud Kubernetes providers; please check with your provider. First, let’s check if the ephemeral containers feature is enabled or not. To do that, run the following command.
$ kubectl debug -it <POD_NAME> --image=busybox
If the feature is not enabled, you will see a similar to the following message displayed.
Defaulting debug container name to debugger-wg54p.
error: ephemeral containers are disabled for this cluster (error from server: "the server could not find the requested resource").
EphemeralContainers=true to the feature gates flag
--feature-gates= in the kubelet, kube-apiserver, kube-controller-manager, kube-proxy, kube-scheduler arguments.
Following is an example of the current flag definition:
Now you need to add EphemeralContainers=true using ‘,’ as a separator.
Now you need to restart relevant services for changes to take effect.
For more information on feature gates and arguments, you can refer to the feature gates docs.
Using Ephemeral Containers
Now that your cluster supports the Ephemeral Containers feature, let’s try it. To create ephemeral containers, you will use the
debug subcommand of the
kubectl command-line tool.
First, let’s create a deployment that we can use to simulate our application using nginx as an image.
$ kubectl create deployment nginx-deployment --image=nginx
The API server response should be successful.
Now you should get the pod name that you want to debug.
$ kubectl get podsNAME READY STATUS RESTARTS AGE
nginx-deployment-66b6c48dd5-frsv9 1/1 Running 6 62d
The following command will create a new ephemeral container in the pod nginx-deployment-66b6c48dd5-frsv9. The ephemeral container’s image will be
-t flags allow us to attach to the newly created container.
$ kubectl debug -it pods/nginx-deployment-66b6c48dd5-frsv9 --image=busyboxDefaulting debug container name to debugger-r44v5.
If you don't see a command prompt, try pressing enter.
Now we can quickly start debugging.
/ # ping 220.127.116.11PING 18.104.22.168 (22.214.171.124): 56 data bytes
64 bytes from 126.96.36.199: seq=0 ttl=112 time=9.797 ms
64 bytes from 188.8.131.52: seq=1 ttl=112 time=9.809 ms
^C/ # nc --helpBusyBox v1.34.1 (2021-11-11 01:55:05 UTC) multi-call binary.
Usage: nc [OPTIONS] HOST PORT - connect
nc [OPTIONS] -l -p PORT [HOST] [PORT] - listen
When you use the
kubectl describe pods <POD_NAME> command, you can see a new field, "Ephemeral Containers," this section holds the ephemeral containers and their attributes.
$ kubectl describe pods nginx-deployment-66b6c48dd5-frsv9Name: nginx-deployment-66b6c48dd5-frsv9
Start Time: Mon, 30 Aug 2021 21:50:17 +0200
Controlled By: ReplicaSet/nginx-deployment-66b6c48dd5
Container ID: containerd://6367af3713afb85ecb1e1a057ba9db4e3b2c48f39fee6a248cd2811e198001aa
Container ID: containerd://eec23aa9ee63d96b82970bb947b29cbacc30685bbc3418ba840dee109f871bf0
Image ID: docker.io/library/busybox@sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353
Host Port: <none>
Started: Mon, 15 Nov 2021 20:28:57 +0100
Restart Count: 0
Process Namespace Sharing with Ephemeral Containers
Process namespace sharing has been an excellent troubleshooting option, and this feature can be used with ephemeral containers. Process namespace sharing can not be applied to an existing pod, so a copy of the target pod must be created.
--share-processesflag enables process namespace sharing when used with
--copy-to. These flags copy the existing pod spec definition into a new one with process namespace sharing enabled in the spec.
$ kubectl debug -it nginx-deployment-66b6c48dd5-frsv9 --image=busybox --share-processes --copy-to=debug-pod
Let’s run the
ps command to see the running process.
/ # ps aux
As you can expect, you can see
/pause from the busybox container and an
nginx process from the nginx-deployment container.
PID USER TIME COMMAND
1 root 0:00 /pause
6 root 0:00 nginx: master process nginx -g daemon off;
11 101 0:00 nginx: worker process
12 root 0:00 sh
17 root 0:00 ps aux
With process namespace, sharing container filesystems are accessible too, which is very useful for debugging.
You can reach the container with
/proc/<PID>/root link. From the above output, we know that
nginx has PID 6.
# ls /proc/6/root/etc/nginx
Here we can see the Nginx directory structure and configuration files on the target container.
conf.d koi-utf mime.types nginx.conf uwsgi_params fastcgi_params koi-win modules scgi_params win-utf
The ephemeral containers feature certainly brings a lot of opportunities, and process namespace sharing allows advanced debugging capabilities. If you work with applications running in Kubernetes clusters, it would be worth your time to experiment with these features. It’s not hard to imagine some teams even automating workflows using these tools, like fixing other containers automatically when their readiness probes fail.
- Kubernetes documentation on ephemeral containers
- Debugging with an ephemeral debug container
- Share process namespace between containers in a pod
Originally published at https://loft.sh.