Using Kubernetes Ephemeral Containers for Troubleshooting

A time lapse picture of blue and red streaks of light
  • To have a smaller attack vector area.
  • To have faster-scanning performance.
  • Reduced image size.
  • To have a faster build and CD/CI cycle.
  • To have fewer dependencies.

Configuration of Ephemeral Containers

Ephemeral containers share the same spec as regular containers. However, some fields are disabled, and some behaviors are changed. Some of the significant changes are listed below; check the ephemeral container spec for a complete list.

  • They are not to be restarted.
  • Resources definition is not allowed.
  • Ports are not allowed.
  • Startup, liveness, and readiness probes are not allowed.

Enabling Ephemeral Containers in Your Cluster

As this feature is in the alpha state, it needs to be explicitly enabled using feature gates. Change to feature gates flag might not be allowed in most if not all managed cloud Kubernetes providers; please check with your provider. First, let’s check if the ephemeral containers feature is enabled or not. To do that, run the following command.

$ kubectl debug -it <POD_NAME> --image=busybox
Defaulting debug container name to debugger-wg54p.
error: ephemeral containers are disabled for this cluster (error from server: "the server could not find the requested resource").
...
--feature-gates=RemoveSelfLink=false
...
...
--feature-gates=RemoveSelfLink=false,EphemeralContainers=true
...

Using Ephemeral Containers

Now that your cluster supports the Ephemeral Containers feature, let’s try it. To create ephemeral containers, you will use the debug subcommand of the kubectl command-line tool.

$ kubectl create deployment nginx-deployment --image=nginx
deployment.apps/nginx-deployment created
$ kubectl get podsNAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-66b6c48dd5-frsv9 1/1 Running 6 62d
$ kubectl debug -it pods/nginx-deployment-66b6c48dd5-frsv9 --image=busyboxDefaulting debug container name to debugger-r44v5.
If you don't see a command prompt, try pressing enter.
/ #
/ # ping 8.8.8.8PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=112 time=9.797 ms
64 bytes from 8.8.8.8: seq=1 ttl=112 time=9.809 ms
^C
/ # nc --helpBusyBox v1.34.1 (2021-11-11 01:55:05 UTC) multi-call binary.

Usage: nc [OPTIONS] HOST PORT - connect
nc [OPTIONS] -l -p PORT [HOST] [PORT] - listen
...
$ kubectl describe pods nginx-deployment-66b6c48dd5-frsv9Name:         nginx-deployment-66b6c48dd5-frsv9
Namespace: default
Priority: 0
Node: node1/x.x.x.x
Start Time: Mon, 30 Aug 2021 21:50:17 +0200
Labels: app=nginx
pod-template-hash=66b6c48dd5
Annotations: <none>
Status: Running
IP: 10.0.0.110
IPs:
IP: 10.0.0.110
Controlled By: ReplicaSet/nginx-deployment-66b6c48dd5
Containers:
nginx:
Container ID: containerd://6367af3713afb85ecb1e1a057ba9db4e3b2c48f39fee6a248cd2811e198001aa
Image: nginx:1.14.2
...
...
Ephemeral Containers:
debugger-thwrn:
Container ID: containerd://eec23aa9ee63d96b82970bb947b29cbacc30685bbc3418ba840dee109f871bf0
Image: busybox
Image ID: docker.io/library/busybox@sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353
Port: <none>
Host Port: <none>
State: Running
Started: Mon, 15 Nov 2021 20:28:57 +0100
Ready: False
Restart Count: 0
Environment: <none>
Mounts: <none>

Process Namespace Sharing with Ephemeral Containers

Process namespace sharing has been an excellent troubleshooting option, and this feature can be used with ephemeral containers. Process namespace sharing can not be applied to an existing pod, so a copy of the target pod must be created.

$ kubectl debug -it nginx-deployment-66b6c48dd5-frsv9 --image=busybox --share-processes --copy-to=debug-pod
PID   USER     TIME  COMMAND
1 root 0:00 /pause
6 root 0:00 nginx: master process nginx -g daemon off;
11 101 0:00 nginx: worker process
12 root 0:00 sh
17 root 0:00 ps aux
# ls /proc/6/root/etc/nginx
conf.d koi-utf mime.types nginx.conf uwsgi_params fastcgi_params koi-win modules scgi_params win-utf

Conclusion

The ephemeral containers feature certainly brings a lot of opportunities, and process namespace sharing allows advanced debugging capabilities. If you work with applications running in Kubernetes clusters, it would be worth your time to experiment with these features. It’s not hard to imagine some teams even automating workflows using these tools, like fixing other containers automatically when their readiness probes fail.

Further reading

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Loft Labs

Loft Labs

637 Followers

>> www.loft.sh << Build Your Internal Kubernetes Platform With Virtual Clusters, Namespace Self-Service & Secure Multi-Tenancy