Troubleshooting Applications In Kubernetes

At Broadcom, many of our Identity Security products run on Kubernetes. Given this, I will be covering a series of discussions focused on Kubernetes, in order to provide our customers with best practices and advice on how to troubleshoot applications in Kubernetes. 

Introduction

Running applications in Kubernetes provides a scalable, easy to deploy and easy to upgrade option. Container images are made small and efficient to reduce their size as well as to make them more secure by reducing their attack surface. Moreover, done correctly, a container should not run as root and will not allow installing additional tools in it. However, when things don’t work as expected, troubleshooting becomes more of a challenge. 

In this article, we will discuss some methods and ideas you can use to troubleshoot your containerized application.

Initial Inspection

The first step in troubleshooting your applications is to check the status of the application pod. This article is not meant to explore the basic troubleshooting steps available to you by Kubernetes. Nevertheless, we want to make sure that the reader is aware of these steps.

• Make sure that the pod’s containers are all up and that the pod is in a ready state. If the pod is not ready, use ‘kubectl describe pod’ to find out why.
  ‘kubectl describe pod’ provides information about the pod as well as a list of events that describe the pod initialization process and indicate issues in it.
  For more information about Kubernetes standard tools for debugging application pod status check out Kubernetes documentation at: 
  • https://kubernetes.io/docs/tasks/debug-application-cluster/
  • https://kubernetes.io/docs/tasks/debug-application-cluster/debug-init-containers/

• Check the container logs using:
  kubectl logs -f <podname> -c <containername> -n <podnamespace>
  
  Where:
  -f is used to continue and stream the log file content (follow)
  -c is the name of the container inside the application pod – this parameter is necessary only if the application’s pod has more than one container.
  -n is the name of the namespace the application is running in. it is required only if the application’s pod is running on a namespace that is not the current default namespace.
  If your container has previously crushed, use the following command to access the previous container’s log:
  kubectl logs –previous <podname> -c <containername> -n <podnamespace>

Inspect other log and configuration files by running a shell inside the application’s container. This is done using:
kubectl exec -it <podname> -c <containername> -n <podnamespace> — <cmd>

Where:
-it allows us to interact with the shell 

-c is the name of the container inside the application pod – this parameter is necessary only if the application’s pod has more than one container.
-n is the name of the namespace the application is running in. it is required only if the application’s pod is running on a namespace that is not the current default namespace.

<cmd> should be replaced with the command you want to execute (cmd + args). In order to open a shell use /bin/bash or shNote that what you can execute depends on what’s available on the container and your permissions.

See Kubernetes documentation for more information: https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/

• Kubernetes uses health probes to make sure your pod is healthy and can serve requests. These probes may be configured for each container in your pod. Liveness probes are used by the kubelet to decide when a pod has to be restarted in order to try to recover it. These restarts may interfere with your debugging, restarting the application pod while you are trying to debug it. In this case you may choose to disable the Liveness probe for your application container while debugging it.

Copying files to and from the container

When troubleshooting an application running in a container, we will find ourselves in many cases having to copy files into and out of a running container. These files can be logs that are not being exposed from the container, configuration files, or even core dump files. 

In Kubernetes, an easy way to get files off of a running container can be done using the kubectl command ‘kubectl cp’. This command allows us to get a file, or a folder off of a running container without having to know where this container actually runs. 

The syntax of the command is as follows:

                    ⬐     source (where to copy from)            ⬎    ⬐     destination      ⬎

kubectl cp <namespace>/<podname>:<path to file or folder>  <local path to copy to> -c <containername>

Example:

The following command will copy the entire folder ‘/opt/SecureSpan/Gateway/config’ from a container called ‘gw’ in a pod called ‘gw-dc-87568c597-xn4pz’ to the local folder ‘/config’:

kubectl cp default/gw-dc-87568c597-xn4pz:/opt/SecureSpan/Gateway/config  /config/ -c gw

Copying from the local machine into the container would just require switching the order of the source and target, as explained above.

Troubleshooting DB connectivity issues

The rest of the article will examine a scenario where we need to troubleshoot an application’s failure to connect to a MySQL database. The application is running in the cluster whereas the database can either be running in the cluster or is an external database, running outside of the cluster.

First, we will want to make sure that the hostname we use to access the database service is resolvable within the cluster. This can be done by starting a busybox container and running the nslookup command, as follows:

kubectl run busybox --image=busybox:1.28 --rm -ti --restart=Never --command -- nslookup my-db-service

Note that you’d want to make sure to use the 1.28 version of busybox since in the later releases nslookup doesn’t work well (see https://github.com/kubernetes/website/pull/9901)

Troubleshooting Using A MySQL Client Container

The first thing we will need to do is to make sure we are able to connect to the database from within the cluster. This test will overrule any network, firewall, or security policies that prevent us from connecting to the database. The following command allows us to start a mysql client and connect to a mysql DB (external or one that is running inside the cluster). Use this to make sure that there is no network issue (firewall, DNS, etc.) that prevent connecting to the database.

mysql -h  myssgsqldb.mysql.database.azure.com -uUser@Host  -pPassword

Can be started as a pod in the Kubernetes cluster by using the following command:

kubectl run -it --rm --image=mysql:5.7 --restart=Never mysql-client -- mysql -h host -uuser@host -ppassword

Note: If you want to force the MySQL Client container to run on the same node as your application pod (to make sure this is not a problem specific to that node), you can add the attribute spec.template. spec.nodeSelector to its pod definition with the value:
kubernetes.io/hostname: <node hostname>

Where <node host name> is the hostname of the node running your application (you can use kubectl get nodes -L kubernetes.io/hostname to check the hostname value of the node)

Capturing POD’s network communications

The following method uses a pattern called sidecar to run a debug container inside the application’s pod, alongside the application container.

Creating a tcpdump container

The following steps will create a tcpdump container. In order for Kubernetes to use it, you’ll have to push it to a docker registry and point your pod to use it.

Alternatively, you can choose to skip this step and use my image on the Docker hub registry, docker.io/cohenuzi01/tcpdump:latest.

Let’s start by creating a container with tcpdump. We will use Ubuntu as the base image and use apt-get to install tcpdump.

Enter the following command into the terminal:

docker build -t tcpdump:latest -<<EOF
FROM ubuntu
RUN apt-get update && apt-get install -y tcpdump
CMD tcpdump -i eth0
EOF

The following output appears, indicating the the image was successfully built:

Sending build context to Docker daemon 2.048 kB
Step 1/3 : FROM ubuntu 
—> 2ca708c1c9cc
Step 2/3 : RUN apt-get update && apt-get install -y tcpdump 
—> Using cache 
—> 5412ced51771
Step 3/3 : CMD tcpdump -i eth0 
—> Running in db7f2d2e16d0 
—> a59987c6e373
Removing intermediate container db7f2d2e16d0
Successfully built a59987c6e373

Next, we tag and push the image into our registry, you should replace “cohenuzi01/tcpdump:latest” with the name of your image, including the registry hostname and port, followed by the repository name, image name and tag:
e.g.
  registry-host:5000/myrepo/tcpdump:latest

docker tag tcpdump:latest cohenuzi01/tcpdump:latest
docker push cohenuzi01/tcpdump:latest

The following output appears:

The push refers to a repository [docker.io/cohenuzi01/tcpdump]
84a96d842479: Pushed
e80c789bc6ac: Pushed
6c3332381368: Pushed
ef1a1ec5bba9: Pushed
a1aa3da2a80a: Pushed
latest: digest: sha256:a335a105ed696547c6fbfbec25e06feda92628d1fa51bf3df560a8993232a5ba size: 1364

Adding the tcpdump container to a pod:

Now that we have the image in a docker registry, accessible to our Kubernetes cluster, we will update the pod of the application we want to debug and add the tcpdump image as a sidecar container. We will name the sidecar container tcpdumper (any name would do). Adding the sidecar container can be done by editing the pod’s specification manifest yaml files and using kubectl apply to apply them, or alternatively, by using the kubectl edit command to update the pod definition in memory:

kubectl edit deployment <applicationPod’s deployment name>

If the pod is a part of a higher Kubernetes construct, like a Deployment, StatefulSet, or a DaemonSet, we should edit the object that contains the pod’s definition deployment. Going forward we will assume it is a deployment for ease of writing.

Note: When using the kubectl edit command, any saved updates will be applied automatically upon exiting the editor.

When editing the pod’s manifest, add the following container to the containers section in the pod definition’s spec (make sure to use the proper line indentation, as those are important in yaml files):

  containers:
  – name: tcpdumper
    image: docker.io/cohenuzi01/tcpdump
    imagePullPolicy: Always
    args:
    – /bin/sh
    – -c
    – tcpdump -nn -s0 -w – port 3306 2>/dev/null |gzip -c |base64
    resources: {}

Notice the command we specify for the container. This tcpdump command is set to capture port 3306 (MySQL port), the output of the tcpdump command is going to be zipped, encoded with base64, and sent to the container’s stdout.

Adding the tcpdumper container into the pod can also be done using a kubectl patch command (similar to the kubectl edit command, the updates to the pod manifest will be immediately applied): 

kubectl patch deployment <applicationPod’s deployment name> --patch '{ "spec": {"template": { "spec": {"containers": [ {"name": "tcpdumper","image": "docker.io/cohenuzi01/tcpdump", "imagePullPolicy": "Always", "args": [ "/bin/sh", "-c","tcpdump -nn -s0 -w - port 3306 2>/dev/null |gzip -c |base64"],"resources": {}}]}}}}'

Note: Once changes to the pod are applied all of the application’s pods (that are using the modified Deployment), will be restarted to use the new configuration.

Saving the output of the tcpdump side-car container to a file

As we mentioned above, the container writes the captured data, compressed and encoded via base64, to its stdout. This output can be saved to a file using the following command:

kubectl logs <applicationPod> -c tcpdumper >myapp.pcap.gz.base64

*make sure you replace <applicationPod> with the application’s pod name.

Extracting the TCPDump Capture Data:

In order to extract the saved data back to a pcap file use the following command:

base64 -d myapp.pcap.gz.base64 |gunzip -c >myapp.pcap

Viewing And Analyzing The Capture Results

In a machine that has Wireshark installed, open the pcap file and review the captured data.

You can also use tshark from the command line to filter down the captured data.
E.g.

Show the lines with a certain field.

tshark -r myapp.pcap -T fields -e mysql.user -Y mysql.user

Or 

Print communication from a certain protocol:

tshark -r myapp.pcap -Y mysql -O mysql

This article will not cover the analysis of the network capture as this is a subject by itself.

Summary

Running applications in Kubernetes presents additional challenges when trying to troubleshoot issues. In this article, we’ve presented just a few troubleshooting tools that we can use when troubleshooting applications running in Kubernetes.