Kubernetes Readiness Probes - Examples & Common Pitfalls

Kubernetes is a great platform to deploy our microservices and applications to. One of the excellent features is that pods are restarted or removed from a service when they don't work correctly. Kubernetes needs our help to understand if a pod is working or not. This is configured via Container Probes.

Any application can be in an unhealthy state due to bugs, missing dependencies on external resources, and so on. To make sure our traffic flows correctly, we need to make sure probes are configured.

Kubernetes Probes Series

Kubernetes Probes: Startup, Liveness, Readiness

Kubernetes Startup Probes - Examples & Common Pitfalls

Kubernetes Liveness Probes - Examples & Common Pitfall

Kubernetes Readiness Probes - Examples & Common Pitfalls

Probes

Probes are kubelet's answer to the health checks, there are three handlers:

Each type of probe has common configurable fields:

These parameters need to be configured per your application's spec.

Exec Probe

Exec action has only one field, and that is command. Exit status of the command is checked, and the status of zero (0) means it is healthy, and other value means it is unhealthy.

readinessProbe:
  initialDelaySeconds: 1
  periodSeconds: 5
  timeoutSeconds: 1
  successThreshold: 1
  failureThreshold: 1
  exec:
    command:
      - cat
      - /etc/nginx/nginx.conf

TCP Probe

We need to define host and port parameters, host parameter defaults to the cluster-internal pod IP.

readinessProbe:
  initialDelaySeconds: 1
  periodSeconds: 5
  timeoutSeconds: 1
  successThreshold: 1
  failureThreshold: 1
  tcpSocket:
    host:
    port: 80

HTTP Probe

HTTP Probe has additional options to configure.

Tip: If you need to set the Host header of the HTTP, please do so on httpHeaders, instead of the host parameter.

        readinessProbe:
          initialDelaySeconds: 1
          periodSeconds: 2
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 1
          httpGet:
            host:
            scheme: HTTP
            path: /
            httpHeaders:
            - name: Host
              value: myapplication1.com
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5

Readiness Probes in Kubernetes

Kubernetes runs readiness probes to understand when it can send traffic to a pod, i.e., to transition the pod to Ready state. For example, if a pod is used as a backend endpoint for a service, a readiness probe will determine if the pod will receive traffic or not. The readiness probe is executed throughout the pod's lifetime; this means that readiness probes continue to be executed after reaching the Ready state. Also, our application can make itself unavailable for maintenance or for some background tasks as well by responding with different values to the probe.

A pod transitions the following states:

Common Pitfalls for Readiness Probes

Readiness probes check the configured action for the duration of the container's lifetime; therefore, any disruption or delay in answering can cause service disruption. Sometimes the application needs to read a large amount of data experiencing latency issues or make an expensive calculation, which might prevent such an answer from being delivered. This behavior should be considered when configuring readiness probes. Testing thoroughly before going live with a readiness probe is highly recommended.

Cascading Failures

Consider this, we have configured a readiness probe with HTTPGet, and our application checks the database connection before answering the probe. Let's suggest there is a database problem; in this case, all pods will be unreachable unless the database is back. Do we want this behavior? Certainly no, we could feed the database error to the front end and notify the user of the unavailability; even if we increase failureThreshold we might not be able to avoid a cascading failure. Avoiding external dependencies on readiness probe answers is what I would recommend in any case.

External Dependencies

Let's assume that you have a key/value store used for caching, and your readiness probe endpoint checks this connection as well. The application is capable of running without the key/value store as it will only slow down but it will run a full database query every time it needs to access data. In this case, if the readiness probe fails because of the key/value store not being available, your whole application will be down for some time -until you fix the key/value store which is certainly not desirable.

You need to clearly design your probes considering how your application behaves and how the application responds when there is a failure. You might want to restart the pod (via liveness checks) or remove it from service with a readiness probe.

Example: Sample Nginx Deployment

To see readiness probes in action, we can configure a sample app, a simple NGINX web server in this case. Here we have defined a simple deployment configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-probes
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80
          readinessProbe:
            initialDelaySeconds: 1
            periodSeconds: 2
            timeoutSeconds: 1
            successThreshold: 1
            failureThreshold: 1
            httpGet:
              host:
              scheme: HTTP
              path: /
              httpHeaders:
                - name: Host
                  value: myapplication1.com
              port: 80

Save this configuration to a file called k8s-probes-deployment.yaml and apply it with kubectl apply -f k8s-probes-deployment.yaml.

apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx
  name: nginx
  namespace: default
spec:
  ports:
    - name: nginx-http-port
      port: 80
  selector:
    app: nginx
  sessionAffinity: None
  type: NodePort

Save this configuration to a file called k8s-probes-svc.yaml and apply it with kubectl apply -f k8s-probes-svc.yaml.

There is no separate endpoint for readiness probes, but we can access events using the kubectl describe pods <POD_NAME> command, for example, to get the current status.

Use kubectl get pods command to see the pods' status.

kubectl get pods

Pods and their status and ready states will be displayed, our pod is running as expected.

NAME                          READY   STATUS    RESTARTS   AGE
k8s-probes-7d57f897dd-td2q2   1/1     Running   0          1m23s

With kubectl describe pod <POD_NAME>

kubectl describe pod k8s-probes-755b4bf57c-jgcqw

At the bottom of the output Events will be displayed:

Events:
  Type    Reason     Age   From               Message----------------
  Normal  Scheduled  15m   default-scheduler  Successfully assigned default/k8s-probes-755b4bf57c-jgcqw to k8s-probes
  Normal  Pulling    15m   kubelet            Pulling image "nginx"
  Normal  Pulled     15m   kubelet            Successfully pulled image "nginx" in 1.418647753s
  Normal  Created    15m   kubelet            Created container nginx
  Normal  Started    15m   kubelet            Started container nginx

Let's look at the endpoints with kubectl get endpoints command.

kubectl get endpoints

We can see Nginx service has an endpoint.

NAME         ENDPOINTS           AGE
kubernetes   10.132.0.42:16443   8h
nginx        10.1.85.203:80      6m11s

To see more detail we can use kubectl describe endpoints nginx command.

Name:         nginx
Namespace:    default
Labels:       app=nginx
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2020-11-13T20:34:13Z
Subsets:
  Addresses:          10.1.85.203
  NotReadyAddresses:  <none>
  Ports:
    Name             Port  Protocol-------------
    nginx-http-port  80    TCP

Configure the readiness probe's port parameter to 81 and apply the configuration.

Again let's check the pods status

kubectl get pods

As we have updated our deployment by default, a new pod is created, and as you can see, it is not in the ready state.

NAME                          READY   STATUS    RESTARTS   AGE
k8s-probes-755b4bf57c-jgcqw   1/1     Running   0          13m
k8s-probes-7d57f897dd-td2q2   0/1     Running   0          2m49s

Here status is running however READY state determines our pod is not ready yet. Let's explore the status and events with kubectl describe pods <POD_NAME> command.

kubectl describe pod k8s-probes-7d57f897dd-td2q2

We can see that the readiness probe failed due to the connection being refused therefore pod will not receive any traffic.

Events:
  Type     Reason     Age                From               Message----------------
  Normal   Scheduled  58s                default-scheduler  Successfully assigned default/k8s-probes-7d57f897dd-td2q2 to k8s-probes
  Normal   Pulling    57s                kubelet            Pulling image "nginx"
  Normal   Pulled     56s                kubelet            Successfully pulled image "nginx" in 1.211289526s
  Normal   Created    56s                kubelet            Created container nginx
  Normal   Started    56s                kubelet            Started container nginx
  Warning  Unhealthy  4s (x10 over 49s)  kubelet            Readiness probe failed: Get "http://10.1.85.202:81/": dial tcp 10.1.85.202:81: connect: connection refused

Let's check the status directly:

kubectl get pods k8s-probes-7d57f897dd-td2q2 -o jsonpath='{.status.containerStatuses[0].state}'

The state is "running".

{"running":{"startedAt":"2020-11-17T11:47:21Z"}}

Let's get the ready status:

kubectl get pods k8s-probes-7d57f897dd-td2q2 -o jsonpath='{.status.containerStatuses[0].ready}'

As you can see, the container is not ready.

false

Let's look at endpoints again:

kubectl get endpoints

Services and related endpoints are listed:

NAME         ENDPOINTS           AGE
kubernetes   10.132.0.42:16443   8h
nginx                            11m

Here we can see that our Nginx service doesn't have any endpoints as the container is not ready to take the traffic.

Conclusion

We have seen the effects of the readiness probe and explored the parameters we can configure. Although we worked on the HTTP check, the concepts shown here apply to the other checks as well. You need to understand your application's architecture and dependencies to be able to configure and operate readiness probes. Knowing your application is crucial as badly configured readiness probes can render your application unavailable. And please do not forget that readiness probes are run throughout the container's lifetime.

Additional Articles You May Like:

Photo by Ildefonso Polo on Unsplash