Kubernetes Startup Probes - Examples & Common Pitfalls

Kubernetes brought an excellent deployment platform to work on. Even monolithic applications can be run in a container. For some of these monolithic applications and for some microservices, a slow start is a problem. Even if we configure readiness and liveness probes using initialDelaySeconds, this is not an ideal way to do this. For this specific problem, startup probes are developed.

Kubernetes Probes Series

Kubernetes Probes: Startup, Liveness, Readiness

Kubernetes Startup Probes - Examples & Common Pitfalls

Kubernetes Liveness Probes - Examples & Common Pitfall

Kubernetes Readiness Probes - Examples & Common Pitfalls

Probes

Probes are executed by kubelet to determine pods' health.

All three types of probes have common settings to configure.

Configuring these parameters vital and crucial.

Exec Probe

Exec probe runs a command inside the container as a health check; the command's exit code determines the success.

startupProbe:
  initialDelaySeconds: 1
  periodSeconds: 5
  timeoutSeconds: 1
  successThreshold: 1
  failureThreshold: 1
  exec:
    command:
      - cat
      - /etc/nginx/nginx.conf

TCP Probe

TCP probe checks if the specified port is open or not; an open port points to success.

startupProbe:
  initialDelaySeconds: 1
  periodSeconds: 5
  timeoutSeconds: 1
  successThreshold: 1
  failureThreshold: 1
  tcpSocket:
    host:
    port: 80

HTTP Probe

HTTP probe sends an HTTP GET request with defined parameters.

HTTP Probe has additional options to configure.

Tip: Host header should be set in httpHeaders.

        startupProbe:
          initialDelaySeconds: 1
          periodSeconds: 2
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 1
          httpGet:
            host:
            scheme: HTTP
            path: /
            httpHeaders:
            - name: Host
              value: myapplication1.com
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5

Startup Probes in Kubernetes

Startup probes are a newly developed feature supported as a beta in Kubernetes v.1.18.
These probes are very useful on slow-start applications; it is much better than increasing initialDelaySeconds on readiness or liveness probes.
Startup probe allows our application to become ready, joined with readiness and liveness probes, it can dramatically increase our applications' availability.

Example: Sample Nginx Deployment

Let's deploy Nginx as a sample app and see startup probes in action.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-probes
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80
          startupProbe:
            initialDelaySeconds: 1
            periodSeconds: 2
            timeoutSeconds: 1
            successThreshold: 1
            failureThreshold: 1
            exec:
              command:
                - cat
                - /etc/nginx/nginx.conf

Create a file and paste the above, if you named the file k8s-probes-deployment.yaml and apply it with kubectl apply -f k8s-probes-deployment.yaml command.

apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx
  name: nginx
  namespace: default
spec:
  ports:
    - name: nginx-http-port
      port: 80
  selector:
    app: nginx
  sessionAffinity: None
  type: NodePort

Create a file and paste the above, if you named the file k8s-probes-svc.yaml and apply it with kubectl apply -f k8s-probes-svc.yaml command.

Common Pitfalls for Startup Probes

Although it is great to have such a probe, especially on a legacy application or any application that might take a while to become ready, it is quite important that parameters must be correctly configured. Otherwise, it can break our application and availability.

Restart Loop

Startup probes - if misconfigured- can cause a loop of restarts. Let's assume we have an application running on java, and this application takes a while to become ready. If we don't allow enough time for the startup probe to get a successful response, the kubelet might restart the container prematurely, causing a loop of restarts.

Troubleshooting Startup Probes

After applying the deployment file, we should see the pod is up and running; let's have a look.

kubectl get pods

We can see that the pod is up and running.

NAME                          READY   STATUS    RESTARTS   AGE
k8s-probes-6cbf7ccbf8-97hz5   1/1     Running   0          7s

Have a look at the events using the kubectl describe pods command.

kubectl describe pods k8s-probes-6cbf7ccbf8-97hz5

Let's scroll down to events.

Events:
  Type    Reason     Age   From               Message----------------
  Normal  Scheduled  3s    default-scheduler  Successfully assigned default/k8s-probes-6cbf7ccbf8-97hz5 to k8s-probes
  Normal  Pulling    2s    kubelet            Pulling image "nginx"
  Normal  Pulled     1s    kubelet            Successfully pulled image "nginx" in 925.688112ms
  Normal  Created    1s    kubelet            Created container nginx
  Normal  Started    1s    kubelet            Started container nginx

As you can see, the probe was successful, and no error or warning event is recorded.

Let's check the applied configuration.

kubectl describe pods k8s-probes-6cbf7ccbf8-qcpt7 | grep Startup

We can see that the startup probe is configured with the parameters we have set.

    Startup:        exec [cat /etc/nginx/nginx.conf] delay=1s timeout=1s period=2s #success=1 #failure=1

Now change the command parameter to /etc/nginx/nginx.conf-dont-exists on the deployment file and apply with kubectl apply -f k8s-probes-deployment.yaml.

Let's check the events of the pod.

kubectl describe pods k8s-probes-5fcc896b6f-97wpg

Let's scroll down to events.

...
Events:
  Type     Reason     Age                     From               Message----------------
  Normal   Scheduled  13m                     default-scheduler  Successfully assigned default/k8s-probes-5fcc896b6f-97wpg to k8s-probes
  Normal   Pulled     13m                     kubelet            Successfully pulled image "nginx" in 944.990287ms
  Normal   Pulled     12m                     kubelet            Successfully pulled image "nginx" in 972.83673ms
  Normal   Pulled     12m                     kubelet            Successfully pulled image "nginx" in 958.559546ms
  Normal   Pulled     11m                     kubelet            Successfully pulled image "nginx" in 1.056812046s
  Normal   Created    11m (x4 over 13m)       kubelet            Created container nginx
  Normal   Started    11m (x4 over 13m)       kubelet            Started container nginx
  Warning  Unhealthy  11m (x4 over 13m)       kubelet            Startup probe failed: cat: /etc/nginx/nginx.conf-dont-exists: No such file or directory
  Normal   Pulling    9m44s (x5 over 13m)     kubelet            Pulling image "nginx"
  Normal   Killing    8m21s (x5 over 12m)     kubelet            Container nginx failed startup probe, will be restarted
  Warning  BackOff    3m41s (x13 over 7m16s)  kubelet            Back-off restarting the failed container

As you can see startup probe executed the command; however, due to a non-existent file, the command returned a non-zero exit code.

Conclusion

Startup probes are very helpful to determine our application has started correctly.
We have explored the config options on a sample Nginx application; we have checked the configuration file as a probe; this is an example of a dynamic configuration generation application, i.e., assume that Nginx configuration is generated dynamically using etcd or similar key-value store.
Although very useful, we have also explored the side effects of the startup probes. Make sure you allow enough time to let your application startup.

Further Reading

Photo by Braden Collum on Unsplash