GitOps + Kubernetes: Managing Kubernetes Secrets

Hrittik Roy

Apr 7, 2023

13 min read

Get Started Now

#GitOps + Kubernetes Series

GitOps + Kubernetes Explained
GitOps + Kubernetes: Managing Kubernetes Secrets
Comparing Argo CD vs. Jenkins X vs. Flux vs. Spinnaker
GitOps + Kubernetes: Adopting GitOps in Enterprises

GitOps is a modern approach to software delivery and infrastructure management. It emphasizes version control and collaboration with Git as the source of truth to manage code, configurations, and infrastructure as code (IaC) from a repository. GitOps enables developers to deploy infrastructure and applications in a consistent, automated, and auditable manner.

Teams can use GitOps to automate the entire software delivery pipeline, lowering the risk of human error and ensuring consistency across environments. This is particularly helpful in use cases such as continuous delivery, continuous deployment, and continuous compliance. GitOps also allows teams to bring back features from Git and effortlessly roll back to previous versions or recover from failures.

In this article, you’ll learn how to use Sealed Secrets, a Kubernetes custom resource to manage secrets when using GitOps with Kubernetes through a demo application, along with Argo CD, one of the most popular GitOps tools from the CNCF Cloud Native Landscape.

#Why Sealed Secrets?

GitOps relies heavily on Git for storing application logic, infrastructure manifests, and application manifests. However, securely managing secrets can be a challenge, as storing them in Git repositories can pose a security risk if they are not encrypted and managed properly.

You can use third-party vaults, manually enforce secrets, or use the Sealed Secrets tool from Bitnami, which can help you with encrypting and storing the secrets in your Git repositories, making it easy to follow GitOps principles for all of your objects.

The kubeseal utility, which is bundled with Sealed Secrets, employs public-key cryptography to encrypt your secrets. This ensures that only authorized parties, such as the sealed secret controller installed in your cluster, can decrypt the secrets. The best part about the utility is that it manages the RSA certificate internally, making the process simpler for you.

#Managing Secrets with GitOps

With an understanding of how kubeseal and Sealed Secrets work, getting hands-on experience to understand things in depth will be helpful. This section describes how to create a basic Python application that can read secrets and how to use Sealed Secrets to encrypt the secret store. The deployment process involves using the Argo CD GitOps tool.

Sealed Secrets architecture

If you need to update your secrets, Git commits can be made to update your sealed secrets, which will trigger new deployments and update the secrets.

#Prerequisites

You’ll need a Kubernetes cluster and a code editor to complete the tutorial. For the Kubernetes cluster, this example uses Azure Kubernetes Service (AKS) to deploy the application, but you have the option to run any cluster you prefer. It is important to ensure that your cluster has at least two nodes, as tools like Argo CD can require significant compute. As for the code editor, any editor will do, but the tutorial uses VS Code.

You’ll also need all the following tools to build the application, so make sure you have them installed:

kubectl: A command line tool for managing Kubernetes clusters and resources.
Helm: A package manager for Kubernetes that will help you install the kubeseal controller.
Docker: The go-to platform for building, shipping, and running applications in containers. Don’t forget to create a Docker Hub account to store your container images.
Git: The most popular version control system for tracking changes in source code.
GitHub: One of the most popular web-based platforms for hosting and managing Git repositories, with many additional benefits for your GitOps engine to monitor. Create an account if you don’t already have one.
Python (>=3.7): A popular high-level programming language used for building the sample applications.

#Creating Your Python Application

To understand how secrets work, you first need to create an application that can use the secrets and display them on an endpoint. This tutorial focuses on using Flask, a popular Python web framework.

You can find all the code for the tutorial in this GitHub repository if you want to clone the repository and follow along.

#Importing Packages

To get started, you’ll need to install Flask. You can do this by running the following in a terminal or command prompt:

pip install flask

Once Flask is installed, you can import it into your main.py file, along with the built-in os module, which will be used for accessing environment variables:

# Import Flask module from flask package
from flask import Flask

# Import os module for accessing environment variables
import os

#Creating Your Flask Logic

Create a new instance of a Flask application using the Flask(__name__) constructor, which will be used to define the application’s routes and other settings:

# Create a new instance of the Flask application
app = Flask(__name__)

The following code defines a route for the default home page of the Flask application using the @app.route() decorator:

# Define a route for the default home page of the application
@app.route("/")
def index():
   # Retrieve the value of the COLOR_ENV environment variable
   env_var = os.environ.get("COLOR_ENV")
   # Return a string that includes the value of the COLOR_ENV environment variable
   return f"Hey {env_var}!"

The "/" argument specifies the URL path for the route, and def index() defines a Python function that will handle requests to this route.

Here, the Python function takes the COLOR_ENV environment variable with the help of the os package and then returns the value of the environment variable to the home page.

#Starting the Flask App

With the functions ready to serve your users, you need to add the following block of code to allow Flask to run on a specified port, which helps you assign ports while creating the port mapping:

if __name__ == "__main__":
    app.run(host='0.0.0.0', port='5000')

You can run your application with the command below:

python main.py

#Dockerizing Your Application

With your application ready and running, the next step is to containerize it for your container orchestrator to use. You can do this by using a Dockerfile and building your Docker image. First, freeze your dependencies using pip freeze > requirements.txt and put them in the same directory as main.py so that the correct dependencies are installed during the image build.

With that, the Dockerfile for this tutorial looks like the following:

# Use the official Python 3.8 image from the Alpine Linux repository
FROM python:3.8-alpine

# Copy the contents of the "deploy/" directory from the build context to the "/deploy" directory in the container
COPY deploy/ /deploy

# Set the working directory to "/deploy"
WORKDIR /deploy

# Install the Python dependencies listed in the "requirements.txt" file using pip
RUN pip install -r requirements.txt 

# Expose port 5000 for incoming connections
EXPOSE 5000

# Set the default command to run when the container starts
CMD ["python", "app.py", "--host=0.0.0.0"]

The next step is to build and push your Docker image, which you can do using the following commands if you’re authenticated to your Docker Hub account:

sudo docker build -t <your-username-here>/test-tester:latest . && docker push <your-username-here>/test-tester:latest

If successful, you’ll get the following output:

Docker build and push

#Installing kubeseal

With your Kubernetes cluster up and running, you’ll need to install the kubeseal utility for signed secret creation. This tutorial uses the Linux system on AMD64 architecture. For other systems and architectures, you can read the official documentation.

Use the following code to start the installation on Linux:

wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.20.1/kubeseal-0.20.1-linux-amd64.tar.gz &&
tar -xvzf kubeseal-0.20.1-linux-amd64.tar.gz kubeseal &&
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

To verify the installation, use kubeseal version:

hrittik@hrittik:~$ kubeseal --version
kubeseal version: 0.20.1

With your command line tool up and running, the next step involves installing the sealed secrets controller on your cluster, which is required for sealing and unsealing your secrets.

#Installing the Sealed Secrets Controller

Your controller is responsible for decrypting your secrets when they’re applied to your cluster. Installing it is an easy process with tools like Helm:

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets && 
helm install sealed-secrets -n kube-system --set-string fullnameOverride=sealed-secrets-controller sealed-secrets/sealed-secrets

Run the following command to see the status of your deployment:

kubectl get deployment.apps/sealed-secrets-controller -n kube-system

The output should show 1/1 under READY if things are working as expected:

hrittik@hrittik:~$ kubectl get deployment.apps/sealed-secrets-controller -n kube-system
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
sealed-secrets-controller   1/1     1            1           42h

#Encrypting Secrets with kubeseal

Now that you’ve set up all the necessary tools for creating your sealed secrets, you can create them in two steps.

#Creating a Secret Object

Create a normal Kubernetes secret with the required secret data. For example, let’s say you need to create a secret named “color” with the value “blue” for the key COLOR_ENV. You can create this secret using the following command:

kubectl create secret generic color --dry-run=client --from-literal COLOR_ENV=blue -o yaml > mysecret.yaml

This command creates a Kubernetes secret with the name color and the data COLOR_ENV=blue. The --dry-run=client flag ensures that the secret is not actually created, but the YAML for the secret is output to the console.

The output is then redirected to a file named mysecret.yaml:

apiVersion: v1
data:
  COLOR_ENV: Ymx1ZQ==
kind: Secret
metadata:
  creationTimestamp: null
  name: color

Note: This file is not secure and shouldn’t be committed to Git as the data can be easily decoded with Base64 decoding.

#Creating a SealedSecret Object

Use the following command to have the kubeseal utility encrypt the secret data from the mysecret.yaml file and create a sealed secret in YAML format:

kubeseal --format yaml --controller-name=sealed-secrets-controller --controller-namespace=kube-system -f mysecret.yaml > mysealedsecret.yaml

In the command above, the --format yaml flag specifies that the output should be in YAML format, and the --controller-name and --controller-namespace flags specify the name and namespace of the Kubernetes controller that will manage the sealed secret.

The output of this command is then redirected to a file named mysealedsecret.yaml, which should look something like this:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: color
  namespace: default
spec:
  encryptedData:
    COLOR_ENV: AgAldUW38Ja8d7dGko2wYwMxJn7cLXmq8WRV0JzAXvkpvgmAud2d4hm9+aWz4iaeGITdM35CYABl7EuwTVgIk9A1FvhPMbIdloixSkKQ8OumBKw0G8SjYUTY4hGtWxqDDO7G3eikZFRjwsx8wayplEVk4YnnJEscBQxeCBd156gBruCf923oiYqwdV9oOQ+CoLGv9cfU1Z6vsb7ISIi+5R+P61U+dYNYA4afKYyEQASYupmf8cZ+kBCN+UNud5S0ucKCgr/S+r23EHF1LKtDp7irV/Tkl6teEq/5PUccHhfx6FZFkhDNvZzEFEiYPSlTw/4hOh1xvvpN4TJpBY0tYH2ckmXd02DW0PWPRIIXm6w7JPpvSAvv2yNybuG/QMi/itJWn3TfGgb4DEuei0DXDWLdg6DMWs31HNVqIXHpN/UJNCyHRfOFLf0paBciqd+egFw2hEZB/bHUPWgvtb092JNcYklTPDYTpTLArvxbuCkSdMbBRzLXTP1GxETrXVp9PAXyhbGfXHRX8FP5rMmWctRiiDaxPerKd7uAspnNgzzoTKJVt2rJyfBi1CD1azWloffWJGK36/GCahx6ih4EChQW7KPZrFkm0vMry1EmpkceQRsqmhXRRKrR5a4a5CwbeaUndG2io7sp9DOrFcPJH4M2VhK/VMI9lV5a0LLDN/ae8uiDJjqjty85SN9SWA5Ocx7OWMa8
  template:
    metadata:
      creationTimestamp: null
      name: color
      namespace: default

This sealed secret can now be committed to your source control system and deployed to your Kubernetes cluster without the risk of compromise.

#Creating Manifests

With your sealed secrets and Argo CD set up, you’ll need to create manifests for your application to run.

Deploying your application allows you to generate and control the targeted configuration of your application’s pods. The deployment’s manifest will resemble the following:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deploy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-demo
  template:
    metadata:
      labels:
        app: app-demo
    spec:
      containers:
        - name: app-demo
          image: hrittik/test-tester:latest
          ports:
           - containerPort: 5000
          env:
          - name: COLOR_ENV
            valueFrom:
              secretKeyRef:
                name: color
                key: COLOR_ENV

Take special note of the env: section, as it’s used to reference your color secret from the previous steps and is necessary to reference your secrets in the demo:

          env: # Environment variables to set
          - name: COLOR_ENV
            valueFrom:
              secretKeyRef: # Reference to the "color" secret
                name: color
                key: COLOR_ENV

The service manifest will provide a stable endpoint for the pods in your deployment, and this endpoint enables external users or services to access your pods. In this demo, a LoadBalancer is used in the manifest to expose the application because it provides a simple option for obtaining an IP endpoint on a cloud infrastructure. However, depending on your specific requirements, you can use any other Kubernetes service type to expose your application.

The code for the service manifest is as follows:

apiVersion: v1
kind: Service
metadata:
  name: app-lb-service
spec:
  selector:
    app: app-demo
  ports:
    - port: 5000
      targetPort: 5000
  type: LoadBalancer

Make sure to keep these three manifests in one folder (this demo uses manifests), as this will be required in the upcoming steps. The manifests should also be stored in your source code management (SCM) tool as has been done in this GitHub repository.

#Setting Up GitOps with Argo CD

Argo CD is a popular open source tool used for continuous delivery and deployment of applications to Kubernetes clusters. You’ll be using it to deploy the secret reader that you created in the previous stage. Installing Argo CD is a simple process. It involves creating a namespace for it and applying the installation manifest from the Argo CD stable repository:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

After installing Argo CD, you can expose its web user interface using a load balancer, which is useful when using Argo CD as a managed cloud provider. This can be done by patching the service with the type LoadBalancer:

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

To access the web UI, you need the IP address, which you can get by running the following command:

kubectl get svc -n argocd argocd-server  --output jsonpath='{.status.loadBalancer.ingress[0].ip}'

Visiting the resulting IP address should take you to a login screen:

Argo CD portal

The default username for Argo CD is admin, and to obtain the password, you can run the following command:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

#Deploying Your Applications

To deploy the application, access your dashboard and click CREATE APPLICATION or NEW APP to start your deployment process:

Argo create application

Choose any name for the Application Name field and set the Project Name to default:

Basic details

To configure the source, you should specify the GitHub repository link that contains the manifests and select the folder that contains the manifests as the path, as Argo CD will deploy your application using the manifests found in this folder:

Source repository

Once you have configured these details, you can click CREATE in the top left corner to register the application as below:

Repository added

For the next step, start the sync process to implement the manifest in your cluster. You can do this by clicking SYNC and then SYNCHRONIZE:

Synchronize state

After a short wait, the application will be successfully deployed to your cluster, and you can observe all the objects in the cluster:

Cluster deployment complete

To access the website, navigate to the load balancer, which can be found in the network tab:

Network flow

Visit the page, and then you can see your application and read your sealed secret:

Endpoint showing secret

#Updating Your Secrets

When you need to update your secrets, there are two steps involved. First, you need to create a new sealed secret with the updated information you want to use in your application. You can follow the same steps as you did for creating a sealed secret with COLOR_ENV=night or something else of your choice.

Once you have created the new sealed secret, you need to push the changes to the Git repository where your application’s manifests are stored. This will notify Argo CD that there has been a change in the state of the application.

After you have pushed the changes, you can go to the Argo CD dashboard and check the state of your application. You should see that your cluster is out of sync:

Change state

To update the secrets, you need to trigger redeployment. For that, click SYNC and then SYNCHRONIZE with the FORCE and REPLACE flags:

Secret update

The steps will trigger Argo CD to apply the changes to the Kubernetes cluster, including updating the secrets and triggering redeployment of dependent resources like your pods.

Finally, access your web page using the new endpoint to view the updated secrets:

Updated manifests

You manually applied the changes in this example, but in different environments, you can force Argo CD to automate the whole process.

#Conclusion

Sealed secrets offer an effective solution for securing sensitive information in Kubernetes clusters. They ensure that secrets are protected and only accessible to authorized users and your cluster, while maintaining the advantages of Git and GitOps without any security issues. kubeseal and similar tools have made the process of creating and managing secrets much simpler, allowing companies to easily adopt them.

However, managing multiple virtual Kubernetes clusters can be more challenging for enterprises, particularly in terms of cost efficiency and scalability.

Loft addresses this challenge by offering a virtualization solution that enables enterprises to create virtual Kubernetes clusters and gain multitenancy, velocity, and self-service for their developers without compromising governance. With Loft, enterprises can manage their virtual clusters easily, reduce infrastructure costs, and ensure that their developers have the tools they need to be productive and efficient.