Ephemeral Kubernetes Environments on CI/CD Systems with vCluster

An increasing number of GitOps teams are recognizing the need to construct their own automated Kubernetes CI/CD platform. This trend arises from the desire to faithfully replicate the production environment, create customized app testing workflows, gain better control over resources, achieve better security, and manage costs effectively.

In this sense, vCluster is a powerful open source solution developed by Loft that allows organizations to launch virtual Kubernetes clusters with a single command. Such convenience can be useful for several use cases, including achieving true Kubernetes multitenancy isolation or building ephemeral Kubernetes environments for CI/CD purposes. This tutorial focuses on the latter.

This tutorial provides an overview of the challenges involved in building a Kubernetes-based CI/CD platform. Additionally, you'll learn what makes vCluster an ideal tool for Kubernetes CI/CD pipelines. Finally, you'll gain firsthand experience leveraging vCluster for efficient CI/CD in your Kubernetes environment.

Challenges in Building a Kubernetes-Based CI/CD Platform

Building a Kubernetes-based CI/CD platform presents several challenges. The time it takes to tear down and start up environments can substantially disrupt your workflow. Complexity in resource management often arises due to the dynamic nature of Kubernetes, with resource allocation and optimization becoming a tedious task.

Moreover, achieving consistency and security across multiple environments is another hurdle, as each environment may require different dependencies or access control levels. Managing persistent data could also pose a significant challenge due to the ephemeral nature of Kubernetes pods, which can lead to data loss if not handled correctly.

So, how can vCluster help overcome these challenges?

Why Consider vCluster for Your CI/CD Pipeline?

Introducing you to vCluster deserves a dedicated blog post. Therefore, the best way to review its suitability for CI/CD pipelines is to contrast its features with the challenges described in the previous section.

The list of vCluster benefits goes on and on. However, nothing beats experiencing such benefits firsthand.

Getting Started with vCluster

In this section, you'll install the vcluster CLI, create a virtual cluster within your Kubernetes cluster, and configure vCluster to suit your specific organization's requirements.

Prerequisites

You'll need the following for the tutorial:

Setting Up the vCluster CLI

You can use the vCluster documentation to install the vcluster command line tool on Mac (Intel/AMD/ARM), Linux (AMD/ARM), or Windows machines.

For instance, if you have a Mac, you can install vcluster locally using Homebrew as follows:

brew install vcluster

To check that everything went as expected, run vcluster --version:

$ vcluster --version
vcluster version 0.15.7

Spinning Up Your First Virtual Cluster

You can now start deploying virtual clusters to your Kubernetes cluster.

You'll notice that the documentation mentions three different ways to create a virtual cluster: the CLI, Helm charts, and kubectl. This provides you with the flexibility to use vCluster in different scenarios.

If you want to experiment with vCluster, the vcluster CLI is ideal. Just run vcluster create my-vcluster on your machine. The output should be similar to the following:

info   Creating namespace vcluster-my-vcluster
info   Detected local kubernetes cluster k3d. Will deploy vcluster with a NodePort & sync real nodes
info   Chart not embedded: "open charts/vcluster-0.15.7.tgz: file does not exist", pulling from helm repository.
info   Create vcluster my-vcluster...
info   execute command: helm upgrade my-vcluster https://charts.loft.sh/charts/vcluster-0.15.7.tgz --kubeconfig /var/folders/rc/0g2r30c52j152ph518g25mc40000gn/T/3979167366 --namespace vcluster-my-vcluster --install --repository-config='' --values /var/folders/rc/0g2r30c52j152ph518g25mc40000gn/T/1848473749
done √ Successfully created virtual cluster my-vcluster in namespace vcluster-my-vcluster
info   Waiting for vcluster to come up...
info   Starting proxy container...
warn   Error exposing local vcluster, will fallback to port-forwarding: error starting kind proxy:  exec: "docker": executable file not found in $PATH
done √ Switched active kube context to vcluster_my-vcluster_vcluster-my-vcluster_k3d-loft-tutorial
warn   Since you are using port-forwarding to connect, you will need to leave this terminal open
- Use CTRL+C to return to your previous kube context
Forwarding from 127.0.0.1:10624 -> 8443
Forwarding from [::1]:10624 -> 8443
Handling connection for 10624

As you can see, several things happen behind the scenes:

Now, all you have to do is open another terminal, as indicated. If you run kubectl get ns, you will get an output like this:

$ kubectl get ns  
NAME              STATUS   AGE
default           Active   14m
kube-system       Active   14m
kube-public       Active   14m
kube-node-lease   Active   14m

Congratulations! You're now inside a virtual cluster! To verify it, close the terminal where the port-forward is running and list namespaces again:

kubectl get ns
NAME                   STATUS   AGE
default                Active   24h
kube-system            Active   24h
kube-public            Active   24h
kube-node-lease        Active   24h
vcluster-my-vcluster   Active   16m

The namespace vcluster-my-vcluster is where the virtual cluster lives. Alternatively, you can discover active virtual clusters using vcluster list:

$ vcluster list
NAME          NAMESPACE              STATUS    CONNECTED   CREATED                         AGE     CONTEXT            
my-vcluster   vcluster-my-vcluster   Running               2023-09-17 12:26:46 -0400 -04   20m6s   k3d-loft-tutorial

To recap, the vcluster CLI uses kubectl and helm to facilitate deploying virtual clusters. Put another way, you could also deploy a virtual cluster by running the following:

kubectl create namespace vcluster-my-vcluster
helm template my-vcluster vcluster --repo https://charts.loft.sh -n vcluster-my-vcluster | kubectl apply -f -

This approach is interesting because it opens the doors for you to deploy virtual clusters programmatically.

Deploying Virtual Clusters Programmatically

Deploying virtual clusters using the vcluster CLI is straightforward. However, manually managing your team members' virtual clusters is inefficient. In this sense, GitOps teams can leverage Terraform's Kubernetes and Helm providers to deploy virtual clusters programmatically.

Suppose you need to deploy three virtual clusters. To do so, create a new folder, navigate to it, and create a file called main.tf with the following code:

terraform {

  backend "kubernetes" {
    secret_suffix = "common"
    config_path   = "~/.kube/config"
  }
}

provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}

resource "helm_release" "multiple-vcluster" {
  count = 3
  namespace = "vcluster-${count.index}"
  name      = "vcluster-${count.index}"
  repository       = "https://charts.loft.sh"
  chart            = "vcluster"
  create_namespace = true
}

The first two code blocks instruct Terraform to use kubernetes and helm providers. The third code block declares the multiple-vcluster resource responsible for deploying three virtual clusters using Helm.

You'll also need to initialize Terraform using terraform init and apply the changes with terraform apply. Run kubectl get ns when the process finishes, and you will see the namespaces corresponding to each virtual cluster:

...
vcluster-my-vcluster   Active   3h48m
vcluster-0             Active   2m12s
vcluster-1             Active   2m12s
vcluster-2             Active   2m12s

As before, you can connect via port-forward to any virtual cluster using vcluster connect. You could also use other methods to expose the virtual cluster, such as LoadBalancer, NodePort, or ingress, if required.

For instance, to connect to vcluster-1 using the CLI, run vcluster connect vcluster-1:

$ vcluster connect vcluster-1
done √ Switched active kube context to vcluster_vcluster-1_vcluster-1_k3d-loft-tutorial
warn   Since you are using port-forwarding to connect, you will need to leave this terminal open
- Use CTRL+C to return to your previous kube context
- Use `kubectl get namespaces` in another terminal to access the vcluster
Forwarding from 127.0.0.1:10404 -> 8443
Forwarding from [::1]:10404 -> 8443
Handling connection for 10404

Thanks to Terraform, you can create and destroy virtual clusters at your convenience by simply changing count = 3 to the desired number of virtual clusters and running terraform apply afterward. That said, deploying virtual clusters using Terraform and Helm charts provides additional benefits, such as the flexibility to configure vCluster to your requirements using values.yaml.

Customizing vCluster Settings

In the previous examples, vCluster was deployed using its default values. However, let's say your team needs to test an application on a different version of Kubernetes. You can use values.yaml to tell helm which version of Kubernetes vCluster will use.

Let's test it. Edit main.tf to look like the following:

terraform {

  backend "kubernetes" {
    secret_suffix = "common"
    config_path   = "~/.kube/config"
  }
}

provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}

resource "helm_release" "vcluster-k3s1281" {
  namespace = "vcluster-k3s1281"
  name      = "vcluster-k3s1281"
  repository       = "https://charts.loft.sh"
  chart            = "vcluster"
  create_namespace = true
  values = [file("${path.module}/vcluster-k3s1281.yaml")]
}

Next, create a file called vcluster-k3s1281.yaml in the same directory and paste the following inside:

vcluster:
  image: rancher/k3s:v1.28.1-k3s1 # Image to use for the virtual cluster

If you run terraform apply now, it will create a new virtual cluster that will use the settings specified in vcluster-k3s1281.yaml. Once you connect to the virtual cluster, you can verify it:

$ kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.1+k3s1

This is a simple example, but you can adjust many other parameters. For instance, you can dramatically increase security in multitenant virtual cluster environments by enabling isolated mode or ensure that no virtual cluster consumes too many resources in the host cluster by changing the default ResourceQuota values. To look at all the values that you can adjust, download the vCluster Helm chart using helm pull and check values.yaml.

Now that you have a virtual cluster running the latest version of Kubernetes, you just have to deploy the app you want to test. The best way to do this is to implement a CI/CD pipeline within the virtual cluster.

How to Use vCluster to Create CI/CD Pipelines

Implementing a CI/CD pipeline within the virtual cluster allows you to create an automated workflow with steps that will be executed every time you push a change to the app Git repository.

vCluster enables organizations to embrace GitOps and DevOps best practices without any extra effort from your team. In fact, you've already seen how vCluster fits seamlessly into your Terraform IaC workflow. This allows organizations to manage their entire infrastructure declaratively and even create CI/CD pipelines to deploy virtual clusters on demand.

Furthermore, you could use vCluster to create customized ephemeral CI/CD workflows for your developers and address use cases like the one described in the previous section. Argo CD is a great tool for this multitenant scenario.

Start by deploying the Argo CD operator within the last virtual cluster you created (vcluster-k3s1281):

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/core-install.yaml

Once all components are installed, use the argocd CLI to extract the temporary password generated during Argo CD deployment. Copy the password, as you'll need it to access the Argo CD UI:

argocd admin initial-password -n argocd

Next, port-forward argocd-server to access the web UI:

kubectl port-forward svc/argocd-server -n argocd 8080:443

Enter admin as the username, and use the password you just obtained. This should take you to the main dashboard:

Click the NEW APP button located at the top left.

Follow the instructions to create apps via UI to deploy and synchronize the guestbook demo app on your virtual cluster. Basically, you'll need to enter guestbook as the application name, use the default project, leave the sync policy as "Manual," and set the repository URL to https://github.com/argoproj/argocd-example-apps.git.

Once deployed, you can sync the app by pressing the SYNC button. The result should look similar to the following:

That's all! You just implemented CI/CD within a virtual cluster that uses the latest Kubernetes version.

From here on, the possibilities are endless. For example, your organization could build an end-to-end CI/CD pipeline that launches virtual clusters with Argo CD or any other dev or testing environment you require, all programmatically.

Conclusion

This tutorial demonstrated the flexibility and convenience that vCluster offers when creating ephemeral CI/CD environments in Kubernetes. Loft can significantly amplify these capabilities, empowering GitOps teams and providing enterprise features to elevate the virtual cluster experience. Loft provides high availability (HA), a top-tier UI, CLI, API, and air-gapped and VPC environment support. Furthermore, Loft's multicluster, multicloud, and multiregion support and cross-cluster sync capabilities provide a robust platform for your virtual cluster environments. Loft isn't just a tool but an essential partner that takes your virtual cluster experience to new heights of efficiency and productivity. Embrace Loft and experience a seamless, comprehensive approach to managing your virtual cluster environments.

If you want to know more about vCluster and Loft, you can check out how to create virtual Kubernetes clusters with vCluster and Helm or explore the benefits of virtual clusters for CI/CD testing.