Bootstrapping Virtual Kubernetes Clusters with vcluster

vcluster is an open source tool for creating and managing virtual Kubernetes clusters. A virtual Kubernetes cluster runs within a namespace on a host cluster, but it appears to the user as if it's a full-blown, standalone cluster.

But what happens once the virtual cluster is created? Like any new cluster, there's usually a lot more that needs to happen before it’s usable. The gap between a newly provisioned cluster and a working dev environment for one of your developers may be very large. You may need to install tools to manage other setups or configurations, for example. 

In this hands-on tutorial, we'll look at how to configure newly provisioned virtual clusters with init manifests (Kubernetes YAML) and Helm charts that are applied automatically after the virtual cluster is created. This feature lets you configure your virtual clusters fully or install other tools that will configure them, depending on your needs.

If you're new to vcluster, you may want to go through our introductory hands-on tutorial first.

Prerequisites

The only prerequisite is a Kubernetes cluster that you can run the virtual cluster in. Any Kubernetes cluster, local or remote, should work.

Install the vcluster CLI

If you use a Mac with Homebrew, you can install the CLI with this command:

brew install vcluster

For other platforms, see the quickstart in the docs. The CLI is supported on macOS, Linux, and Windows.

Apply a manifest on initialization

The manifests we're referring to are any valid Kubernetes YAML that you could apply with `kubectl apply -f`. Being able to apply Kubernetes YAML immediately after the cluster is created allows us to automate just about anything we'd want to do when creating or configuring a cluster.

Let's look at an example. Create a file called values.yaml with the following contents:

init:
  manifests: |-
    apiVersion: v1
    kind: Namespace
    metadata:
      name: nginx---apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nginx-deployment
      namespace: nginx
    spec:
      selector:
        matchLabels:
          app: nginx
      replicas: 2
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx:1.25.1
            ports:
            - containerPort: 80

Let's break down what's happening here.

The first two lines specify that this is the `init.manifest` section of the vcluster YAML file and that what follows are the manifests to be applied.

The next four lines create a namespace in the virtual cluster called nginx.

Then there is a separator, which needs to be present between each manifest.

The rest of the file creates a deployment to launch two NGINX pods in the nginx namespace that was just created. It will launch two pods in the virtual cluster with a label of nginx.

Next, let's create a virtual cluster that will automatically apply those manifests. The manifests will be applied in the order they appear in the file.

Make sure you are pointed at the kube context for the host cluster you use. Then create the virtual cluster.

vcluster create init-tutorial -f values.yaml

This will create a virtual cluster with the name init-tutorial, and a namespace to run it in.

You should see output like:

info   Creating namespace vcluster-init-tutorial
info   failed to find IPv6 service CIDR: couldn't find host cluster Service CIDR ("Service "test-service-9wpzn" is invalid: spec.clusterIPs[0]: Invalid value: []string{"2001:DB8::1"}: IPv6 is not configured on this cluster")
info   Detected local kubernetes cluster docker-desktop. Will deploy vcluster with a NodePort & sync real nodes
info   Create vcluster init-tutorial...
info   execute command: helm upgrade init-tutorial /var/folders/gy/d3_c4t1x731_hl8qtrfkhr_h0000gn/T/vcluster-0.15.2.tgz-86186483 --kubeconfig /var/folders/gy/d3_c4t1x731_hl8qtrfkhr_h0000gn/T/3759283775 --namespace vcluster-init-tutorial --install --repository-config='' --values /var/folders/gy/d3_c4t1x731_hl8qtrfkhr_h0000gn/T/1562320995 --values values.yaml
done √ Successfully created virtual cluster init-tutorial in namespace vcluster-init-tutorial
info   Waiting for vcluster to come up...
done √ Switched active kube context to vcluster_init-tutorial_vcluster-init-tutorial_docker-desktop
- Use `vcluster disconnect` to return to your previous kube context
- Use `kubectl get namespaces` to access the vcluster

The exact output will depend a bit on how your host cluster is set up.

If your host cluster is local (Docker Desktop, Minikube, etc.), vcluster will configure it to use a NodePort and connect to it automaticaly.

For remote clusters, vcluster will connect using port forwarding.

Now that we're connected to the virtual cluster, we can run kubectl commands against it like any other cluster. Let's look at the pods that are running.

kubectl get pods -n nginx

You should see output like:

NAMESPACE     NAME                                READY   STATUS    RESTARTS   AGE
nginx         nginx-deployment-775b6549b5-nr8jh   1/1     Running   0          68s
nginx         nginx-deployment-775b6549b5-vcx7w   1/1     Running   0          68s

In this example the two NGINX pods are running in the nginx namespace. The pods and the namespace were created automatically as soon as the vcluster was created.

Now let's disconnext from the virtual cluster and delete it to clean up.

vcluster disconnect

Disconnecting will switch you back to the context of the host cluster. Then run:

vcluster delete init-tutorial

Apply a Helm chart on initialization

Now let's create a NGIX namespace and deployment again, but this time instead of creating the deployment manually, we'll install NGINX with a public Helm chart.

First, edit the values.yaml. Remove the previous contents and set them to:

init:
  helm:
    - chart:
        name: nginx
        repo: https://charts.bitnami.com/bitnami
      # optional field
      values: |-
        replicaCount: 2
      release:
        name: nginx
        namespace: nginx

As you can see, we specify the chart name and repo first. Then we can optionally include Helm values for the chart inline (we've set the replica count to 2). And last, the release section contains the name (the pods will use this name) and the namespace that's being used.

Create the vcluster:

vcluster create init-tutorial-2 -f values.yaml

Once again you should be automatically connected to the vcluster. And we can view the NGINX pods with:

kubectl get pods -n nginx

Your output should look like:

NAME                     READY   STATUS    RESTARTS   AGE
nginx-769c898b4f-g922n   1/1     Running   0          108s
nginx-769c898b4f-vwh6w   1/1     Running   0          108s

And that's the process for automatically applying a Helm chart when the vcluster is initialized.

We can clean up once again with:

vcluster disconnect
vcluster delete init-tutorial-2

You can also apply private Helm charts when vclusters are initialized from a private repo or your local filesystem. For more info, see the docs.

Troubleshooting

A quick note on troubleshooting. When a virtual cluster is initialized with manifests or Helm charts, you won't see any errors that result in the output from the `vcluster create` command. You can see more information in the host cluster's logs, though.

First, if you are connected to the virtual cluster, disconnect with the vcluster disconnect command. Then run this command against the host cluster:

kubectl logs -n vcluster-NAME -l app=vcluster,release=NAME -c syncer

Substitute the name you gave your virtual cluster in the vcluster connect command for NAME in both places. By default, vcluster prepends the string vcluster- to the name of the virtual cluster’s namespace.

More resources

We've walked through how to apply Kubernetes manifests and Helm charts to virtual clusters when they're initialized. This is a great way to bootstrap new virtual clusters and configure them.

Here are some resources to learn more: