High Availability with vcluster

vcluster is an open source tool for provisioning virtual Kubernetes clusters and managing their lifecycles. This tutorial will cover how to run virtual clusters with high availability (HA). If you're new to vcluster, feel free to start with our Intro to vcluster Tutorial, which covers the basics of using vcluster and how virtual clusters work.

vCluster Series

By default, vcluster runs one instance of each of its components. That's fine for many use cases, like ones that are very ephemeral (dev environments, CI/CD, etc.). But suppose your situation requires virtual clusters with more redundancy. In that case, you can use vcluster's High Availability feature to run multiple copies of the vcluster components so that the cluster is more resistant to individual pods failing.

Prerequisites

For this tutorial, you will need a multi-node Kubernetes cluster. This can be a local cluster, created with tools like Minikube or kind, or a remote cluster managed or hosted by a cloud provider. Or even a bare metal cluster if you are so inclined. You just need more than one node in the cluster.

You also need your kube config pointing to that cluster (you can connect and run kubectl commands against it).

1. Install the vcluster client

If you’re on a Mac and using Homebrew, you can install the vcluster CLI with this command:

brew install vcluster

For other platforms, see the installation instructions.

2. Understanding vcluster distros

A distro in vcluster is the Kubernetes distribution that runs inside the virtual cluster. There are four supported distros at the time of this writing:

The vcluster HA feature is only supported by the k3s and k8s distros. You need to choose one of those to take advantage of HA.

For this tutorial, we will use the k8s distro.

3. Create a values.yaml file

The values.yaml file is used to specify configuration options for the virtual cluster. In the case of the HA feature, we will specify the number of replicas we want to run for each vcluster component.

Create the file called values.yaml on the computer that the vcluster client is installed on with these contents:

# Enable HA mode
enableHA: true

# Scale up syncer replicas
syncer:
  replicas: 3

# Scale up etcd
etcd:
  replicas: 3

# Scale up controller manager
controller:
  replicas: 3

# Scale up api server
api:
  replicas: 3

# Scale up DNS server
coredns:
  replicas: 3

That first line enables the HA feature, and the rest specify the number of copies you want to run of each component. Set the number of replicas for each to equal the number of compute nodes your cluster has. In this example I'm using a four node Minikube cluster (one control plane node and three compute nodes), so I set the number of replicas to three in values.yaml.

kubectl get nodes

NAME           STATUS   ROLES           AGE    VERSION
minikube       Ready    control-plane   2m5s   v1.26.3
minikube-m02   Ready    <none>          105s   v1.26.3
minikube-m03   Ready    <none>          93s    v1.26.3
minikube-m04   Ready    <none>          83s    v1.26.3

4. Create the HA virtual cluster

To create a virtual cluster using the vcluster CLI, we run the vcluster create command. To enable HA, we'll need to specify the distro and the values.yaml file to use.

vcluster create ha-tutorial --connect=false --distro k8s -f values.yaml

We've named the virtual cluster ha-tutorial. By default, the vcluster create command connects to the virtual cluster, but for the purposes of this tutorial, we've disabled that with the --connect=false flag. And we've specified the distro and the values.yaml file to use when creating the virtual cluster.

You should see output like this:

info   Creating namespace vcluster-ha-tutorial
info   failed to find IPv6 service CIDR: couldn't find host cluster Service CIDR ("Service "test-service-tm4c9" is invalid: spec.clusterIPs[0]: Invalid value: []string{"2001:DB8::1"}: IPv6 is not configured on this cluster")
info   Detected local kubernetes cluster minikube. Will deploy vcluster with a NodePort & sync real nodes
info   Create vcluster ha-tutorial...
info   execute command: helm upgrade ha-tutorial /var/folders/gy/d3_c4t1x731_hl8qtrfkhr_h0000gn/T/vcluster-k8s-0.15.2.tgz-1797632188 --kubeconfig /var/folders/gy/d3_c4t1x731_hl8qtrfkhr_h0000gn/T/3126958598 --namespace vcluster-ha-tutorial --install --repository-config='' --values /var/folders/gy/d3_c4t1x731_hl8qtrfkhr_h0000gn/T/2770602786 --values values.yaml
done √ Successfully created virtual cluster ha-tutorial in namespace vcluster-ha-tutorial.
- Use 'vcluster connect ha-tutorial --namespace vcluster-ha-tutorial' to access the virtual cluster

Some of your output may differ depending on whether you use a local or remote cluster.

As you can see, vcluster has created a namespace called vcluster-ha-tutorial. The virtual cluster lives inside that namespace on the host cluster. Next, let's see what pods are running in that namespace.

kubectl get pods -n vcluster-ha-tutorial

NAME                                      READY   STATUS    RESTARTS   AGE
ha-tutorial-7c5c5844c5-27j2v              0/1     Running   0          20s
ha-tutorial-7c5c5844c5-gb2sm              0/1     Running   0          20s
ha-tutorial-7c5c5844c5-pwn7k              0/1     Running   0          20s
ha-tutorial-api-74f8665656-jhjnj          0/1     Running   0          20s
ha-tutorial-api-74f8665656-t5wcp          0/1     Running   0          20s
ha-tutorial-api-74f8665656-z5xl8          0/1     Running   0          20s
ha-tutorial-controller-75fb977dc5-pw5sb   0/1     Running   0          20s
ha-tutorial-controller-75fb977dc5-qzxgm   0/1     Running   0          20s
ha-tutorial-controller-75fb977dc5-wzf5v   0/1     Running   0          20s
ha-tutorial-etcd-0                        0/1     Running   0          20s
ha-tutorial-etcd-1                        0/1     Running   0          20s
ha-tutorial-etcd-2                        0/1     Running   0          20s

There are now three replicas of each component of the virtual cluster running. If one API server pod were down, the virtual cluster would continue functioning.

If you'd like more information about how the vcluster pods were scheduled, add the -o wide flag to that previous command.

kubectl get pods -n vcluster-ha-tutorial -o wide

The hostnames of the nodes will be listed in the NODES column.

5. Connect to the virtual cluster

We can connect to the vcluster using the vcluster connect command.

vcluster connect ha-tutorial

info   Starting proxy container...
done √ Switched active kube context to vcluster_ha-tutorial_vcluster-ha-tutorial_minikube
- Use `vcluster disconnect` to return to your previous kube context
- Use `kubectl get namespaces` to access the vcluster

vcluster connect automatically switches our kube context for kubectl to the virtual cluster. Now we can see the namespaces inside of the virtual cluster by running this command:

kubectl get namespaces

NAME              STATUS   AGE
default           Active   31s
kube-node-lease   Active   33s
kube-public       Active   33s
kube-system       Active   33s

Our virtual cluster only contains the default namespaces that are created by Kubernetes.

Now let's disconnect from the virtual cluster.

vcluster disconnect

This will switch your kube context back to the host cluster.

6. Cleanup

One of the great things about vcluster is that it's very fast and easy to clean up the virtual clusters when you're done using them.

vcluster delete ha-tutorial

That will delete the vcluster and the namespace it was in.

7. To learn more

Here are some resources if you'd like to learn more about vcluster and HA.

Also, the Loft Community Slack has a #vcluster channel where the maintainers and many users can answer questions. It's a great place to go if you need some help with vcluster's HA feature or if you want to meet more folks in the community. You can sign up here.