Table of Contents
vcluster is an open source tool for provisioning virtual Kubernetes clusters and managing their lifecycles. This tutorial will cover how to run virtual clusters with high availability (HA). If you’re new to vcluster, feel free to start with our Intro to vcluster Tutorial, which covers the basics of using vcluster and how virtual clusters work.
#vCluster Series
- Introduction to Virtual Clusters in Kubernetes
- Kubernetes Namespaces vs. Virtual Clusters
- vcluster Hands-on Tutorial
- High Availability with vcluster
- Virtual Clusters For Kubernetes - Benefits & Use Cases
- Development Environments with vcluster
- How Virtual Kubernetes Clusters Can Speed Up Your Local Development
- Using Virtual Clusters for Development and CI/CD Workflows
- Kubernetes: Virtual Clusters For CI/CD & Testing
- How Codefresh Uses vcluster to Provide Hosted Argo CD
- Creating Virtual Kubernetes Clusters with vCluster and Helm
- Bootstrapping Virtual Kubernetes Clusters with vcluster
By default, vcluster runs one instance of each of its components. That’s fine for many use cases, like ones that are very ephemeral (dev environments, CI/CD, etc.). But suppose your situation requires virtual clusters with more redundancy. In that case, you can use vcluster’s High Availability feature to run multiple copies of the vcluster components so that the cluster is more resistant to individual pods failing.
#Prerequisites
For this tutorial, you will need a multi-node Kubernetes cluster. This can be a local cluster, created with tools like Minikube or kind, or a remote cluster managed or hosted by a cloud provider. Or even a bare metal cluster if you are so inclined. You just need more than one node in the cluster.
You also need your kube config pointing to that cluster (you can connect and run kubectl commands against it).
#1. Install the vcluster client
If you’re on a Mac and using Homebrew, you can install the vcluster CLI with this command:
brew install vcluster
For other platforms, see the installation instructions.
#2. Understanding vcluster distros
A distro in vcluster is the Kubernetes distribution that runs inside the virtual cluster. There are four supported distros at the time of this writing:
- k3s (the default distro)
- k0s
- eks (the Kubernetes that is installed in AWS EKS clusters)
- k8s (a Kubernetes with etcd)
The vcluster HA feature is only supported by the k3s and k8s distros. You need to choose one of those to take advantage of HA.
For this tutorial, we will use the k8s distro.
#3. Create a values.yaml file
The values.yaml file is used to specify configuration options for the virtual cluster. In the case of the HA feature, we will specify the number of replicas we want to run for each vcluster component.
Create the file called values.yaml on the computer that the vcluster client is installed on with these contents:
# Enable HA mode
enableHA: true
# Scale up syncer replicas
syncer:
replicas: 3
# Scale up etcd
etcd:
replicas: 3
# Scale up controller manager
controller:
replicas: 3
# Scale up api server
api:
replicas: 3
# Scale up DNS server
coredns:
replicas: 3
That first line enables the HA feature, and the rest specify the number of copies you want to run of each component. Set the number of replicas for each to equal the number of compute nodes your cluster has. In this example I’m using a four node Minikube cluster (one control plane node and three compute nodes), so I set the number of replicas to three in values.yaml.
kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane 2m5s v1.26.3
minikube-m02 Ready <none> 105s v1.26.3
minikube-m03 Ready <none> 93s v1.26.3
minikube-m04 Ready <none> 83s v1.26.3
#4. Create the HA virtual cluster
To create a virtual cluster using the vcluster CLI, we run the vcluster create command. To enable HA, we’ll need to specify the distro and the values.yaml file to use.
vcluster create ha-tutorial --connect=false --distro k8s -f values.yaml
We’ve named the virtual cluster ha-tutorial. By default, the vcluster create command connects to the virtual cluster, but for the purposes of this tutorial, we’ve disabled that with the --connect=false flag. And we’ve specified the distro and the values.yaml file to use when creating the virtual cluster.
You should see output like this:
info Creating namespace vcluster-ha-tutorial
info failed to find IPv6 service CIDR: couldn't find host cluster Service CIDR ("Service "test-service-tm4c9" is invalid: spec.clusterIPs[0]: Invalid value: []string{"2001:DB8::1"}: IPv6 is not configured on this cluster")
info Detected local kubernetes cluster minikube. Will deploy vcluster with a NodePort & sync real nodes
info Create vcluster ha-tutorial...
info execute command: helm upgrade ha-tutorial /var/folders/gy/d3_c4t1x731_hl8qtrfkhr_h0000gn/T/vcluster-k8s-0.15.2.tgz-1797632188 --kubeconfig /var/folders/gy/d3_c4t1x731_hl8qtrfkhr_h0000gn/T/3126958598 --namespace vcluster-ha-tutorial --install --repository-config='' --values /var/folders/gy/d3_c4t1x731_hl8qtrfkhr_h0000gn/T/2770602786 --values values.yaml
done √ Successfully created virtual cluster ha-tutorial in namespace vcluster-ha-tutorial.
- Use 'vcluster connect ha-tutorial --namespace vcluster-ha-tutorial' to access the virtual cluster
Some of your output may differ depending on whether you use a local or remote cluster.
As you can see, vcluster has created a namespace called vcluster-ha-tutorial. The virtual cluster lives inside that namespace on the host cluster. Next, let’s see what pods are running in that namespace.
kubectl get pods -n vcluster-ha-tutorial
NAME READY STATUS RESTARTS AGE
ha-tutorial-7c5c5844c5-27j2v 0/1 Running 0 20s
ha-tutorial-7c5c5844c5-gb2sm 0/1 Running 0 20s
ha-tutorial-7c5c5844c5-pwn7k 0/1 Running 0 20s
ha-tutorial-api-74f8665656-jhjnj 0/1 Running 0 20s
ha-tutorial-api-74f8665656-t5wcp 0/1 Running 0 20s
ha-tutorial-api-74f8665656-z5xl8 0/1 Running 0 20s
ha-tutorial-controller-75fb977dc5-pw5sb 0/1 Running 0 20s
ha-tutorial-controller-75fb977dc5-qzxgm 0/1 Running 0 20s
ha-tutorial-controller-75fb977dc5-wzf5v 0/1 Running 0 20s
ha-tutorial-etcd-0 0/1 Running 0 20s
ha-tutorial-etcd-1 0/1 Running 0 20s
ha-tutorial-etcd-2 0/1 Running 0 20s
There are now three replicas of each component of the virtual cluster running. If one API server pod were down, the virtual cluster would continue functioning.
If you’d like more information about how the vcluster pods were scheduled, add the -o wide flag to that previous command.
kubectl get pods -n vcluster-ha-tutorial -o wide
The hostnames of the nodes will be listed in the NODES column.
#5. Connect to the virtual cluster
We can connect to the vcluster using the vcluster connect command.
vcluster connect ha-tutorial
info Starting proxy container...
done √ Switched active kube context to vcluster_ha-tutorial_vcluster-ha-tutorial_minikube
- Use `vcluster disconnect` to return to your previous kube context
- Use `kubectl get namespaces` to access the vcluster
vcluster connect automatically switches our kube context for kubectl to the virtual cluster. Now we can see the namespaces inside of the virtual cluster by running this command:
kubectl get namespaces
NAME STATUS AGE
default Active 31s
kube-node-lease Active 33s
kube-public Active 33s
kube-system Active 33s
Our virtual cluster only contains the default namespaces that are created by Kubernetes.
Now let’s disconnect from the virtual cluster.
vcluster disconnect
This will switch your kube context back to the host cluster.
#6. Cleanup
One of the great things about vcluster is that it’s very fast and easy to clean up the virtual clusters when you’re done using them.
vcluster delete ha-tutorial
That will delete the vcluster and the namespace it was in.
#7. To learn more
Here are some resources if you’d like to learn more about vcluster and HA.
- The vcluster website
- The vcluster HA doc
- The vcluster doc on distros
Also, the Loft Community Slack has a #vcluster channel where the maintainers and many users can answer questions. It’s a great place to go if you need some help with vcluster’s HA feature or if you want to meet more folks in the community. You can sign up here.
