Introduction to Virtual Clusters in Kubernetes

With the increasing adoption of Kubernetes within organizations, the need for Kubernetes access for applications and engineers is also growing. Since it is neither feasible nor cost-efficient to always use whole physical Kubernetes clusters, virtualization for Kubernetes is the obvious solution. In this article, I will describe an implementation of such a Kubernetes virtualization: Virtual clusters. I will also explain how virtual Kubernetes clusters work, how they can be used, and why they are a real alternative to current approaches for Kubernetes access.
The idea of virtualization within Kubernetes is not new: In the official Kubernetes documentation, namespaces are labeled “virtual clusters” that span a single physical cluster and provide a joint scope for related Kubernetes objects. With namespaces, it is possible to create separate environments for multiple apps and users in the same Kubernetes cluster.
However, namespaces have some limitations: They cannot contain cluster scoped resources. While this may seem obvious, a lot of applications actually need to create or at least access cluster scoped resources like nodes, cluster roles, persistent volumes and storage classes. As soon as this happens, the application breaks its virtual namespace boundary and cannot be properly isolated from other applications anymore.
Problems go even further if applications need to create their own custom resource definition or extend the API server via an APIService.
To solve these issues and create securely isolated ephemeral environments for applications during testing and development, the pattern of spinning up small, throw-away Kubernetes clusters has emerged.
Spinning up small, throw-away Kubernetes clusters solves the problem of coping with cluster scoped resources and isolation, but it is very cost-inefficient and negates one of the key advantages of Kubernetes itself: Being an orchestration system. Imagine the cost of a single cluster running 1000 containers vs 1000 Kubernetes clusters running a single container:
That is a lot of overhead, which can also result in a significant rise in your infrastructure bill (not even mentioning cluster management fees, such as in AWS and Google Cloud).
Another solution is to extend namespaces and to virtualize Kubernetes itself.
The idea of virtualizing a Kubernetes cluster is similar to virtualizing a physical machine: The host system is used for actual computing, while everything else is emulated.
There are already different implementations of the virtual cluster pattern in Kubernetes:
This article will mostly talk about the implementation of virtual Kubernetes clusters (vClusters) with loft.
The basic idea of a virtual cluster is to spin up a new Kubernetes cluster within an existing cluster and sync certain core resources between those two clusters.
A host cluster runs the actual virtual clusters pods and needs to be a fully working Kubernetes cluster. The virtual cluster itself only consists of the core Kubernetes components: API server, controller manager and etcd.
To reduce virtual cluster overhead for vClusters in loft, loft builds on k3s, which is a fully working, lightweight Kubernetes distribution that compiles the Kubernetes components into a single binary and disables all unnecessary Kubernetes features, such as the pod scheduler.
Besides k3s, there is a Kubernetes hypervisor that emulates a fully working Kubernetes setup in the virtual cluster. This component syncs certain virtual cluster resources to the host cluster and back:
Besides the synchronization of virtual and host cluster resources, the hypervisor also redirects certain Kubernetes API requests to the host cluster, such as port forwarding or pod/service proxying. It essentially acts as a reverse proxy for the virtual cluster.
In the host cluster, all created resources by a virtual cluster are encapsulated in a single namespace (it is also possible to have multiple virtual clusters within a single namespace), which allows system admins to restrict resources of a virtual cluster via resource quotas. With this architecture, virtual clusters improve isolation:
Since the virtual cluster is a working Kubernetes cluster itself, it is also even possible to install virtual clusters within virtual clusters.
To test how virtual clusters work and if they could fit your use case, you can use loft, a multi-tenancy manager for Kubernetes. Virtual clusters should work in most Kubernetes clusters above version v1.14 that support persistent volume claims.
You can set up loft for free on your local Kubernetes cluster or in any cloud provider via the following helm commands (see the official docs for more information):
# Install ingress controller in the cluster
helm install nginx-ingress nginx-ingress --repo https://kubernetes-charts.storage.googleapis.com \
--namespace nginx-ingress \
--set-string controller.config.hsts=false \
--create-namespace \
--wait
# Install loft with self signed certificate
# Change loft.localhost to your desired url
# and make sure the URL points to the ingress
# controller LoadBalancer external ip
helm install loft loft --repo https://charts.devspace.sh/ \
--namespace loft \
--create-namespace \
--set admin.username=admin \
--set admin.password=admin \
--set certIssuer.create=false \
--set ingress.host=loft.localhost \
--set ingress.tls.secret=loft-cert \
--set cluster.connect.local=true \
--wait
Wait until loft is running and make sure you install the loft CLI and log in to your loft instance:
# Login in the UI with admin:admin
loft login https://loft.localhost --insecure
You can now create a new virtual cluster via the loft CLI in any namespace you like:
# if the namespace does not exist, loft will create it for you
loft create vcluster test --space mynamespace
If the command ends successfully, loft will also switch your current local kube-context to access the virtual cluster, which you can test via kubectl
:
kubectl get namespaces
NAME STATUS AGE
default Active 37s
kube-system Active 37s
kube-public Active 37s
kube-node-lease Active 37s
You can now use the vCluster in the same way as any other Kubernetes cluster.
For more information about how to manage vClusters in loft, make sure to check out the official loft documentation.
We think virtual clusters are an interesting new technology that can drastically reduce cost and effort for several use cases, such as ephemeral environments. Compared to the approach of creating many small independent clusters, virtual clusters have multiple advantages:
While virtual clusters seem promising, they also have some limitations that should be taken into consideration:
For a more detailed analysis of benefits and use cases of virtual clusters, take a look at this article.
Virtual clusters have the potential to become an important component in the Kubernetes ecosystem. Being more cost-effective and easier to manage than many small clusters while at the same time being better isolated than namespaces makes virtual clusters a superior solution for many use cases. Examples for this are scenarios in which engineers require access to Kubernetes such as testing, experimentation or cloud-native development. Virtual clusters could so help to foster Kubernetes diffusion in many organizations.