Virtual Clusters (vClusters)

In Loft, users can create virtual Kubernetes clusters which are fully functional Kubernetes clusters, but each vcluster runs inside a namespace of the underlying k8s cluster. It's cheaper than creating separate full-blown clusters and it offers better multi-tenancy and isolation than regular namespaces.

Virtual Clusters in Loft are based on the open source project vcluster. For more in-depth information, please take a look at the official vcluster docs.

Why Virtual Clusters?

Virtual clusters:

  • are much cheaper than "real" clusters (shared resources and sleep mode just like for namespaces)
  • can be created and cleaned up again in seconds (great for CI/CD or testing)
  • allow to test different Kubernetes versions inside a single host cluster which may have a different version than the virtual clusters
  • more powerful than simple namespace (virtual clusters allow users to use CRDs etc.)
  • allow users to install apps which require cluster-wide permissions while being limited to actually just one namespace within the host cluster
  • provide strict isolation while still allowing you to share certain services of the underlying host cluster (e.g. using the host's ingress-controller and cert-manager)

Workflows

Create vCluster

Loft UI - Create Virtual Cluster

Use vCluster

You can either use the Loft CLI to create a kube config automatically or you can create your own kube config with an Access Key.

Run this command to add a kube-context for the virtual cluster to your local kube-config file or to switch to an existing kube-context of a virtual cluster:

loft use vcluster # shows a list of all available vclusters
loft use vcluster [VCLUSTER_NAME] # optional flags: --cluster=[CLUSTER_NAME] --space [VCLUSTER_NAMESPACE]

Then, run any kubectl command within the virtual cluster:

kubectl get namespaces
RBAC Permissions

In order for a user to access a virtual cluster the user needs the RBAC permission get on the resource virtualclusters in the api group storage.loft.sh with api version v1 either in the namespace where the virtual cluster was created in or cluster wide

Delete vCluster

Run this command to delete a virtual cluster:

loft delete vcluster [VCLUSTER_NAME] # optional flags: --cluster=[CLUSTER_NAME] --space [VCLUSTER_NAMESPACE]


Advanced Configuration

You can configure virtual clusters using the VirtualCluster CRD or using the Loft UI. Under the hood, Virtual clusters are deployed using a Helm chart (see the vcluster project on GitHub). You are able to change how the virtual cluster is deployed via the Advanced Options section in the Loft UI.

For more specific use cases and greater in-depth documentation, please take a look at the official vcluster docs.

Networking

By default, Services and Ingresses are synced from the virtual cluster to the host cluster in order to enable correct network functionality for the virtual cluster.

That means that you can create ingresses in your virtual cluster to make services available via domains using the ingress-controller that is running in the host cluster. This helps to share resources across different virtual clusters and is easier for users of virtual clusters because otherwise, they would need to install an ingress-controller and configure DNS for each virtual cluster.

Because the syncer keeps annotations for ingresses, you may also set the cert-manager ingress annotations to use the cert-manager in your host cluster to automatically provision SSL certificates from Let's Encrypt.

If you do not want ingresses to be synced and instead use a separate ingress controller within a virtual cluster, add the following syncer configuration in the virtual cluster advanced options:

syncer:
extraArgs: ["--disable-sync-resources=ingresses"]

Overriding vcluster defaults

There are default values how a new virtual cluster will be deployed by Loft either through the UI (vClusters > Create vCluster) or through the Loft CLI (loft create vcluster). These default values can be overridden on cluster level by setting the annotation virtualcluster.loft.sh/default-values to the new values.

For example, if you want to disable ingress syncing by default, you would use this as default values:

# Below you can configure the virtual cluster
vcluster:
extraArgs:
- --service-cidr=10.96.0.0/12
image: rancher/k3s:v1.21.2-k3s1
syncer:
extraArgs: ["--disable-sync-resources=ingresses"]

You can now navigate to Clusters > Show Yaml and set the annotation accordingly.

vCluster Set Default Values
vCluster Set Default Values

The next time a new virtual cluster is created through the Loft CLI or UI in this connected cluster these new default values will be applied.

How does it work?

The basic idea of a virtual cluster is to spin up a functional but incomplete new Kubernetes cluster within an existing cluster and sync certain needed core resources between the two clusters. The virtual cluster itself only consists of the core Kubernetes components: API server, controller manager and etcd. Besides some core kubernetes resources like pods, services, persistentvolumeclaims etc. that are needed for actual execution, all other kubernetes resources (like unbound configmaps, unbound secrets, deployments, replicasets, resourcequotas, clusterroles, crds, apiservices etc.) are purely handled within the virtual cluster and NOT synced to the host cluster. This makes it possible to allow each user access to a complete own kubernetes cluster with full functionality, while still being able to separate them in namespaces in the actual host cluster.

A virtual Kubernetes cluster in Loft is tied to a single namespace. The virtual cluster and hypervisor run within a single pod that consists of two parts:

  • a k3s instance which contains the Kubernetes control plane (API server, controller manager and etcd) for the virtual cluster
  • an instance of a virtual cluster hypervisor which is mainly responsible for syncing cluster resources between the k3s powered virtual cluster and the underlying host cluster
vCluster Architecture
vCluster Architecture Overview

Host Namespace

All Kubernetes objects of a virtual cluster are encapsulated in a single namespace within the underlying host cluster, which allows system admins to restrict resources of a virtual cluster via account/resource quotas or limit ranges. Because the virtual cluster uses a cluster id suffix for each synced resource, it is also possible to run several virtual clusters within a single namespace, without them interfering with each other. Furthermore it is also possible to run virtual clusters within virtual clusters.

Control Plane

The virtual cluster technology of Loft is based on k3s to reduce virtual cluster overhead. k3s is a certified Kubernetes distribution and allows Loft to spin up a fully functional Kubernetes cluster using a single binary while disabling all unnecessary Kubernetes components like the pod scheduler which is not needed for the virtual cluster because pods are actually scheduled in the underlying host cluster.

Hypervisor

Besides k3s, there is a Kubernetes hypervisor that emulates a fully working Kubernetes setup in the virtual cluster. Besides the synchronization of virtual and host cluster resources, the hypervisor also redirects certain Kubernetes API requests to the host cluster, such as port forwarding or pod / service proxying. It essentially acts as a reverse proxy for the virtual cluster.