Table of Contents
TL;DR: In this article, you will learn how to evaluate isolation, ease of management and costs for multi-tenant setups in Kubernetes.
When planning your infrastructure, one of the fundamental questions is: how many Kubernetes clusters should you have?
One big cluster or multiple smaller clusters?
Should the team share resources, or to each their own?
Kubernetes is designed as a common platform that tenants can share.
This is usually referred to as “soft multi-tenancy”.
Soft multi-tenancy is appropriate when you trust your tenant and want to maximize resource utilization.
However, there are better solutions than this if you plan to build a platform that runs third-party code.
A tenant escaping from their sandbox could interact with others.
Kubernetes has several mechanisms to restrict what users and workloads can do in the cluster.
In this article, you will explore three different multi-tenant options with stringer isolation and their trade-offs:
- The Hierarchical Namespace Controller (HNC).
- vCluster.
- Karmada.
#Kubernetes namespaces
You may be wondering why Kubernetes namespaces aren’t on that list.
Namespaces are a building block for grouping in Kubernetes but aren’t explicitly designed for multitenancy.
To understand the difference, imagine setting up a shared Kubernetes platform for two teams that look after one project each.
The first team is reassigned, and their project is moved to the second team, which looks after two projects across two namespaces.
If the team works (hard) to align the two projects, they are still faced with duplicate resources:
- Both namespaces will have similar network policies.
- They might have the same ResourceQuotas and LimitRanges.
- RBAC is also duplicated.
In this scenario, the challenge is that policies such as Quotas and NetworkPolicies are applied to the namespace, not the team (tenant) working on both projects simultaneously.
Kubernetes doesn’t have a concept of the tenant.
However, you can get close by copying resources across the namespace.
And that happens when you install the Hierarchical Namespace Controller (HNC) in your cluster.
#Hierarchical Namespace Controller
The Hierarchical Namespace Controller is a component you install in the cluster that lets you nest namespaces.
The (clever) idea behind it is that all child namespaces inherit resources from the parent and can be infinitely nested.
So, if you create a Role in the parent namespace, the same resource is made available to its children.
Under the hood, the controller computes the difference between the two namespaces and copies the resources.
Let’s have a look at an example.
After you installed the controller, you can create a root namespace with the following command:
$ kubectl hns create <CHILD_NAMESPACE> -n <PARENT_NAMESPACE>
You can create a role in the parent namespace with:
$ kubectl -n <PARENT_NAMESPACE> create role test1 --verb=* \ --resource=pod
The role grants you unlimited access to pods in the parent namespace.
What happens when you list the roles in the child namespace?
$ kubectl get roles -n <CHILD_NAMESPACE>
NAME CREATED AT
test1 2024-02-29T20:16:05Z
The role was propagated!
The Hierarchical Namespace Controller has no mechanism for overrides at this point, but it’s something that the maintainers are considering.
Is multi-tenancy solved in Kubernetes now?
Not so fast.
#Namespaced vs global resources
In Kubernetes, resources such as Pod and Deployments can be deployed in a namespace.
However, some resources are global to the cluster, such as ClusterRoles, ClusterRoleBindings, Namespaces, PersistentVolumes, Custom Resource Definitions (CRDs), etc.
If tenants can manage Persistent Volumes, they can see all persistent volumes in the cluster, not just theirs.
But it doesn’t stop there.
If you decide to install a Custom Resource Definition (CRD) for a tenant, the type definition is not namespaced (but the resource itself could be).
You will face the same issue with Validating and Mutating admission webhooks: their resources are global even if they validate or mutate only resources in a specific namespace.
Some of those issues could be mitigated with proper RBAC controls.
However, there’s something else you should keep in mind.
Kubernetes has a single control plane for the entire cluster.
The entire cluster will suffer if any tenant abuses the API server or DNS.
So, how can you provide a control plan for tenants?
#vCluster: a control plane per tenant
You could provide a cluster per tenant or take a lightweight approach: run a control plane as a pod in your cluster.
Tenants connect directly to the control plane in the pod and create their resources there.
Since the control plane is just for them, you immediately remove any global resource and contention issues.
But where is the pod scheduled if you run just a control plane?
vCluster took this approach and devised an ingenious solution: a controller that copies resources from the tenant’s control plane to the host control plane.
When you schedule a Deployment in the nested control plane, the resulting pod specs are copied to the host control plane, where they are assigned and deployed to actual nodes.
vCluster has some interesting trade-offs:
- Each tenant has an entire control plane and the flexibility of a real Kubernetes cluster.
- This control plane is only used to store resources in a database.
- The controller can be instructed to copy only specific resources.
In other words, a careful synching mechanism lets you selectively decide how to propagate resources from the tenant cluster.
If you are a tenant, you can experiment as much as you’d like with what if feels like a real cluster.
As an admin, you can have a granular strategy to manage tenants and their resources.
Still, workloads deployed in vCluster end up in the same cluster.
What if you need to segregate workloads into different clusters for regulatory reasons?
#A cluster for each tenant
For example, two apps cannot be on the same network.
Or one app is bound to be in a particular region for data protection.
In this case, you only have the option to have a dedicated cluster per tenant.
However, how do you manage multiple clusters at scale?
You could use Karmada to manage the tenant cluster and deploy common workloads all once across all clusters.
Karmada’s architecture is similar to vcluster.
First, a cluster manager control plane is aware of multiple clusters.
You usually deploy it in a specially design cluster that doesn’t run any workloads.
Then, Karmada employs an agent that receives instructions from the Karmada control plane and forwards them to the local cluster.
In the end, you have the following arrangement:
- The Karmada control plane can schedule workloads across all clusters.
- Each cluster can still deploy workloads independently.
Of all options, this is the most expensive to maintain and operate.
Just imagine (the horror! of) upgrading dozens of clusters to the latest version.
It’s also the solution that offers more isolation, as all tenants have a dedicated cluster to work with.
#Isolation, ease of management and costs
The three options (HNC, vCluster and Karmada) offer different trade-offs for isolation and ease of management.
And they also come with different costs.
Running extra management clusters (Karamada) or control planes (vCluster) costs money compared to a simple controller that propagates resources (HNC).
But this barely scratches the surface.
What happens when you want to run a monitoring solution like Prometheus?
With vCluster and HNC, you can run a single Prometheus instance per cluster.
With Karmada, you are probably forced to have one Prometheus instance per tenant.
So the costs don’t go up linearly but exponentially.
#Summary
Which multi-tenancy solution is right for you?
Multi-tenancy in Kubernetes doesn’t have a right or wrong answer (or tool).
It’s about meeting the requirements while balancing costs, isolation and ease of management.
Hopefully, this article helped you identify some constraints you should consider when evaluating.
