Table of Contents
Multi-tenant Kubernetes environments allow a single cluster to be shared by several teams, applications, customers, or environments. Workloads are isolated from each other; they reside in individual slices of the cluster.
Choosing a multi-tenant architecture can improve resource utilization efficiency, cut costs, and reduce administrative overheads. However, poorly implemented multi-tenancy exposes you to new security and performance issues. Inadequate isolation could let tenants access each other's data or consume excessive resources, leaving other workloads running slowly.
This article explains how to properly implement multi-tenant isolation in your clusters. By following the best practices shown here, you'll be able to confidently share clusters between multiple workloads while avoiding the pitfalls of this approach.
Understanding Multi-Tenancy in Kubernetes
Multi-tenancy is the practice of sharing one Kubernetes cluster between several logically distinct users ("tenants"). A tenant can be anything that you partition your cluster by. One of the most common cases is dividing a cluster between teams so that each group has a private environment to deploy to.
Many organizations seize upon multi-tenancy as the optimal way to run Kubernetes at scale. It's convenient, low-cost, and efficient because you only need to provision one physical cluster. Tenants can share resources and utilize any spare capacity that's available. When a new team or customer arrives, you can simply add them as a tenant instead of creating and maintaining a whole new cluster.
Problems with Multi-Tenant Kubernetes
The benefits of multi-tenant Kubernetes are substantial. However, the model also has some drawbacks that you'll need to address before adopting it. The security implications of running multiple workloads in one environment are a significant concern. Without correctly configured isolation, a security vulnerability in one app could be used to compromise other apps in neighboring tenants.
Similarly, it's essential that tenants are subject to resource quotas that prevent them from destabilizing the cluster. Otherwise, increased resource usage in one tenant could cause performance slowdowns or even evictions in another tenant.
Multi-tenancy also affects fault tolerance. Although Kubernetes is naturally fault-tolerant, there are situations where extra physical clusters can be helpful to increase redundancy. When you use a single multi-tenant cluster, any cluster-level failure will affect all your teams, customers, and apps.
True Multi-Tenancy in Kubernetes
True multi-tenant Kubernetes isn't always straightforward to achieve. Naive implementations, such as simply creating a namespace for each tenant, will not fully isolate your workloads. To qualify as multi-tenancy, tenants shouldn't be able to affect or interact with each other—whether directly (by requesting data) or indirectly (for example, by causing performance issues).
DIY multi-tenancy is possible—as you'll see in this guide—but it imposes additional overheads on your DevOps teams. As an alternative to starting from scratch, you can use vcluster or Loft to achieve multi-tenancy with virtual clusters. These tools take the strain out of multi-tenancy by automating the setup of strongly isolated in-cluster virtual environments. Virtual clusters are equivalent to real ones, even including their own API servers, but run within a namespace in your physical cluster.
Achieving Isolated Multi-Tenancy in Kubernetes
To achieve robust multi-tenant isolation, you need to set up mechanisms that provide the following capabilities:
- Grouping of each tenant's objects: The objects that belong to a tenant should be grouped together so they can be easily identified.
- Access control: Users should only be able to interact with their tenant's resources.
- Resource allocation: Tenants mustn't be able to consume excessive cluster resources such as CPU or memory in order to ensure continuous quality of service is maintained for other tenants.
- Network isolation: Tenanted workloads shouldn't be able to discover or communicate with workloads in neighboring tenants.
- Security standards enforcement: Enforcing minimum security standards for all workloads ensures that tenants can't make unsafe deployments that could affect the cluster environment and other tenants.
Let's explore the Kubernetes features that you can use to attain these requirements.
1. Namespace Isolation
Namespaces are the foundational component of multi-tenant Kubernetes workflows. They group related resources together and provide name scoping. This allows multiple objects with the same name (such as api-server) to exist in your cluster, as long as they're in different namespaces.
The following YAML manifest defines a new namespace called tenant-a:
apiVersion: v1
kind: Namespace
metadata:
name: tenant-a
kubectl commands allow you to interact with a specific namespace by setting the -n or --namespace flag:
# Gets pods belonging to tenant-a
$ kubectl get pods -n tenant-a
# Gets pods belonging to tenant-b
$ kubectl get pods -n tenant-b
A namespace is the basic unit representing tenants in your cluster, but namespaces provide very limited separation by default. So far, there's nothing to stop Tenant A from accessing the resources in Tenant B's namespace and vice versa. Once you've created a namespace, you can layer in additional features to achieve isolation.
2. Role-Based Access Control (RBAC)
Role-based access control (RBAC) is a mechanism for controlling who can access which resources in your cluster. RBAC is built into Kubernetes, and it allows you to define roles that are then bound to cluster users. The binding grants the user the capabilities provided by the role.
In the context of multi-tenancy, RBAC prevents users from accessing resources that belong to other tenants. As this is a critical security requirement, RBAC must always be enabled in your cluster when you're using multi-tenant Kubernetes.
The following role permits users to interact with all resources in the tenant-a namespace but not tenant-b or any other namespace:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: tenant-a
name: pod-access
rules:
- apiGroups: [""]
resources: ["*"]
verbs: ["*"]
After creating a role, use a RoleBinding to grant the role to users and service accounts:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
namespace: tenant-a
name: pod-access-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: pod-access
subjects:
- kind: User
name: user-1
In practice, roles should be kept as granular as possible. Assign users the minimum set of roles they need to carry out their tasks in the cluster. This eliminates the risks associated with overprivileged accounts when credentials are lost or stolen.
3. Resource Quotas and LimitRanges
Kubernetes resource quotas let you control the CPU and memory resources that a namespace is allowed to consume.
Setting a resource quota for each of your multi-tenant namespaces ensures cluster capacity is shared in a predictable way. Individual tenants will be prevented from consuming more resources than they've been allocated. This maintains consistent performance for all tenants, even when some workloads encounter demand spikes.
Resource quotas sit atop the request and limit mechanism that controls pod resource allocations. The following example quota limits the tenant-a namespace so that its pods can have a total request of 2 CPU cores and 2 Gi of memory, while the total limit is set at 4 CPU cores and 4 Gi of memory:
apiVersion: v1
kind: ResourceQuota
metadata:
namespace: tenant-a
name: tenant-a-quota
spec:
hard:
requests.cpu: 2
requests.memory: 2Gi
limits.cpu: 4
limits.memory: 4Gi
Resource quotas can also cap the total number of pods that can exist in a namespace. This is useful to ensure individual tenants don't exceed your cluster's per-node pod cap when one applies:
apiVersion: v1
kind: ResourceQuota
metadata:
namespace: tenant-a
name: tenant-a-quota
spec:
hard:
pods: 10
What about LimitRanges?
Resource quotas aren't your only tool for managing resource allocations. LimitRanges are an accompanying mechanism that can prevent the monopolization of resources by a single object.
Resource quotas cap a namespace's total consumption, but there's nothing to stop a single pod in the namespace from consuming the entire quota. LimitRanges prevent this from happening. They can help ensure fair allocation for a tenant's workloads, which will reduce the probability that the resource quota is exceeded. This provides a better experience for the tenant's users.
LimitRanges also allow you to apply default resource requests and limits to every pod in a namespace. This ensures all pods run with resource constraints, even if the user who created them forgets to set any limits.
The following sample LimitRange specifies that pods in tenant-a have a default CPU limit of 1, a default memory limit of 500Mi, and a default CPU request of 500m. It also ensures all pods will have a minimum CPU limit of 250m and a maximum CPU limit of 2.
apiVersion: v1
kind: LimitRange
metadata:
namespace: tenant-a
name: tenant-a-limit-range
spec:
limits:
- type: Container
default:
cpu: 1
memory: 500Mi
defaultRequest:
cpu: 500m
max:
cpu: 2
min:
cpu: 250m
4. Network Policies
Network policies define permissible traffic flows in your Kubernetes cluster. They can be used to prevent resources in different tenants from communicating with each other.
Ordinarily, all Kubernetes pods and services are able to freely communicate. This is too lenient for multi-tenancy because a pod that belongs to tenant-a could reach out to a service from tenant-b even though they're in separate namespaces.
By creating network policies, you can precisely define which pods can communicate. Policies target a set of pods and apply granular rules for both ingress (incoming) and egress (outgoing) traffic.
The following network policy has an empty podSelector field, meaning it applies to all pods in the tenant-a namespace. The policy limits both ingress and egress traffic to other pods in the same namespace:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
namespace: tenant-a
name: tenant-a-network-policy
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress:
- from:
- namespaceSelector:
matchExpressions:
- key: namespace
operator: In
values: ["tenant-a"]
egress:
- to:
- namespaceSelector:
matchExpressions:
- key: namespace
operator: In
values: ["tenant-a"]
You'll need to use network policies to secure any DIY multi-tenancy implementation. In practice, you should try to make your policies more precise than the example above. The example fulfills the requirement of preventing cross-tenant access but is still unrestricted within the tenant namespace. Ideally, you should prevent all communication by default, then open up the specific routes your app actually requires:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
namespace: tenant-a
name: deny-all-network-policy
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress---apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
namespace: tenant-a
name: database-network-policy
spec:
podSelector:
matchLabels:
component: db
policyTypes:
- Ingress
- Egress
ingress:
- from:
- namespaceSelector:
matchExpressions:
- key: namespace
operator: In
values: ["tenant-a"]
podSelector:
matchLabels:
component: api
egress:
- to:
- namespaceSelector:
matchExpressions:
- key: namespace
operator: In
values: ["tenant-a"]
podSelector:
matchLabels:
component: api
This network policy blocks all traffic except anything flowing between the tenant's component: api and component: db-labelled pods.
5. Pod Security Admission Rules
Multi-tenancy demands a stable security baseline. When you've got multiple teams sharing a cluster, there's an increased risk that one of them will introduce a misconfiguration or vulnerability that negatively affects other tenants.
You can use Pod Security Admission (PSA) standards to enforce that pods deployed into tenanted namespaces meet predefined security requirements. PSAs can also be deployed at the cluster level to hold all your tenants to a minimum standard.
PSAs are the replacement for the deprecated and removed Pod Security Policies supported in older Kubernetes releases. Kubernetes comes with three default PSA standards you can enforce with the built-in PSA controller:
PSAs are enabled for a namespace by setting labels in its manifest. The following namespace opts in to all pods being compliant with the Restricted policy:
apiVersion: v1
kind: Namespace
metadata:
name: tenant-b
labels:
pod-security.kubernetes.io/enforce: restricted
More Methods for Multi-Tenant Isolation
You'd be forgiven for thinking these steps look like a lot of hard work to set up a basic multi-tenancy implementation. To achieve strong isolation, you must manually configure and maintain RBAC rules, resource quotas, network policies, and PSAs. This can be burdensome for cluster administrators. The high number of components also creates a risk that some steps will be forgotten and tenants will end up being misconfigured.
Fortunately, the Kubernetes multi-tenancy challenge has already been solved by third-party tools and extensions. These offer enhanced isolation and a heavily simplified management experience. You can focus on managing cluster operations instead of micromanaging tenants.
The virtual cluster pattern provided by vcluster is a powerful model for provisioning fully isolated tenants in one physical cluster. Users within the virtual cluster see a fully-fledged Kubernetes environment with admin access, including the ability to create namespaces, install CRDs, and configure their own RBAC rules without affecting the host cluster.
Alternatively, you can use Loft's multi-tenancy platform to give platform teams self-service access to new tenanted environments. Loft can provision virtual clusters within GitOps, CI/CD, and IDE-based workflows. It includes support for SSO, audit logs, and shared secrets so you can secure access to your tenants and monitor activity.
These tools provide a simple and reliable multi-tenant experience. Virtual clusters are more powerful than manually configured namespaces and RBAC policies because tenants also gain access to a virtualized Kubernetes control plane. This is true multi-tenancy that gives tenanted users all the benefits of Kubernetes without having to create a new physical cluster every time.
Multi-Tenant Kubernetes: True Isolation Is Essential
Multi-tenancy is a popular way to improve Kubernetes efficiency by sharing a single cluster between your teams and applications. While this technique facilitates easy setup, deployment, and administration, it can also cause security, privacy, and capacity issues if your tenants are improperly isolated.
The safe use of multi-tenant Kubernetes environments depends on your ability to achieve true isolation. Using the techniques discussed above—including namespaces, network policies, resource quotas, and RBAC—will ensure your tenants stay separated from each other, preventing cross-tenant data access and outsized resource consumption.
If managing these DIY features adds too much operational overhead, you can use vcluster or Loft to create virtual clusters. This approach allows your tenants to have their own API servers and Kubernetes namespaces, all within a single physical cluster.
