Table of Contents
Multi-tenancy in Kubernetes can seem like an appealing solution to many problems. Maybe it's to give your developers their own space and save costs by doing it inside a single cluster. Maybe you provide a service or host a product for your customers, and it makes sense to host it in a single cluster. The decision to use multi-tenancy usually comes down to saving costs or making management easier.
With more and more businesses focusing on getting their workloads running in Kubernetes, it's no surprise that there's popularity around being able to run it in a single cluster. It seems like the obvious way to save money.
However, it may not be as easy to manage as you think, and it may not be a simple case to save on costs. In this article you'll see some of the challenges that can arise when you go down the multi-tenancy route, as well as some tools to avoid them.
The Challenges and Solutions
When you first start diving into the world of multi-tenancy, you'll quickly discover some challenges that you need to address. Some of them are unique to the situation of multi-tenancy, whereas others are just more prevalent in this scenario. Let’s talk about what issues to look for and how to handle them.
Role-based Access Control
The first challenge you'll be facing is not necessarily unique to multi-tenancy, but it's certainly more critical than it would be outside of multi-tenancy. Role-based Access Control (RBAC) is the process of not giving users access based on who they are, but rather the roles they have been assigned.
After creating Roles, you can use RoleBindings to give a specific user access to whatever that Role has specified. Kubernetes comes with a lot of built-in Roles, but it's also possible to specify your own, which is likely needed in most multi-tenancy situations. Managing RBAC inside Kubernetes is especially important—you want to make sure one tenant doesn't get access to another.
While RBAC is not particularly the toughest aspect of multi-tenancy, there are tools to help you with it. RBAC Manager was developed by Fairwinds to make security just a bit easier.
Network Isolation
Once you've made sure that no users can access restricted resources, you also need to ensure that no network traffic can pass onto another tenant. While this may not seem like an issue at first, you don't want a tenant starting to hit the internal API of another tenant. That's just one example, of course, but in general, you want to make sure that network traffic cannot bounce between tenants and that they are treated as entirely separate hosts.
Fortunately, Kubernetes comes with a built-in method for implementing network isolation, in the form of Network Policies. You can set up a policy that will prevent a namespace from accessing another.
Cost Monitoring/Controlling
As we’ve already mentioned, one of the toughest challenges of multi-tenancy is managing the cost. It doesn't matter if your multi-tenancy is purely for internal use for developers or if you're providing it as a service for customers; either way, getting an overview of cost is important. Internally you need to know if a department is costing more than you’ve budgeted for. If you're providing a service to customers, you need to be able to bill them correctly.
A very popular tool to monitor cost inside of Kubernetes is Kubecost. Kubecost gives you a complete overview of how cost is divided across your cluster.
Of course, sometimes it's not enough to only monitor the cost of your clusters—you also need a way of controlling it. In this case, you can also look into Loft's Sleep Mode, which will automatically turn off unused resources.
vcluster
While namespaces do provide a lot of separation in multi-tenancy, like being able to isolate network traffic, they certainly do have limitations. There will always be things that can only be done at a cluster level. For example, you cannot install Custom Resource Definitions (CRDs) in only a single namespace.
To get rid of these limitations, you can look into using vcluster. A vcluster is essentially an entire Kubernetes cluster, running inside of a regular namespace on the host cluster, giving you all the advantages of multi-tenancy, with close to none of the cons.
Resource Management
Whether you decide to go with a classic namespace solution or with vclusters, you will have to keep an eye on resource usage. Not only do you need to make sure that resources aren’t wasted, you also need to avoid disproportionate resource usage across tenants. It's a common best practice, both in multi-tenancy and otherwise, to implement resource quotas.
Resource quotas allow you to control total resource usage like CPU, RAM, and storage for an entire namespace. These resource quotas are built directly into Kubernetes, but you need to put them into effect. Cluster admins also have to ensure that tenants cannot create, update, patch, or delete resource quotas.
Finally, it's crucial that tenants are not able to reach other tenants' resources. This can be implemented using RBAC, as mentioned earlier.
User Management
Whether user management is a challenge with multi-tenancy or not depends entirely on who the end-user is. Many big organizations have their own authentication provider and are very reluctant to use any other. So you want to be sure you can interact with GitHub, Microsoft, and Google, for example.
Fortunately, there's an open-source tool called Dex that can help you with that. Dex uses OpenID Connect to drive authentication for other apps. This allows you to lean back and take comfort in the knowledge that your tenants can authenticate with their existing provider.
Audit Logs
When you're working with multiple tenants, you're bound to run into issues at some point, and in those cases it's important to know what caused them. Kubernetes doesn't come with any logging that you can audit, or at least it doesn't come with a native way for you to view them.
Thankfully, there are many tools that can ensure you have a complete overview of what's happening inside your cluster. Undoubtedly the most popular choice is Prometheus, and Prometheus in combination with Grafana can give you incredibly valuable insights.
Alternatives to Building It Yourself
As mentioned earlier, there is a solution that can help you with most of the challenges of multi-tenancy: vclusters.
vclusters provide many benefits over traditional multi-tenancy built via namespacing. We’ve covered a few of them already, like being able to install CRDs in a single tenant. Other advantages include the ability to give tenants access to a cluster instead of a namespace, meaning you don't have to focus as much on getting RBAC just right. You can read more about the differences between vclusters and namespaces here.
If you want to go a bit further than vclusters, you can look into Loft.sh, which gives you not only virtual clusters but also a complete self-service platform.
Conclusion
Hopefully by now you know more about what it actually means to run multi-tenancy. You know that it comes with a set of unique challenges like isolating tenants on the network and making sure that access to resources is tightly locked down. However, you’ve also got new insight into all the options available for solving those problems.
Whichever way you decide to go with multi-tenancy, it's good to take a step back and make sure you know where the challenges will most likely appear.
Photo by Ricardo Gomez Angel on Unsplash
