Individual Kubernetes Clusters vs. Shared Kubernetes Clusters for Development

Excellent Isolation. A clear advantage of having individual clusters for each developer is that their work environments are isolated very well. This means that it is very unlikely that one developer interferes with another so that they can work independently and do not have to fear to break the development system for the whole team.

Full Cluster Access. Due to the good isolation of the individual work environments, it is not a problem anymore to give the developers full cluster access, so they can configure anything they want or need (you potentially might want to limit the maximum resource consumption though). This gives the engineers the flexibility to modify the configuration of the cluster themselves and they can experiment with different Kubernetes configurations without disturbing anyone.

Easy Start. Another benefit of the one-cluster-per-developer method is that it can be introduced in a team fairly fast and easy. The simplest option would be to just share the cloud account credentials and password with everyone. This, however, is of course very insecure and should only be done in very small teams with very trustable members if at all. A better solution is to give everybody their own cloud account with appropriate limitations or the clusters are simply created and then distributed by the cloud manager.

Problematic in Private Clouds. A first limitation of giving individual clusters to engineers is that it does not work well in most private clouds. While public clouds such as AWS and Azure provide an advanced user management, most private clouds lack a wide range of user settings and limitations. The alternative solution to give everyone admin access to the cloud environment is usually too dangerous and having one cloud administrator care for the provisioning of the clusters creates a problematic bottleneck.

Kubernetes Knowledge and Tools Necessary. If a developer is supposed to fully interact with Kubernetes and has to create and configure their own cluster, they need to have some Kubernetes knowledge and tools such as kubectl. Although it is possible to streamline these processes to some extent with detailed instructions for the tool installation and step-by-step guides for the cluster setup, everyone in a team has to repeat the same steps (which, by the way, eliminates one of the advantages of using the cloud instead of local Kubernetes environments).

Maintenance Effort. After a successful setup of the Kubernetes clusters, the work is not done. They rather have to be continuously maintained and updated. Since the individual cluster approach can lead to a lot of clusters depending on the size of the engineering team, the maintenance effort becomes immense for dozens or even hundreds of clusters.

Limited Oversight and Central Control. Due to the abundance of clusters, it can be very hard to supervise the whole development system in the cloud. While it should be possible to see how much resources are consumed and what clusters are running, it is challenging to see what is currently needed and what is running idle and could be deleted. The complexity of managing the whole environment so becomes very high, especially for larger teams.

Computing Cost. A general downside of using a cloud environment for development is that it can be costly, especially in public clouds. Since a system with a lot of clusters also creates a lot of redundancy for basic Kubernetes elements such as API servers, the one-cluster-per-developer approach is particularly expensive. To make things worse, some providers such as AWS and GCP charge a cluster management fee for every cluster, which is very costly if you use many small clusters as you usually would for development. (Google Cloud has only recently announced the introduction of a cluster management fee creating some outrage by its users.)

Public Clouds. Generally, due to the extensive user management in most public clouds, this approach works better with public cloud environments than private clouds where only limited user management is available.

Teams of Kubernetes Experts. In teams of Kubernetes experts, the downsides of individual setup and maintenance are less important, and the great flexibility can be a positive factor for this solution. There are even special cases in which this is the only possible approach for cloud-based development, such as if a solution or tool for Kubernetes management itself is developed.

Small Teams. In smaller teams, only a few clusters are needed for this approach and generally, it is easier to set-up and manage fewer clusters. Additionally, it is possible to guide the developers through the set-up process individually and to communicate informally and fast if the team size is relatively small.

No Budget Constrains. While there eventually will be budget constraints, the cloud cost is not always an important concern. This could be as the absolute cost is negligible or because the company has received credits for a public cloud, which many startups have. In these cases, there might be more important things to optimize than the cloud cost. (In a separate blog post, I describe how you can save cost by reducing the number of clusters.)

First experiments. Since you can get started pretty fast with this approach, it can often give you a good first feeling about how cloud environments and cloud-native development with Kubernetes works.

Any Cloud Environment. One advantage of the shared cluster approach is that it can be used in any cloud, public and private, as long as this cloud is running Kubernetes. In contrast to the one-cluster-per-developer approach, there are no downsides associated with either public or private clouds and it is even possible to use hybrid systems.

Cost-Efficiency. While there will always be some cost associated with using the cloud as development environment, using shared clusters is at least a computing resource and thus cost-efficient solution. All basic Kubernetes functionality is shared to avoid any redundancy and the individual namespaces can sometimes even be automatically shut-off. This can reduce the Kubernetes computing cost dramatically. For example, if namespaces are always scaled down or deactivated when the developer is not working (at night, during weekends or holidays), this can often save 70% of the total computing cost of the development system.

Central Control. Since there is only one or very few clusters, it is possible to monitor the whole system relatively easy. Of course, you still need a user management and must enforce appropriate limits for the developers using the cluster. However, there are already solutions such as Loft that offer all of this together with a dashboard to make sense of who uses what and what limits are enforced. This also makes it possible to shut-off individual namespaces automatically or manually from a central position.

Only 1 System. From a maintenance perspective, it is much easier to manage only one larger cluster than a large number of very small ones. The cluster administrator only has to update and configure one cluster instead of repeating this process over and over for each cluster individually, which saves a lot of time and effort.

Ease of Use. The developers can easily work with a system that allows on-demand namespace generation. From their perspective, the whole system will feel like a normal PaaS-system and Kubernetes is mostly a technology running in the background that they do not always have to directly interact with. For this, little to no Kubernetes knowledge and hardly any new tools are needed on the developer’s side.

Isolation Necessary. Using shared clusters to give developers access to Kubernetes, however, also comes with some disadvantages. The first one is that you need a solution for isolating the individual developers from each other to prevent chaos in your cluster. While namespaces are a good starting point and special solutions for this problem exist, user isolation in Kubernetes needs to be taken care of and can be tricky, especially if you require hard multi-tenancy. Fortunately, in most development settings, the tenants on the same cluster can be trusted and thus soft multi-tenancy is sufficient. Still, you need to introduce Network Policies to prevent namespaces to communicate with other namespaces they should not be able to talk to.

User Limitation and Management Necessary. Besides the technical limitation between the users, you also need a limitation of the individual users, so they cannot excessively use computing resources and network traffic. This leads to an additional requirement of a solid user management system, which is also necessary to give every developer access to the cluster in the first place. Again, such multi-tenancy platform solutions exist, but if you want to build the system from scratch on your own, this is an additional topic to consider. (In a separate article, I describe how to build such an internal Kubernetes platform.)

Setup Effort. In any case, you will face some setup effort to get a shared Kubernetes cluster running and to configure it properly. Even with off-the-shelf solutions, you need to set them up at the beginning, which is at least some more effort than the easiest one-cluster-per-developer-approach. Still, this one-time effort can be made up by the reduced maintenance work later on.

Any Cloud. Sharing a Kubernetes cluster is possible in both public and private clouds, which makes it a universal solution from a purely technical perspective.

Teams without Kubernetes Knowledge. Since the shared cluster will be centrally managed by an experienced cluster admin, the average developer does not need extensive knowledge about Kubernetes. They potentially might need to install a few tools to be able to deploy to the cluster and to create namespaces in it. However, these processes are fairly easy and standardized.

Trusted Team Members. One of the biggest technical challenges with the shared cluster approach is to establish multi-tenancy. Here, it is much easier to realize soft multi-tenancy, i.e. to isolate users from each other in a way that lets them work without disturbing each other but does not securely prevents malicious behavior. For most development teams, this isolation should be sufficient as all tenants are known and have no intention to destroy anything on purpose.