Table of Contents
If you are one of the many companies using Kubernetes as an infrastructure technology, you might now ask yourself how to guide your engineers to use Kubernetes in the development phase of the software. There are several degrees of how far you want to go with introducing Kubernetes into your development process. One general question you have to answer in any case of developer access to Kubernetes is if they should rather use local clusters or work with remote Kubernetes clusters in the cloud. In this post, I will compare the two general approaches and describe their main strengths and weaknesses.
Local Cluster
For many developers, the first time they are in direct contact with Kubernetes is in a local environment. That means that they are running Kubernetes on their machines instead of the usual cloud environments Kubernetes was initially made for.
Since the developer is the only one who has to access this cluster for development, local clusters can be a feasible solution for this purpose. Over time, several solutions have emerged that are particularly made for running Kubernetes in local environments. The most important ones are Kubernetes in Docker (kind), MicroK8s, minikube and k3s. For a comparison of these local Kubernetes options, you can look at this post.
Advantages of Local Kubernetes Clusters
Running Kubernetes on your local machine during development has some advantages compared to using a cloud-based environment:
Direct Kubernetes Access: At first, the developers are admins of their own clusters and can thus freely configure everything as they need it and they can play around with all available Kubernetes features.
Perfect Isolation: Due to the use of individual clusters on their own computers, the developers are perfectly isolated from each other. They could even use the local clusters without an internet connection. This has the advantage that the different engineers cannot interfere with each other even if someone messes up something. In such cases, it is also easily possible to start from scratch with a clean system.
Additionally, strict isolation enforces a clear responsibility. If something breaks, there is only one person to blame.
No Computing Cost: Finally, local clusters run on the existing machines of the developers and therefore, there will be no extra cost associated with their use (compared to sometimes costly cloud resources). In some companies, this has the additional benefit that no complicated budget approvals are necessary to work with local Kubernetes solutions.
Disadvantages of Local Kubernetes Clusters
While the use of Kubernetes on your local machine might seem to be very attractive at first sight, it also comes with some disadvantages:
Kubernetes Knowledge Necessary: The first drawback of using local clusters during development is that the developers need to have some Kubernetes knowledge. While it would be certainly great if every engineer would be a Kubernetes expert, the complexity of Kubernetes makes this quite unrealistic and undesirable given the effort to master it. However, for the management and the setup of the cluster, at least some knowledge is required.
Repeated Significant Setup Effort: The lack of knowledge can also become a problem if there are issues during the setup process of the local clusters, which can happen if specific configurations are needed or if your local computer is set up in a less compatible way (sometimes, it is even complicated to get a local cluster running on Windows). To make things worse, this setup process needs to be done for every engineer and every computer. And due to the individual local environments, it is not possible to centrally manage and maintain them.
Less Realistic Environment: As can be seen from the name “k3s” as an allusion to the Kubernetes abbreviation “k8s”, the local Kubernetes solutions usually are pared-down versions of Kubernetes. This is necessary to make them run on limited local machines. However, this also means that some features are not available.
In general, a local environment is different from cloud environments which are the real environment of the production systems. This, for example, makes tests in local clusters less realistic compared to remote environments and can lead to undesired “it-works-on-my-machine”-problems that you actually wanted to avoid by giving the developers Kubernetes access in the first place.
Use Cases for Local Kubernetes Clusters
Overall, there are valid reasons to use local Kubernetes clusters during development and some use cases that are particularly compatible with it:
First Kubernetes Experiments: The ability to run local clusters on already available computers without further budget approvals and no access to a cloud environment makes them the typical environment for first experiments with Kubernetes.
Small Teams and Standardized Hardware: They can also be a good solution for smaller teams with standardized hardware, so admins with more Kubernetes experience can either set up the local clusters on every machine themselves or write step-by-step guidelines for their setup with the specific hardware.
Teams of Kubernetes Experts: Using a local environment to run Kubernetes is also a feasible option if every team member is a Kubernetes expert already, so they know how to care for the setup and how to solve potential configuration issues.
Remote Cluster
The alternative for giving developers access to Kubernetes is a remote cluster in a cloud environment. In another article, I described the underlying methodology of cloud development and why it might become a new standard in the future.
Providing developers with a direct access to remote clusters can either be achieved by giving every engineer an own cluster that they can manage themselves in the cloud or by sharing a cluster with several developers with tools such as Loftor kiosk. I wrote a separate post about the difference between these two methods of working with a remote cluster, but both have some common strengths and weaknesses that I want to focus on in this post.
Advantages of Remote Kubernetes Clusters
The opportunity to use a cloud environment for development and testing purposes has some benefits that are contrary to the local environment:
Easy Replicability: A first benefit of working with remote clusters is that the setup process is easy, especially in public clouds, and can be very much standardized as the underlying infrastructure is always the same. This reduces the general setup effort and facilitates the onboarding of new team members.
It also allows experienced Kubernetes experts to help less experienced colleagues as problems can either be replicated or the experts can also get a direct access to the cluster to solve the issue.
Centralized Management: No matter if shared clusters or individual clusters in the cloud are used, the administrator of the shared cluster or the cloud environment has some oversight about what everyone is doing. They can manage the access of the developers and usually also set limits to control the cost. Sometimes, especially in shared clusters, it is possible to set smart defaults that are automatically applied to newly created environments.
Realistic Environment: Another advantage of a remote environment is that it can be configured in a way that is exactly like the production environment. And even if there are minor differences, e.g. in terms of scale, the environments of the development and the live system are essentially the same in terms of all important aspects. This avoids unexpected problems after deployment and eliminates “it-works-on-my-machine”-problems.
Additionally, it can increase the feeling of responsibility among developers to create software that can be deployed to the live system without issues and so supports a DevOps culture.
Disadvantages of Remote Kubernetes Clusters
Of course, using a remote Kubernetes cluster for development is not the holy grail that will solve all your issues. It also has some downsides to go for this approach:
Computing Cost: The first disadvantage of using a cloud environment is the computing cost that is associated with it. While this is mostly a one-time cost in private clouds, using a public cloud can become quite costly in the long run. While it is possible to reduce the cost by limiting the usage of the resources by the developers or by features such as the sleep-mode of Loft, the cost for using cloud-based Kubernetes for development will often be higher than local solutions. I wrote a separate post about how to save cost in this scenario.
Management Effort: While the setup of an environment for the developer is quite easy in the cloud, the management of the cloud environment needs to be taken care of. This can include the management of the access to the cloud environment itself, i.e. who can create clusters in a public cloud, or the management of the Kubernetes cluster that is shared by the developers. For example, if a new engineer wants to start working in the cloud environment, the admin of the cloud or the cluster manager needs to provide the engineer with an own access and some explanation about the processes.
This will lead to some distraction for the designated manager of the development infrastructure. (I believe this role can be described as Developer Experience Owner and could become more important in the future.) Fortunately, it is in the interest of public cloud providers as well as the providers of internal Kubernetes platforms such as Loft to make the onboarding process as easy as possible, which has led to some improvements in this area already.
Isolation and Limits: Especially in a shared environment, the developers need to be isolated so they cannot interfere with each other. It is also important to enforce limits for the computing resource consumption so that they are fairly shared and cost are not exploding. There are some tools that support these multi-tenancy tasks for Kubernetes such as kiosk, but it is still something that needs to be taken into account when using remote Kubernetes clusters for development.
Use Cases for Remote Kubernetes Clusters
Different from local clusters, remote clusters can be used in any scenario as there are no hard limitations for settings that they cannot be used in. However, there are some situations that they seem especially fitting:
Heterogeneous Teams: If the requirements, backgrounds, and needs of the engineers in a team are very different and the hardware of every team member is individual, a remote environment helps to cater for these needs while it still offers the standardization of the processes. An example of such a situation could be a setting in which every engineer can choose their own hardware and they work on different tasks, such as frontend, backend, and testing but all want to work with Kubernetes.
Teams With Little Kubernetes Knowledge: The standardization of the processes in a cloud Kubernetes setting can be very helpful for developers without extensive knowledge about Kubernetes. With additional tools such as DevSpace or Skaffold, the Kubernetes development workflow can be streamlined to a large extent, so it is possible to work with Kubernetes seamlessly. This also prevents distractions of developers from their actual focus at work.
Special Hardware Requirements: A remote cluster solution is also the best approach if specific hardware is required for running the developed software. Nowadays, this often means GPUs that are quite expensive to buy but are available on-demand in the cloud. The same goes for some specific services of cloud providers that can be only used if the software is also running in the cloud.
I wrote a separate article where I compare the different Kubernetes development environments in some more detail.
Conclusion
Both approaches to give developers direct access to Kubernetes come with pros and cons and are the preferred solution in some situations. While the engineers are not distracted by additional tasks in remote environments, additional cost for infrastructure, its management, and additional tools are often inevitable. Local clusters will not cost you money at the beginning, but their use can still be expensive if the time engineers spend on additional tasks is accounted for.
Overall, this means that using Kubernetes during development comes at a price even if you use free, open-source solutions. However, I believe that it will pay off over time given the clear drawbacks of no Kubernetes access: unrealistic environments, separation of dev and ops (“works-on-my-machine”-problems), and missing replicability of issues. Finally, more complex applications may even force companies to give developers access to Kubernetes.
This is why I believe that it will become more common in the future for developers to directly interact with Kubernetes, one way or the other. Potentially, even a combination of both approaches becomes the standard, e.g. if local environments are used only for quick tests while major changes as well as the staging environment are running in a remote environment.
Photo by Clément H on Unsplash
Photo by panumas nikhomkhai from Pexels
