Table of Contents
Kubernetes has matured so much recently that it even expanded beyond its original space as operations technology. So, also at least 1.7 million developers are already using Kubernetes as “The State of Cloud Native Development” by the CNCF stated for Q2 2019. And one can only expect this number to rise even further.
While such a fast growth is impressing, it also means that the ecosystem needs to evolve fast to solve the challenges that come with using Kubernetes for other scenarios than operations, such as development. One central challenge, that is also a basic requirement to make efficient Kubernetes development possible in the first place, is how to provide developers a Kubernetes work environment.
Since the access to the infrastructure is so critical for efficient workflows, developers even name “waiting for central IT to provide access to infrastructure” their top impediment to work productively, as VMWare’s “The State of Kubernetes 2020” study found out.
Interestingly, only 6% of executives share this opinion, which might be a reason why this topic has been underestimated in the past. However, executives should obviously take this matter very seriously if they want to increase development productivity.
In this post, I will describe the problems with the ways Kubernetes access is currently provided to developers and why the rather new concept of virtual Kubernetes clusters (vClusters) might be a better alternative. I will then also shortly describe how to use vClusters to efficiently solve the Kubernetes access challenge.
Current Approaches for Kubernetes Access
Local Kubernetes
With local Kubernetes environments such as minikube or k3s, developers can create their own Kubernetes clusters on their local computers. This often leads to developers struggling with the management and setup of these pared-down Kubernetes technologies that are also not completely realistic compared to “real-world”, cloud-based environments. The upside of this approach is that the developers have full control over their environment and can independently create it whenever they need it.
Pipelines
By using a pipeline-based approach, the developers are not in direct contact with developers but rather trigger a pipeline to get their code running in Kubernetes. Such pipelines separate the developers from Kubernetes completely, which means that they usually do not have any option to make configuration changes. The advantage of pipeline-access to Kubernetes is that the developers do not have to manage their Kubernetes environments themselves, but admins can do this. Pipelines also are a relatively robust way of providing Kubernetes access because the developers have so limited access.
Namespaces
The Kubernetes-native construct of namespaces allows for multi-tenancy on a shared cluster as every developer gets an individual namespace to work with. However, namespaces are only a soft form of multi-tenancy, which means that the overall system can be less stable because it is possible that one engineer breaks the whole system. Since namespaces are only a construct within Kubernetes, developers are also limited in their options to make configuration changes to Kubernetes. Nevertheless, namespaces are a rather efficient approach to provide Kubernetes access to engineers as the sharing of a cluster is cost-efficient and the ability to create namespaces on-demand is easy for engineers. With existing solutions such as Loft (which also offers vClusters, see below), the implementation of a namespace-platform for developers is also quite simple and easy to maintain for admins.
Individual Clusters
In principle, every engineer could just get an individual Kubernetes in the cloud which is even managed by the cloud providers. However, this approach to provide Kubernetes to developers is rarely used as it can easily become extremely expensive, especially as every developer would need some form of admin rights for the cloud provider to be able to create new clusters. Additionally, such a system is very hard to oversee, which leads to an increasing amount of unused computing resources and thus ever-increasing cost. Still, such a setup has some appeal because it would isolate developers perfectly while they still have full cluster access.
To get a deeper understanding of the pros and cons of individual clusters vs. shared clusters, take a look at my blog post about this topic.
In a separate article, I provide a more detailed comparison of different Kubernetes development environments.
Virtual Clusters for Kubernetes Access
Virtual Clusters are fully functional clusters that are running on Kubernetes themselves. As such, they are conceptually similar to virtual machines and give the users the feeling of real clusters, while the vClusters actually run on a single physical Kubernetes cluster.
For more more information, read this article introducing virtual Clusters.
Virtual Kubernets clusters so combine the advantages and strengths of the current approaches in a way that makes them ideal for development settings and a simple Kubernetes access distribution for developers.
Full Flexibility/Access. As every developer gets an individual virtual cluster, they have full admin access to it and can do whatever they want with it. This includes changing all available configurations, even the Kubernetes version, independently from all the other users that are also working on the same physical cluster.
Secure Isolation. Even though the developers have all flexibility and even admin access, they are securely isolated within their vClusters that act as dev sandboxes. Breaking out of these is very unlikely which secures the stability of the underlying physical cluster considerably, especially in comparison to namespace-based cluster sharing approaches.
Limited Admin Effort. Since every dev environment is running on the same physical cluster, there only needs to be one cluster for all engineers. For this, the effort to provide this is much reduced for the sys admins. Again, the secure isolation further reduces the admin effort as it prevents the underlying cluster from breaking due to misconfigurations by the developers. At the same time, the underlying cluster can be a "dumb" cluster without many additional installtions and add-ons.
Self-Service. Similar to namespace-based solutions, developers are also enabled to create virtual clusters themselves whenever they need them (as long as they stay within their resource limits). This eliminates the previously mentioned problem of “waiting for central IT to provide access to infrastructure” completely.
No Management Effort for Developers. Besides the self-service, also the fact that developers do not need to set up or manage their dev environments themselves improves their productivity a lot. After the admins set up the vCluster platform once and the engineers have access to it, the engineers do not have to care about the infrastructure anymore and can rather focus on their actual development tasks.
Cost Efficiency. Finally, virtual clusters save some cloud computing cost as the underlying environment is shared. Additional features such as an automatic sleep mode makes this approach even more cost efficient as idle times that are a problem with other cloud-based approaches can almost be eliminated. (To learn more about cost savings with shared Kubernetes clusters, take a look at this post.)
How to provide vClusters to developers efficiently
Virtual clusters and even a direct Kubernetes access for developers are rather new concepts. However, there are already specific solutions to solve both challenges:
Loft
Loft is a proprietary solution that facilitates the provisioning of virtual clusters and namespaces to developers and provides the virtual cluster technology in the first place. loft can either be run in your infrastructure (on-premise) or you can use a managed version of it.
In any case, you connect your own Kubernetes cluster to Loft and in this cluster, the virtual clusters will be created. Then, you add the developers that are supposed to be able to create vClusters in a self-service way and you can set limits for their computing resource consumption. Loft cares for the user management (with optional SSO) and the enforcement of the limits. Additionally, it provides cost saving features such as a sleep mode that reduces cost for unused vClusters.
To get started with Loft, follow the Loft Quickstart.
Alternatives: The concept of virtual clusters is rather new, so only few solutions for it exist. However, there are also open-source proof-of-concepts for a virtual cluster technology, such as k3v or the project from the Kubernetes multi-tenancy SIG.
DevSpace
While Loft is a solution to create the work environments for the developers, the developers themselves rarely interact directly with Loft. They can rather use solutions that are made for development specific use cases, such as the open-source software DevSpace. DevSpace facilitates the introduction of Kubernetes development by standardizing and automating workflows. For example, developers can easily deploy to Kubernetes with the command devspace deploy or they can even develop software directly in Kubernetes with devspace dev.
To make the introduction of Kubernetes into the development workflow even easier, there is a loft-devspace-plugin that integrates the Loft commands to create virtual clusters directly into DevSpace, so the developers do not even need to know that they are using Loft at all.
If you want to try out DevSpace, follow the DevSpace getting started-guide.
Alternatives: There are some alternatives for Kubernetes development tools. Besides DevSpace, Skaffold and Tilt are probably most notable.
Conclusion
Many challenges that companies face or will face when they introduce Kubernetes for developers are already addressed today. The most fundamental of these challenges is how to provide developers a Kubernetes access. Here, virtual Kubernetes clusters are the newest solution that combine the advantages of existing approaches but without their disadvantages. Developers so get a fully isolated, very flexible self-service Kubernetes access that is also cost-efficient and does not require much effort from the sys admins. This makes vClusters a powerful new approach to solve the number one challenge of providing developers with infrastructure access to Kubernetes.
Solutions such as Loft provide all of this out-of-the-box in an engineering-friendly fashion. Combining Loft’s vCluster platform with dev tools such as DevSpace makes the transition to Kubernetes even easier. Eventually, this can lead to a situation in which developers can mostly “ignore” the fact that they are using Kubernetes while they still have the freedom to directly interact with Kubernetes if they need it. Such a system will reduce organization-internal resistance against Kubernetes and so contributes to further Kubernetes adoption even during development.
