Table of Contents
Kubernetes is wonderful but complex software that can present significant ‘Day 2’ challenges when put into production. Developers who are new to Kubernetes (and most are) face a large knowledge gap about how to sustain and optimize Kubernetes clusters. In this piece, I will share a number of ways to achieve this.
Cost optimize your Kubernetes cluster
With the increased adoption of Kubernetes within organizations, the need for cluster access for applications and engineers is also growing. However, it is neither feasible nor cost-efficient to always use whole physical Kubernetes clusters to achieve this goal. Let’s look at how to reduce the cost of a Kubernetes cluster. In a scenario of 100 developers, we calculate a 78% savings by using open source virtual clusters.
By leveraging virtual clusters with open source software such as open-source vcluster or VirtualCluster, Kubernetes operators have the ability to run multiple virtual clusters within a single physical cluster, thereby increasing the tenancy of Kubernetes clusters. By utilizing computing resources via this more communal method, organizations are able to save on computing costs as opposed to the alternative method of operating entirely separate Kubernetes clusters.
Increase tenant isolation
Another great benefit of virtual clusters is their strict isolation from other users on the cluster. This has the effect of giving each user their own workspace that looks and feels exactly like a physical Kubernetes cluster and managing virtual clusters is simpler than physical ones.
In addition, virtual clusters enable a harder form of multi-tenancy in comparison to namespace-based multi-tenancy. One of the main concerns with namespace-based multi-tenancy is that it cannot contain cluster-scoped resources. Many applications need to create – or at least access – cluster-scoped resources like nodes, cluster roles, persistent volumes and storage classes.
Virtual clusters also provide necessary security benefits by increasing the isolation in multi-tenancy clusters via:
Organizations seeking a solution for multi-tenant applications that provide greater isolation for the resources shared among their clusters should consider virtual clusters as an option. On top of providing cost savings and simpler deployment, it also brings easier management as compared to physical clusters.
Provide integrated development environments
When developing applications on Kubernetes, a common challenge developers face is the manual process of building and deploying applications via tools such as Docker Compose. This creates productivity lag and hampers the overall experience that new developers face when getting started with Kubernetes.
One option to address this challenge is to leverage integrated development environments such as open-source DevSpace, Skaffold, or Tilt. With integrated development environments, users can leverage features such as:
These tools allow teams to define their development workflows as code, which allows for transparency, aids in troubleshooting problems, and can help speed up developer onboarding. Development environments need to be disposable and easy to reproduce, not handcrafted artistry.
Implement software security guardrails
According to the 2022 State of Cloud Security report, a major limiting factor with moving to the cloud is inefficient processes. Too often, cloud engineers are tasked with dealing with the fallout of hastily planned security processes. Misconfiguration incidents result in security issues that often require significant work from both security and operations teams.
By leveraging policy engines, it’s possible to implement software security guardrails on your cloud-native Kubernetes infrastructure so that these processes are not only codified in your pipeline, but in your processes as well. Some popular policy engines such as Open Policy Agent (OPA), jsPolicy, and Kyverno can be used to enforce Kubernetes rules.
A key part of using these tools is admission control. Platform engineers and security teams can define policies and then either reject or modify workloads that don’t conform automatically. Common uses include blocking the creation of privileged pods, which are a huge security risk, and ensuring that pods contain labels.
Know What’s in Your Containers
There has been a lot of attention around software supply chain security since the SolarWinds compromise was revealed in 2020. Knowing what is actually running in your containers is critical and can be even more of a challenge with open source software, where the scrutiny around community contributions can vary greatly. Understanding what’s in your dependencies can be difficult. And if your developers are building containers that other teams or customers consume, they’ll want to ensure that it’s clear who built the containers and what is in them.
In the past, many teams relied on security scanners like open-source Trivy to detect Common Vulnerabilities and Exposures (CVEs) in their containers. Scanning is an essential practice, but it has limitations. A big one is that two different security scanning tools can give you very different results.
Sigstore is another open source project that significantly reduces the effort needed to sign, verify, and monitor what you build. Sigstore’s components include Cosign, an app for signing and verifying Open Container Initiative (OCI) images, and Rekor, a tamper-resistant transparency log that can be queried for information about images before you use them. Sigstore has become very popular in the open source world, and the Kubernetes project now uses Cosign to sign its artifacts.
Wolfi is a new tool for teams that build software images. It’s a Linux “un-distro” (it doesn’t contain a Linux kernel) that is free and can be used to build minimal containers with footprints much smaller than typical Docker images. In addition, Wolfi can generate Software Bills of Materials (SBOMs) for your builds, so users have a clear knowledge of what is inside them.
Whether your developers are consuming open-source dependencies or building artifacts used by your customers, these tools can help prevent security incidents that can sidetrack teams from working on new features for your products.
The Future of Kubernetes Operations
The best practices outlined here are a good start to resolving ‘Day 2’ issues. Kubernetes operations still have a long way to go before turnkey implementations can be enabled. As the ecosystem evolves, we will see more projects begin to establish new methods of deploying Kubernetes clusters more efficiently and best practices adopted. We will see more implementations that give rise to a seamless developer experience and enable developers to do what they do best, create valuable products. The result will be accelerated adoption of Kubernetes and a richer cloud-native ecosystem.
