Kubernetes Multi-Cluster Series

Part 1: Introduction to Kubernetes Multi-cluster Deployments

Part 2: Managing Kubernetes Cluster Lifecycle

Part 3: Authentication and Access Control in Kubernetes

Part 4: Kubernetes Self-Service and User Experience

Part 5: Kubernetes Cost Optimization

This article is the third part of a series focused on Kubernetes multi-cluster technology. For an introduction to this topic, the goals and responsibilities of multi-cluster setups, and how to manage the cluster lifecycle, please see parts one and two of this series.

With the rise in containerized deployments on Kubernetes and other platforms, much effort is needed to efficiently manage hundreds or even thousands of clusters in terms of authentication and access control for various projects.

Common concerns for organizations include how to design an authentication approach and what extent of access control is required for workloads running in a multi-cluster approach. This brings us to authentication and access control in multi-cluster environments.

This article will explore the process of designing an authentication approach by identifying the extent of access control required, how to implement it, and how it plays together with other parts of the system.

Before learning about authentication and access control in a Kubernetes multi-cluster, you should first refresh your knowledge on what a Kubernetes multi-cluster is and why it’s important.

What Is a Kubernetes Multi-Cluster?

Kubernetes is used to orchestrate cluster resources that run workloads. Based on the organization’s needs, Kubernetes can be used to run workloads across multiple nodes and environments.

Multi-cluster architecture in Kubernetes, therefore, is a design strategy for spinning up several clusters to create resource isolation, continued availability, and scalability—allowing organizations to provision their workloads in several clusters, rather than just one. This cluster creation mode allows an infrastructure and its applications to be distributed and maintained across multiple clusters.

Simply put, a Kubernetes multi-cluster goes beyond a single cluster: it’s when an organization coordinates the planning, delivery, and management of several Kubernetes environments using appropriate tools and processes.

Why Do You Need a Kubernetes Multi-Cluster?

As your organization grows, so too do your business needs, and issues concerning cluster maintenance become increasingly likely. Using a Kubernetes multi-cluster setup can help address some of the concerns that may arise.

Running your workloads in a single cluster is fine for smaller deployments. However, some business needs require advanced deployment models, and for such scenarios, a multi-cluster architecture is suitable and improves the performance of your workloads.

Developers need a Kubernetes multi-cluster, therefore, to handle workloads that span various geographical regions, and also to reduce cloud blast radius, to manage compliance and regulatory issues, and to enforce security around clusters and tenants.

Now let’s understand authentication and access in a multi-cluster setting.

System access is a crucial part of software functionality. There are ways to ensure only authenticated users are given access to clusters. Common means are the use of passphrases, tokens, and even certificates. Single sign-on (SSO) is now one of the most popular authentication techniques as it simplifies the actions required from users. SSO takes care of authenticating users into multiple systems through fewer steps compared to other access and authentication techniques.

Setting up SSO for a System

Security concerns are common to both single- and multi-cluster setups. Frequent questions include: how should users access and identify themselves in a cluster and what are they allowed to do within the clusters? Whenever these questions are considered in a multi-cluster setting, the goal should be a centralized management.

SSO is an important technique that consolidates a user’s identity across multiple applications and systems. It eliminates the requirement of creating a new set of credentials for each system accessed. Instead, the activities around authentication are handled by managed service identity providers that allow organizations to implement a single mode of authentication across multiple clusters.

SSO can authenticate users in multiple clusters and, coupled with role-based access control (RBAC), traditional CI/CD workflows, or GitOps, developers can manage permissions into and within multiple clusters. SSO can be set up through various means such as integration with an external identity provider or a cloud provider. Platforms like Loft support major SSO providers such as OIDC, Okta, SAML, and LDAP for authentication.

If you’re looking for a cloud provider that caters to identity and access management, then tools like aws-iam-authenticator (AWS) and Anthos Identity Service (Google) are good places to start.

Audit Logging

Audit logging in Kubernetes provides a better understanding of the operations that are initiated by users belonging to certain clusters. Audit logging is useful for troubleshooting and reporting issues around compliance and standards.

Auditing of clusters provide visibility into your environments and let you know:

In Kubernetes, you need to audit the logs of all actions performed in your multiple clusters. Although auditing is enabled by default for any Kubernetes API server within the clusters, no auditing data is made available until Kubernetes audit logs are set up and managed.

Several tools exist that you can plug into your Kubernetes cluster to have a richer experience when auditing logs. Cloud providers also offer advanced auditing for your Kubernetes clusters by integrating Kubernetes auditing with their auditing tools.

If an incident occurs within your multiple clusters, audit logging can guide you through root cause analysis. Audit2rbac is a reliable tool for auditing, which takes Kubernetes access logs and creates RBAC resource definition from the logs.

Additionally, you can use a platform like Loft to provide security-relevant records, documenting activities in your clusters to help audit the logs generated by users and applications. Simply, Loft audits in a way similar to auditing Kubernetes clusters.

Other tools that can also audit your existing RBAC permissions and Kubernetes setups are rbac-tool and rbac-audit.

Kubernetes provides a variety of logging mechanisms that you can use to collect and analyze logs from containers running on nodes. These include built-in tools like kubectl logs and third-party solutions like Fluentd, Elasticsearch, and Grafana. Here's a comprehensive guide to Kubernetes logging and also check out Kubernetes logging best practices.

Tenant Isolation

Multi-tenant Kubernetes cluster setups require that tenants ideally be isolated from each other to minimize the damage across workloads in the cluster in case of any compromises.

Even though workloads can be configured to have access control at the application layer for defense against network threats, it’s better to isolate them on the network layer. Defining network policies in Kubernetes is a good approach to achieving this isolation.

Network Policies

Network policies basically serve to isolate all service namespaces by default. They’re written to deny communication with any outside services.

Kubernetes cluster pods communicate with each other by design. This feature means that applications on a single cluster introduce security isolation risks. To minimize the risks involved, developers can use network security policies.

Network security policies are Kubernetes objects that check the network traffic going in and out of workloads (often at the IP address or port level). These policies define a security layer in your clusters by isolating traffic into your pods. Network policies basically isolate all services coming into namespaces. They’re written to filter communication with any outside services. Setting up network policies for namespaces allows developers to control access to their pods and ports.

Additionally, network policies allow you to specify how a pod can communicate with various network entities over the network.

RBAC

Role-based access control, or RBAC, is another technique developers can use to achieve tenant isolation. RBAC is an authorization mode used to implement and enforce user roles and permissions when controlling access to clusters. Kubernetes features an extensive, built-in RBAC framework. Kubernetes RBAC differs from traditional RBAC because it defines a separation between permissions that apply to resources in a single namespace and those that are applicable to the whole cluster.

Developers work with Roles to manage permissions within individual namespaces. Other times, they may need to use ClusterRoles to manage rules for cluster-level resources, like nodes.

The Kubernetes API server acts as the entry point for your cluster. All processes and operations pass through this main component, regardless of the origin of the request. When running several tenants, this API server is required to be secured by isolating the tenants.

This can be achieved through RBAC authorization policies to manage the behavior of users and applications clusters. RBAC uses the rbac.authorization.k8s.io API group which contains the API objects below:

Using Loft for Access Control

Effective tenant isolation can also be achieved using a platform like Loft, which has several toolsets that offer managed self-service solutions for you to spin and scale clusters smoothly.
This tool includes self-service environment provisioning and secure Kubernetes multi-tenancy, enterprise-grade access control, and a Kubernetes API gateway for integrating multiple clusters.

It also supports access control for multiple Kubernetes clusters. For secure authentication to Kubernetes clusters, Loft integrates SSO native integrations for OIDC, SAML2, LDAP, OAuth2, and other enterprise access control tools such as Active Directory and Okta.

Conclusion

Teams and divisions in an organization will have varying needs; at the same time, broad access might be needed across an organization. This poses the risk of human error and security concerns when resources are accessed without proper control. Kubernetes provides for the creation of different roles, with relevant permissions attached. These defined roles can then be assigned to different users to achieve access control.

Organizations can also rely on the managed services from several providers to fill the gaps left by Kubernetes’ default access control mechanisms.

This article explained authentication and access control in multi-cluster setups. It also covered SSO, audit logging, and tenant isolation. Additionally, you learned about some of the benefits of using an advanced self-service platform such as Loft to manage authentication and access control for multiple Kubernetes clusters, including SSO, audit logging, and auto-RBAC as out-of-the-box solutions.

Photo by Danielle Rice on Unsplash