[Tutorial] Enforcing RBAC in Kubernetes

Identity and access managers have become accustomed to using role-based access control (RBAC) in physical and virtual systems. Fortunately, the concept of RBAC translates into Kubernetes, and it’s supported by the technologies that allow you to containerize workloads. RBAC remains a crucial component of container security, allowing you to define who has access to what resources in your cluster in an orderly fashion. RBAC is the simplest way to enforce the security concept of “least privilege,” both in manipulating permissions and in binding principals to a role. Least privilege means that a principal has the minimum access necessary to perform its tasks. Least privilege also states that access should be removed when it’s no longer needed.

Role-based access control allows efficient management of permissions across all principals assigned to a role. If the responsibilities of a principal change, then reassigning roles is simple. This is also true if the responsibilities of a role change. The permissions that are assigned with the role can be modified without modifying the permissions for each individual principal. The more efficient the change, the fewer mistakes will be made in updating and assigning permissions across the organization. Least privilege reduces the risk of unauthorized access and limits a security breach’s potential damage.

This article is the fifth installment of the “Platform Engineering Workflows” series and focuses on how to enforce RBAC in Kubernetes. In the first four parts of this series, you learned how to add environment variables and change configurations, how to manage services and dependencies, how to add and change resources, and how to spin up a new environment in Kubernetes.

This article explores the importance of RBAC and how it’s implemented for Kubernetes. It covers how RBAC implementation differs from traditional architectures and why. You’ll also learn how to set up RBAC across a Kubernetes cluster with example commands.

#Platform Engineering + Kubernetes Series

1. Platform Engineering on Kubernetes for Accelerating Development Workflows
2. Adding Environment Variables and Changing Configurations in Kubernetes
3. Adding Services and Dependencies in Kubernetes
4. Adding and Changing Kubernetes Resources
5. Enforcing RBAC in Kubernetes
6. Spinning up a New Kubernetes Environment

#Setting Up RBAC in a Kubernetes Cluster or Namespace

In this step-by-step tutorial, you’ll set up and validate role-based access control for cluster roles and non-cluster roles. You’ll also learn how and why to limit the permissions on the cluster and some other best practices for Kubernetes cluster RBAC.

First, you need to set up a demo Kubernetes deployment if you don’t already have a space to work in. Don’t start this tutorial in your production environment; you should practice setting up Kubernetes RBAC in a low-stakes environment. If you don’t already have a suitable environment, choose a service with a free trial, get permissions for a small environment of your own inside your enterprise’s Kubernetes deployments, or use a local cluster with something like minikube.

You want multiple worker pods so that you can see the difference between permissions across the cluster versus permissions set directly on pods. For that, create a deployment with three pods. Save the following in a file named deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # Tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

Then, apply the configuration by running:

kubectl apply -f deployment.yaml

The next step is to create a user that you can experiment with. You’ll need to install OpenSSL on your computer. The following commands create a user named test-user and define a context named test-user-context:

openssl req -new -key test-user.key -out test-user.csr -subj "/CN=test-user/O=group1"
openssl x509 -req -in test-user.csr -CA ~/.minikube/ca.crt -CAkey ~/.minikube/ca.key -CAcreateserial -out test-user.crt -days 500
kubectl config set-credentials test-user --client-certificate=test-user.crt --client-key=test-user.key
kubectl config set-context test-user-context --cluster=minikube --user=test-user

If you try to use this user to view the pods, you’ll get an error:

$ kubectl --context=test-user-context get pods
Error from server (Forbidden): pods is forbidden: User "test-user" cannot list resource "pods" in API group "" in the namespace "default"

This is expected since the user doesn’t have any permissions. In the next sections, you’ll practice the definition and implementation of RBAC. This tutorial aims to get you familiar with the application of roles, permissions, and bindings.

In this tutorial, you’ll:

• Define roles and assign permissions
• Create a role binding by assigning a role to a principal
• Verify the RBAC configuration is correct

#Defining Roles and Permissions

Creating roles allows you to assign permissions to a role rather than an individual principal. It’s important to remember that the role is not the user, group, or service account (principal). Rather, the role is a name for a collection of activities.

Use the kubectl create role command to create a role:

kubectl create role <role-name> --verb=<verb> --resource=<resource> [--resource-name=<resource-name>] [--namespace=<namespace>]

Here’s the breakdown of this command:

• <role-name>: You should replace this with a meaningful name that reflects the purpose or scope of the role.
• --verb=<verb>: This specifies the desired verbs, or actions, that the role allows. For example, you can use --verb=create,get,list,delete or specify a single action like --verb=get.
• --resource=<resource>: This specifies the resource type for which the role will be applicable. For example, you can use --resource=pods, --resource=services, or --resource=deployments. You can also use wildcards like --resource=pods,* to include all resources in a group.
• --resource-name=<resource-name> (optional): If you want to limit the role to a specific resource or set of resources, you can specify the name or pattern using this option. For example, you can use --resource-name=my-pod or --resource-name=my-deployment-*.
• --namespace=<namespace> (optional): If you want the role to be restricted to a specific namespace, you can specify the namespace using this option. If not provided, the role will be created in the default namespace.

Run the following command to create a role named example-role with read-only access to pods:

kubectl create role example-role --verb=get,list,watch --resource=pods

After executing the command, Kubernetes will create the role with the specified permissions (get, list, and watch permissions on pods). You can then use this role in a role binding to grant access to users, groups, or service accounts.

#Creating a Role Binding

You can now use the kubectl create rolebinding command to create a role binding. The syntax for this command is as follows:

kubectl create rolebinding <binding-name> --role=<role> --user=<user> --group=<group> --serviceaccount=<service-account> [--namespace=<namespace>]

The binding-name and role are required, and you have to choose whether you are binding to a user, group, or service account. namespace is optional and can be combined with any other command.

Within the above syntax, the following options provide granular control of the role binding when created:

• <binding-name>: You should replace this with a meaningful name that reflects the purpose or scope of the binding.

• --role=<role>: This specifies the name of the role you want to bind. The role must already exist and should have permissions associated with it.

• You’ll use one of these three:

  • --user=<user>: This specifies the principal you want to associate with the role binding. This can be the username of an individual account.
  • --group=<group>: This specifies the group you want to associate with the role binding. This can be the name of a group containing multiple users, and that is generally a better choice than a single account.
  • --serviceaccount=<service-account>: This specifies the service account you want to associate with the role binding. The service account must exist within the cluster.

• --namespace=<namespace> (optional): If you want the role binding to be restricted to a specific namespace, you can specify the namespace using this option. If not provided, the role binding will be created in the default namespace.

Create a role binding that connects example-role to test-user with the following command:

kubectl create rolebinding example-binding --role=example-role --user=test-user

Verify that test-user can now list pods:

$ kubectl --context=test-user-context get pods
NAME                            	READY   STATUS	RESTARTS   AGE
nginx-deployment-85996f8dbd-fxx68   1/1 	Running   0      	19m
nginx-deployment-85996f8dbd-w9xbt   1/1 	Running   0      	19m
nginx-deployment-85996f8dbd-xbxq9   1/1 	Running   0      	19m

#Using ClusterRole and ClusterRoleBinding

In the previous section, you created a role binding for a principal and a pod role in the default namespace. You can use the same command and syntax for a cluster role as well. You can replace --role with --clusterrole, but otherwise the command stays the same. It also applies to groups and service accounts.

Earlier, you gave the get, list, and watch permissions on pods to test-user. However, this was only done in the default namespace, as you can verify:

kubectl create ns ns2
kubectl apply -f ~/deployment.yaml --namespace=ns2
kubectl --context=test-user-context get pods --namespace=ns2 # Error: forbidden

You can now create a ClusterRole and ClusterRoleBinding to give test-user the same permissions across the whole cluster:

kubectl create clusterrole example-cluster-role --verb=get,list,watch --resource=pods
kubectl create clusterrolebinding example-cluster-binding --clusterrole=example-cluster-role --user=test-user

Verify that test-user can now list pods in the ns2 namespace as well:

$ kubectl --context=test-user-context get pods --namespace=ns2
NAME                            	READY   STATUS	RESTARTS   AGE
nginx-deployment-85996f8dbd-bncvd   1/1 	Running   0      	3m21s
nginx-deployment-85996f8dbd-nr8k4   1/1 	Running   0      	3m21s
nginx-deployment-85996f8dbd-rs2qt   1/1 	Running   0      	3m21s

#Applying RBAC with kubectl apply

You have the option of using the kubectl apply command to apply RBAC configuration. The command requires a YAML or JSON file containing the RBAC definitions. A single file can contain multiple roles, role bindings, cluster roles, and cluster role bindings, or the information can be included in more than one file. The file can be updated and reapplied whenever any roles or bindings need to change.

There are significant advantages to using kubectl apply here instead of entering RBAC configuration on the command line. The first is simply efficiency. Creating a configuration file and then applying it will take less time overall than manually creating all roles and role bindings. It also allows fewer mistakes and typos on the command line, and it’s easier to correct the file and rerun the command to make updates than to manually correct mistakes across numerous roles and bindings.

The difference between kubectl apply and kubectl create is slightly more nuanced when it comes to how they behave (imperative vs. declarative). You can read more about this here.

Create a file named rbac-example.yaml with the following:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-role
rules:
- apiGroups: [""]  # Empty string represents the core API group
  resources: ["pods"]
  verbs: ["create", "delete"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: my-role
subjects:
- kind: User
  name: test-user

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: my-cluster-role
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "create", "delete"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-cluster-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: my-cluster-role
subjects:
- kind: User
  name: test-user

In this example:

• The file includes four RBAC resources: Role, RoleBinding, ClusterRole, and ClusterRoleBinding.
• The Role named my-role allows the user to perform the specified verbs (create and delete) on the pods resource.
• The RoleBinding named my-role-binding associates my-role with test-user.
• The ClusterRole named my-cluster-role allows the user to perform the specified verbs on the deployments resource within the apps API group.
• The ClusterRoleBinding named my-cluster-role-binding associates my-cluster-role with test-user.

Apply the configuration to your Kubernetes cluster:

kubectl apply -f rbac-example.yaml

Verify that the role assignments work by deleting the nginx-deployment with test-user:

$ kubectl delete deployment nginx-deployment --context=test-user-context
deployment.apps "nginx-deployment" deleted

#Troubleshooting RBAC Issues

When you start using these RBAC commands, you’ll eventually run into configuration errors or errors in the environment. The following methods can be used to troubleshoot common problems.

#Check That the Kubernetes Version Supports RBAC

Kubernetes versions 1.7 and earlier do not support RBAC. If you have significant issues using the commands above to manipulate RBAC for your Kubernetes cluster, first check to see that the Kubernetes version used is 1.8 or higher.

Fortunately, kubectl commands are available in those earlier versions:

kubectl version

The version command displays both the client version (the version of the kubectl command line tool) and the server version (the version of the Kubernetes API server).

From the response, look for the line with Server Version, and check the GitVersion:

Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.15",

As long as the GitVersion is 1.8 or higher, RBAC-related configuration is available.

#Verify the RBAC Configuration Is Correct

kubectl is used to review and manage the cluster roles and role bindings. You should also get familiar with kubectl get and kubectl describe, as these are fantastic resources to help validate the permissions in your environment. The kubectl get command lists the roles or role bindings that have been defined. For any of the listed roles, kubectl can also retrieve detailed information about the associated permissions of a role. The command for this is kubectl describe, with the name of the role or role binding following the verb. The output of the kubectl describe command for a role binding includes the name, labels, annotations, role, kind (usually “Role”), subjects associated with the role binding, namespace, and additional details.

The following are some commonly used commands for get and for describe:

• List all the cluster roles defined in the cluster: kubectl get clusterroles
• List all cluster role bindings in the cluster: kubectl get clusterrolebindings
• Get detailed information about a specific cluster role: kubectl describe clusterrole [role-name]
• Get detailed information about a specific cluster role binding: kubectl describe clusterrolebinding [binding-name]

In addition to those commands, kubectl can-i is a useful command to determine what a principal’s permissions are, straight from the command line.

Check if a specific principal or group has the necessary permissions to perform a certain action on a resource using this syntax:

kubectl auth can-i [verb] [resource]

For example, to check if a user named john_smith can create pods, you would run:

kubectl auth can-i create pods --as john_smith

#Check the RBAC-Related Logs

Kubernetes has several RBAC-related logs available to the administrator. Two of the most important sources are API server logs and RBAC events. RBAC events can be retrieved using the following command:

kubectl get events

This provides records of role creation and changes, role bindings, and cluster-related RBAC modifications.

Methods to access the API server logs vary depending on the system that hosts your API server. If the system is a Linux distribution, then the logs are likely available in /var/log. However, if your cluster is hosted on a commercial service, the provider may make it easier to access the logs through their service management interface.

#Use Tools like RBAC Manager to Identify and Fix Issues

RBAC Manager is an open source tool that simplifies the review and management of roles in Kubernetes. It allows more intuitive management of role bindings and the assignment and removal of role permissions. A newer tool, which complements RBAC Manager, is RBAC Lookup. RBAC Lookup allows you to determine the role bindings for any principal using the command line.

#Best Practices for Enforcing RBAC in Kubernetes

Enforcement of your RBAC scheme is crucial for its success. However, RBAC enforcement is wasted without careful definitions of your RBAC plans and policies. The following best practices should inform your planning for a strong RBAC program.

#Use the Principle of Least Privilege

Your application and cluster architecture requirements should guide your decisions when adding permissions to roles. The best rule is to create a role with exactly the permissions needed for the principal’s tasks. The least necessary permissions for roles will change over time, and you will need to add or remove permissions from roles as the environment evolves. Least privilege also includes checking that all roles assigned to a principal are still necessary.

When applications are decommissioned, workers change positions, or IT departments reorganize, principals will be created, deleted, and bound and unbound from roles.

You should ensure you’re auditing cluster role permissions with kubectl describe clusterroles and role permissions using kubectl describe roles.

#Use ClusterRole and ClusterRoleBindings Only When Necessary

The proliferation of cluster control poses a significant security and operational risk. These roles are the equivalent of administrator privileges across the entire cluster. Principals with cluster-wide permissions have the ability to impact multiple (or all) nodes within the cluster. These roles and bindings are the most common end target of a compromise, as cluster-wide permissions are the most efficient way to gain control over the entire target environment. Environment administrators prefer to have cluster-wide permissions for their main principals so that they can easily work without switching roles, but this also risks any mistake or malicious activity being replicated across the entire Kubernetes cluster.

#Regularly Audit and Review RBAC Configurations

Roles and their assigned permissions should be kept up to date. You should regularly review the need for each role, validate the activities performed with that role, and audit whether the assigned permissions should be modified. This does not always mean that permissions are removed. Sometimes the activities performed with a role change to require greater or different access. It’s vital to review and manage roles and the principals to ensure permissions and roles are still necessary. This should be performed on a set schedule; if you do not have an access review policy in place with a timeline, reviewing every six months is a good idea.

The RBAC Lookup tool provides a simple method for looking up RBAC assignments for all of your principals, whether they’re service accounts, groups, or users. You can use the following syntax:

rbac-lookup <name> [--kind type] [--output wide]

kind allows you to specify between user, group, and serviceaccount. output allows you to specify wide, which will give the binding that provided the permissions as well.

For instance, you can find out what roles the user john is assigned to with the following command:

rbac-lookup john

If you want to find out how the john account got those permissions, you would instead use this command:

rbac-lookup john –output wide

#Consider Using ServiceAccounts Instead of Individual User Accounts

Service accounts follow the principle of least privilege, granting only the necessary permissions to applications and processes. When a service account is the principal, access control is based on the requirements of the individual application. One of the greatest advantages of using service accounts is that they do not require an interactive login for the environment. Therefore, you can create a service account without shell access and lower the chances that it will be compromised from outside of the cluster.

#Take Extra Care When Setting Up RBAC in Multitenant Environments

In a multitenant environment, there is the possibility of naming collisions—for instance, what if someone else has a cluster role named john or someone else is following an RBAC tutorial and has named their demo cluster my-cluster? A simple way to avoid this is to assign a namespace.

A namespace provides a virtual slice of the cluster, and RBAC can be applied to the namespace, avoiding collisions and narrowing permission spread:

As mentioned, a namespace is a virtualized separation within a cluster. It does not create new pods, networks, or clusters, and it doesn’t use separate resources. Instead, a namespace creates a separate name you can refer to for managing your resources. Namespaces also let you apply RBAC policies to control access and provide resource quota management.

The syntax for creating a new namespace is straightforward. You can create the namespace without adding any resources:

kubectl create namespace <namespace-name>

So, if you wanted the username john to have its own JohnPod, you could use the following:

kubectl create namespace JohnPod

You’d assign the pods Pod1 and Pod2 to it by using label:

kubectl label pod Pod1 -n JohnPod
kubectl label pod Pod2 -n JohnPod

You would add a role with permission to view in the new namespace as follows:

kubectl create role JohnPodView --verb=view --namespace=JohnPod

And then, if you wanted the john user to have permission to view the content of the namespace JohnPod, you would create a role binding and assignment:

kubectl create rolebinding JohnPodView --user=john --namespace=JohnPod

#Consider Using a Third-Party Tool to Manage Your Workload’s RBAC Configuration

Depending on the size of your environment, you might have more access setup and changes than you can practically manage. In that case, you should consider offloading some of the work onto a purpose-made third-party tool. For instance, you could use kubeaudit to help you audit Kubernetes clusters for security-related misconfigurations. It can provide detailed reports on RBAC policies, roles, and role bindings with the following command:

kubeaudit rbac

However, it’s important to note that the tool cannot determine whether your setup is correct or not. You still need to perform the critical task of analyzing the suitability of policies, roles, and bindings. Nonetheless, the tool lightens the burden by identifying and reporting on the various components of your RBAC setup.

#Conclusion

While the architecture of a Kubernetes cluster differs from traditional setups, RBAC remains applicable. RBAC enables efficient access management by defining roles and bindings, ensuring users have the minimum necessary privileges to carry out their tasks. By enforcing the principle of least privilege, RBAC reduces the risk of unauthorized access and limits potential damage in the event of a security breach. It provides a scalable and manageable solution for controlling access across the various components of a Kubernetes cluster, including the control plane and worker nodes.

You can manage RBAC with your hosting provider’s default tools. However, the Kubernetes cluster’s access manager must be able to use the kubectl command to review and enforce access configurations on the clusters.