[Tutorial] Enforcing RBAC in Kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # Tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

kubectl apply -f deployment.yaml

openssl req -new -key test-user.key -out test-user.csr -subj "/CN=test-user/O=group1"
openssl x509 -req -in test-user.csr -CA ~/.minikube/ca.crt -CAkey ~/.minikube/ca.key -CAcreateserial -out test-user.crt -days 500
kubectl config set-credentials test-user --client-certificate=test-user.crt --client-key=test-user.key
kubectl config set-context test-user-context --cluster=minikube --user=test-user

$ kubectl --context=test-user-context get pods
Error from server (Forbidden): pods is forbidden: User "test-user" cannot list resource "pods" in API group "" in the namespace "default"

kubectl create role <role-name> --verb=<verb> --resource=<resource> [--resource-name=<resource-name>] [--namespace=<namespace>]

kubectl create role example-role --verb=get,list,watch --resource=pods

kubectl create rolebinding <binding-name> --role=<role> --user=<user> --group=<group> --serviceaccount=<service-account> [--namespace=<namespace>]

Create a role binding that connects example-role to test-user with the following command:

kubectl create rolebinding example-binding --role=example-role --user=test-user

Verify that test-user can now list pods:

$ kubectl --context=test-user-context get pods
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-85996f8dbd-fxx68   1/1     Running   0          19m
nginx-deployment-85996f8dbd-w9xbt   1/1     Running   0          19m
nginx-deployment-85996f8dbd-xbxq9   1/1     Running   0          19m

Using ClusterRole and ClusterRoleBinding

In the previous section, you created a role binding for a principal and a pod role in the default namespace. You can use the same command and syntax for a cluster role as well. You can replace --role with --clusterrole, but otherwise the command stays the same. It also applies to groups and service accounts.

Earlier, you gave the get, list, and watch permissions on pods to test-user. However, this was only done in the default namespace, as you can verify:

kubectl create ns ns2
kubectl apply -f ~/deployment.yaml --namespace=ns2
kubectl --context=test-user-context get pods --namespace=ns2 # Error: forbidden

You can now create a ClusterRole and ClusterRoleBinding to give test-user the same permissions across the whole cluster:

kubectl create clusterrole example-cluster-role --verb=get,list,watch --resource=pods
kubectl create clusterrolebinding example-cluster-binding --clusterrole=example-cluster-role --user=test-user

Verify that test-user can now list pods in the ns2 namespace as well:

$ kubectl --context=test-user-context get pods --namespace=ns2
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-85996f8dbd-bncvd   1/1     Running   0          3m21s
nginx-deployment-85996f8dbd-nr8k4   1/1     Running   0          3m21s
nginx-deployment-85996f8dbd-rs2qt   1/1     Running   0          3m21s

Applying RBAC with kubectl apply

You have the option of using the kubectl apply command to apply RBAC configuration. The command requires a YAML or JSON file containing the RBAC definitions. A single file can contain multiple roles, role bindings, cluster roles, and cluster role bindings, or the information can be included in more than one file. The file can be updated and reapplied whenever any roles or bindings need to change.

There are significant advantages to using kubectl apply here instead of entering RBAC configuration on the command line. The first is simply efficiency. Creating a configuration file and then applying it will take less time overall than manually creating all roles and role bindings. It also allows fewer mistakes and typos on the command line, and it's easier to correct the file and rerun the command to make updates than to manually correct mistakes across numerous roles and bindings.

The difference between kubectl apply and kubectl create is slightly more nuanced when it comes to how they behave (imperative vs. declarative). You can read more about this here.

Create a file named rbac-example.yaml with the following:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-role
rules:
- apiGroups: [""]  # Empty string represents the core API group
  resources: ["pods"]
  verbs: ["create", "delete"]---apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: my-role
subjects:
- kind: User
  name: test-user---apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: my-cluster-role
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "create", "delete"]---apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-cluster-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: my-cluster-role
subjects:
- kind: User
  name: test-user

In this example:

Apply the configuration to your Kubernetes cluster:

kubectl apply -f rbac-example.yaml

Verify that the role assignments work by deleting the nginx-deployment with test-user:

$ kubectl delete deployment nginx-deployment --context=test-user-context
deployment.apps "nginx-deployment" deleted

Troubleshooting RBAC Issues

When you start using these RBAC commands, you'll eventually run into configuration errors or errors in the environment. The following methods can be used to troubleshoot common problems.

Check That the Kubernetes Version Supports RBAC

Kubernetes versions 1.7 and earlier do not support RBAC. If you have significant issues using the commands above to manipulate RBAC for your Kubernetes cluster, first check to see that the Kubernetes version used is 1.8 or higher.

Fortunately, kubectl commands are available in those earlier versions:

kubectl version

The version command displays both the client version (the version of the kubectl command line tool) and the server version (the version of the Kubernetes API server).

From the response, look for the line with Server Version, and check the GitVersion:

Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.15",

As long as the GitVersion is 1.8 or higher, RBAC-related configuration is available.

Verify the RBAC Configuration Is Correct

kubectl is used to review and manage the cluster roles and role bindings. You should also get familiar with kubectl get and kubectl describe, as these are fantastic resources to help validate the permissions in your environment. The kubectl get command lists the roles or role bindings that have been defined. For any of the listed roles, kubectl can also retrieve detailed information about the associated permissions of a role. The command for this is kubectl describe, with the name of the role or role binding following the verb. The output of the kubectl describe command for a role binding includes the name, labels, annotations, role, kind (usually "Role"), subjects associated with the role binding, namespace, and additional details.

The following are some commonly used commands for get and for describe:

In addition to those commands, kubectl can-i is a useful command to determine what a principal's permissions are, straight from the command line.

Check if a specific principal or group has the necessary permissions to perform a certain action on a resource using this syntax:

kubectl auth can-i [verb] [resource]

For example, to check if a user named john_smith can create pods, you would run:

kubectl auth can-i create pods --as john_smith

Check the RBAC-Related Logs

Kubernetes has several RBAC-related logs available to the administrator. Two of the most important sources are API server logs and RBAC events. RBAC events can be retrieved using the following command:

kubectl get events

This provides records of role creation and changes, role bindings, and cluster-related RBAC modifications.

Methods to access the API server logs vary depending on the system that hosts your API server. If the system is a Linux distribution, then the logs are likely available in /var/log. However, if your cluster is hosted on a commercial service, the provider may make it easier to access the logs through their service management interface.

Use Tools like RBAC Manager to Identify and Fix Issues

RBAC Manager is an open source tool that simplifies the review and management of roles in Kubernetes. It allows more intuitive management of role bindings and the assignment and removal of role permissions. A newer tool, which complements RBAC Manager, is RBAC Lookup. RBAC Lookup allows you to determine the role bindings for any principal using the command line.

Best Practices for Enforcing RBAC in Kubernetes

Enforcement of your RBAC scheme is crucial for its success. However, RBAC enforcement is wasted without careful definitions of your RBAC plans and policies. The following best practices should inform your planning for a strong RBAC program.

Use the Principle of Least Privilege

Your application and cluster architecture requirements should guide your decisions when adding permissions to roles. The best rule is to create a role with exactly the permissions needed for the principal's tasks. The least necessary permissions for roles will change over time, and you will need to add or remove permissions from roles as the environment evolves. Least privilege also includes checking that all roles assigned to a principal are still necessary.

When applications are decommissioned, workers change positions, or IT departments reorganize, principals will be created, deleted, and bound and unbound from roles.

You should ensure you're auditing cluster role permissions with kubectl describe clusterroles and role permissions using kubectl describe roles.

Use ClusterRole and ClusterRoleBindings Only When Necessary

The proliferation of cluster control poses a significant security and operational risk. These roles are the equivalent of administrator privileges across the entire cluster. Principals with cluster-wide permissions have the ability to impact multiple (or all) nodes within the cluster. These roles and bindings are the most common end target of a compromise, as cluster-wide permissions are the most efficient way to gain control over the entire target environment. Environment administrators prefer to have cluster-wide permissions for their main principals so that they can easily work without switching roles, but this also risks any mistake or malicious activity being replicated across the entire Kubernetes cluster.

Regularly Audit and Review RBAC Configurations

Roles and their assigned permissions should be kept up to date. You should regularly review the need for each role, validate the activities performed with that role, and audit whether the assigned permissions should be modified. This does not always mean that permissions are removed. Sometimes the activities performed with a role change to require greater or different access. It's vital to review and manage roles and the principals to ensure permissions and roles are still necessary. This should be performed on a set schedule; if you do not have an access review policy in place with a timeline, reviewing every six months is a good idea.

The RBAC Lookup tool provides a simple method for looking up RBAC assignments for all of your principals, whether they're service accounts, groups, or users. You can use the following syntax:

rbac-lookup <name> [--kind type] [--output wide]

kind allows you to specify between user, group, and serviceaccount. output allows you to specify wide, which will give the binding that provided the permissions as well.

For instance, you can find out what roles the user john is assigned to with the following command:

rbac-lookup john

If you want to find out how the john account got those permissions, you would instead use this command:

rbac-lookup john –output wide

Consider Using ServiceAccounts Instead of Individual User Accounts

Service accounts follow the principle of least privilege, granting only the necessary permissions to applications and processes. When a service account is the principal, access control is based on the requirements of the individual application. One of the greatest advantages of using service accounts is that they do not require an interactive login for the environment. Therefore, you can create a service account without shell access and lower the chances that it will be compromised from outside of the cluster.

Take Extra Care When Setting Up RBAC in Multitenant Environments

In a multitenant environment, there is the possibility of naming collisions—for instance, what if someone else has a cluster role named john or someone else is following an RBAC tutorial and has named their demo cluster my-cluster? A simple way to avoid this is to assign a namespace.

A namespace provides a virtual slice of the cluster, and RBAC can be applied to the namespace, avoiding collisions and narrowing permission spread:

As mentioned, a namespace is a virtualized separation within a cluster. It does not create new pods, networks, or clusters, and it doesn't use separate resources. Instead, a namespace creates a separate name you can refer to for managing your resources. Namespaces also let you apply RBAC policies to control access and provide resource quota management.

The syntax for creating a new namespace is straightforward. You can create the namespace without adding any resources:

kubectl create namespace <namespace-name>

So, if you wanted the username john to have its own JohnPod, you could use the following:

kubectl create namespace JohnPod

You'd assign the pods Pod1 and Pod2 to it by using label:

kubectl label pod Pod1 -n JohnPod
kubectl label pod Pod2 -n JohnPod

You would add a role with permission to view in the new namespace as follows:

kubectl create role JohnPodView --verb=view --namespace=JohnPod

And then, if you wanted the john user to have permission to view the content of the namespace JohnPod, you would create a role binding and assignment:

kubectl create rolebinding JohnPodView --user=john --namespace=JohnPod

Consider Using a Third-Party Tool to Manage Your Workload's RBAC Configuration

Depending on the size of your environment, you might have more access setup and changes than you can practically manage. In that case, you should consider offloading some of the work onto a purpose-made third-party tool. For instance, you could use kubeaudit to help you audit Kubernetes clusters for security-related misconfigurations. It can provide detailed reports on RBAC policies, roles, and role bindings with the following command:

kubeaudit rbac

However, it's important to note that the tool cannot determine whether your setup is correct or not. You still need to perform the critical task of analyzing the suitability of policies, roles, and bindings. Nonetheless, the tool lightens the burden by identifying and reporting on the various components of your RBAC setup.

Conclusion

While the architecture of a Kubernetes cluster differs from traditional setups, RBAC remains applicable. RBAC enables efficient access management by defining roles and bindings, ensuring users have the minimum necessary privileges to carry out their tasks. By enforcing the principle of least privilege, RBAC reduces the risk of unauthorized access and limits potential damage in the event of a security breach. It provides a scalable and manageable solution for controlling access across the various components of a Kubernetes cluster, including the control plane and worker nodes.

You can manage RBAC with your hosting provider's default tools. However, the Kubernetes cluster's access manager must be able to use the kubectl command to review and enforce access configurations on the clusters.