Table of Contents
The more popular Kubernetes becomes, the more tempting it becomes for bad actors to target its systems. Attacks like unauthorized access or external attacks on pods can result in data loss, data leakage, or even complete system compromise.
Fortunately, isolation in Kubernetes, with the help of segregation at different levels, can protect your cluster from threats. And if you need multi-tenancy, isolation helps resources coexist without impacting each other.
This article covers the different types of isolation features available in Kubernetes, how they implement isolation, and their limitations. By the end, you should have a strong understanding of Kubernetes’s isolation capability, so you can craft your security posture as required.
Types of Isolation in Kubernetes
In Kubernetes, hardware resources are generally shared with the help of virtual segregation and namespaces. These basic abstraction layers help you create more virtual clusters from one physical cluster.
This introduces a need for isolation. Your virtual clusters should have segregation built in to effectively divide cluster resources between multiple users, teams, or projects. Generally, you want to ensure segregation by network, API, and cluster isolation when a namespace is concerned.
API Isolation
Kubernetes implements API isolation through a variety of approaches, including roles, role bindings, and cluster roles. These techniques offer a way to manage access to the Kubernetes API by limiting access to certain users or groups for various resources.
Of course, role-based access control (RBAC) means you need to configure access to your resources. To demonstrate, let’s create a namespace and deploy two pods in the default namespace and the newly created test-namespace.
kubectl create namespace test-namespace
kubectl run nginx-test --image=nginx -n test-namespace
kubectl run nginx-default --image=nginx
If successful, you can see the following success message on your terminal:
namespace/test-namespace created
pod/nginx-test created
pod/nginx-default created
Create a role that grants permissions to access only the pods resource in the namespace:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: test-namespace
name: test-master-role
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
You’ll also need a RoleBinding that associates the role with a specific user or group. Here, it is alice:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: test-master-binding
namespace: test-namespace
subjects:
- kind: User
name: alice
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: test-master-role
apiGroup: rbac.authorization.k8s.io
Apply the manifests, and when you query your test-namespace namespace for the pods, you’ll get them. However, when you try fetching the resources from different namespaces, the attempt will return a Forbidden error message.
hrittik@Azure:~$ kubectl get pods -n test-namespace --as alice
NAME READY STATUS RESTARTS AGE
nginx-test 1/1 Running 0 44m
hrittik@Azure:~$ kubectl get pods --as alice
Error from server (Forbidden): pods is forbidden: User "alice" cannot list resource "pods" in API group "" in the namespace "default"
You can list the permissions in more detail with the help of the kubectl auth can-i --list -n test-namespace --as alice command. For pods you have access to, you can see get, list, and watch as the user alice.
hrittik@Azure:~$ kubectl auth can-i --list -n test-namespace --as alice
Resources Non-Resource URLs Resource Names Verbs
selfsubjectaccessreviews.authorization.k8s.io [] [] [create]
selfsubjectrulesreviews.authorization.k8s.io [] [] [create]
pods [] [] [get list watch]
As you can see, segregation in Kubernetes namespaces is possible with RBAC. But there’s also ClusterRole, which allows users or groups to have access to certain cluster-level resources. They can override API isolation at the namespace level and are intended for cluster administrators. Clearly, configuring access at different levels is crucial in order to provide appropriate isolation.
To demonstrate this, create a ClusterRole that grants access to all pods across all namespaces:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: pod-reader
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
You’ll also need a ClusterRoleBinding that associates the ClusterRole with a user. Here, it is bob or a group:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: pod-reader-binding
subjects:
- kind: User
name: bob
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: pod-reader
apiGroup: rbac.authorization.k8s.io
Apply these manifests, and when you query the permissions, you’ll see it has the capacity to get, list, and watch pods for all namespaces.
hrittik@Azure:~$ kubectl auth can-i --list --as bob
Resources Non-Resource URLs Resource Names Verbs
selfsubjectaccessreviews.authorization.k8s.io [] [] [create]
selfsubjectrulesreviews.authorization.k8s.io [] [] [create]
pods [] [] [get list watch]
You can verify this by using kubectl get pods -A to get pods for all the namespaces, including pods from kube-system. Here’s the output:
NAMESPACE NAME READY STATUS RESTARTS AGE
default busybox 1/1 Running 0 157m
default nginx 1/1 Running 0 69m
default nginx-default 1/1 Running 0 68m
kube-system ama-logs-4qgt2 2/2 Running 0 24h
kube-system ama-logs-6xfrx 2/2 Running 0 24h
kube-system ama-logs-rs-66798f6f45-dkv8h 1/1 Running 0 24h
kube-system azure-ip-masq-agent-9hsxq 1/1 Running 0 24h
kube-system azure-ip-masq-agent-b8vgq 1/1 Running 0 24h
kube-system cloud-node-manager-b5tqd 1/1 Running 0 24h
kube-system cloud-node-manager-zsjqm 1/1 Running 0 24h
kube-system coredns-6b9cb549f4-8ftxw 1/1 Running 0 23h
kube-system coredns-6b9cb549f4-8wr29 1/1 Running 0 24h
kube-system coredns-autoscaler-7fd9d8d7cf-mmvtj 1/1 Running 0 24h
kube-system csi-azuredisk-node-mrrwk 3/3 Running 0 24h
kube-system csi-azuredisk-node-sgf5m 3/3 Running 0 24h
kube-system csi-azurefile-node-d2swc 3/3 Running 0 24h
kube-system csi-azurefile-node-rmsgk 3/3 Running 0 24h
kube-system konnectivity-agent-f9cb5574f-6l2gk 1/1 Running 0 23h
kube-system konnectivity-agent-f9cb5574f-cgqp4 1/1 Running 0 23h
kube-system kube-proxy-mrm24 1/1 Running 0 18h
kube-system kube-proxy-qc79z 1/1 Running 0 18h
kube-system metrics-server-59996b9895-6jjk8 2/2 Running 0 24h
kube-system metrics-server-59996b9895-96q6r 2/2 Running 0 24h
test-namespace nginx-test 1/1 Running 0 65m
test nginx-test 1/1 Running 0 68m
While you can query for a pod object, a non-secure cluster with badly designed ClusterRoles introduces a chance for compromised security. Even with proper RBAC, you must still provide isolation to required entities.
Network Isolation
Network isolation between namespaces is a way to prevent communication between different networks or network segments, and therefore helps secure access. However, there’s not much that Kubernetes does for network isolation across namespaces, but let’s have a look at your options firsthand.
First, create a namespace where you want to deploy your applications:
kubectl create namespace network
With a namespace successfully created, you can now deploy, say, a sample NGINX image to your namespace and expose the ports so you can access the endpoints.
kubectl run nginx --image=nginx --labels app=nginx --namespace network --expose --port 80
A successful deployment will create a Pod and a Service in your namespace:
service/nginx created
pod/nginx created
To list all the created objects, query with the kubectl get command and pass the namespace name with the -n flag:
hrittik@Azure:~$ kubectl get all -n network
NAME READY STATUS RESTARTS AGE
pod/nginx 1/1 Running 0 45s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/nginx ClusterIP 10.0.124.23 <none> 80/TCP 45s
Now, you can query your IP, as you’ll need to check if a pod can access the endpoint. You can either run kubectl describe and then find the IP field, or run the following command to filter out the IP field:
kubectl describe pod nginx -n network | grep -w 'IP:' -m 1 | awk '{print $2}'
To test the isolation, create a pod in the default namespace and start a shell session. You can run the pod in the network namespace using the following command:
kubectl run busybox --rm -ti --image=alpine -- /bin/sh
Install the required curl utility:
apk add curl
Once the busybox pod is created, the curl utility is installed, and the IP address of the deployed NGINX pod is available, you can access NGINX content from the busybox pod. As you can see, the NGINX pod running in the network namespace may indeed be accessed by the busybox pod.
You can conduct the same experiment with several namespaces and achieve the same outcome.
/ # curl 10.244.1.12:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
You probably noticed that your Kubernetes cluster’s default behavior does not ensure any network-level isolation across the namespaces. There’s no network filtering happening as a layer of security. That means you’ll need to configure and maintain network policies if you want to configure network isolation.
Cluster Isolation
There’s no cluster-level isolation built into Kubernetes by default. If a namespace is compromised, it can potentially affect other namespaces and resources in the same cluster, therefore compromising your entire cluster. For example, an attacker who gains access to a pod in one namespace can potentially access other pods and services in the same cluster, even if they are in a different namespace.
Let’s demonstrate this. First, build a privileged pod and then access the node by launching a shell session. The session should have full access to the node host's file system, PID, and network namespaces (Linux namespaces, specifically).
You can build a privileged pod with the following manifest and apply it to your cluster using kubectl apply -f <file_name>:
apiVersion: v1
kind: Pod
metadata:
name: busybox # Name of the pod
spec:
containers:
- name: shell # Name of the container
image: ubuntu # Docker image used for the container
stdin: true # Allocate a tty for interactive shell
securityContext:
privileged: true # Give the container privileged access
volumeMounts:
- name: host-root-volume # Name of the volume to be mounted
mountPath: /host # Mount path inside the container
readOnly: true # Mount volume as read-only
volumes:
- name: host-root-volume # Name of the volume to be created
hostPath:
path: / # Path on the host machine to mount as a volume
hostNetwork: true # Use the host network namespace
hostPID: true # Use the host PID namespace
Once the creation is completed, use the following command to look at which node your pod is running. You’ll need to verify it in the lateral steps:
controlplane $ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 1/1 Running 0 14m 172.30.2.2 node01 <none> <none>
With that data, you can exec into your pods using the following command and have access to a shell session:
kubectl exec -it busybox -- chroot /host
The hostname command returns the name of the node your container was running on—clearly, it’s a simple matter to be compromised without proper isolation.
$ hostname
node01
With access to the node, a bad actor can access the whole system. Installing malware or changing system configuration will not be difficult, which will compromise your security posture. Moreover, there’s no namespace limitation, as you’re accessing the host system directly.
Enforcing isolation that extends to your cluster is important for a strong security posture.
Conclusion
Kubernetes helps you quickly meet the demands of users by abstracting away a lot of infrastructure complexity. However, as you’ve seen in this article, Kubernetes is lacking when it comes to isolation and security. You’ll need to achieve network, namespace, and cluster isolation in ways that don't compromise your security posture.
In Part 2 of this series, you’ll see how to achieve proper isolation in Kubernetes with Loft Labs. You’ll learn how to unlock Kubernetes efficiency at scale with solutions like vcluster for multi-tenant clusters. You’ll be able to enforce strict security boundaries and provide isolation to your cluster with very low management overheads.
