Securing Multi-Tenant Kubernetes Clusters with Falco

We all agree that Kubernetes is running almost everywhere now, and nearly every other company has adopted it. The next step companies are taking is virtualizing this layer and creating multiple virtual Kubernetes clusters on a single host cluster.

This is what we call multi-tenancy in Kubernetes, where multiple Kubernetes clusters run on a single host cluster while sharing the platform stack. With this adoption, ensuring the security of multi-tenant Kubernetes clusters becomes a crucial challenge. These clusters, shared among multiple teams or applications, require robust isolation and monitoring to detect and mitigate potential threats. Falco, a CNCF graduated project, provides runtime security for cloud-native environments, making it an ideal choice for enhancing the security of multi-tenant clusters.

In this blog, we will explore the concepts of multi-tenancy in Kubernetes using vCluster, introduce Falco's powerful runtime threat detection capabilities, and showcase a demo integrating vCluster and Falco to monitor and secure a multi-tenant environment.

What is Multi-Tenancy in Kubernetes?

Multi-tenancy in Kubernetes allows multiple users, teams, or workloads to share a single Kubernetes cluster while maintaining isolation. This model is referred to as control plane per tenant over cluster per tenant, meaning every user or team gets their own control plane as part of a virtual cluster rather than a separate physical cluster.

This approach enables better resource utilization and cost efficiency and offers several key benefits:

1. Better Resource Isolation: Ensures workloads do not interfere with each other.
2. Autonomy: Grants admin access to tenants, giving them greater control over their environments.
3. Reduced Complexity: Simplifies the creation and distribution of clusters.

vCluster has been leading the multi-tenancy front, making it super simple to create virtual Kubernetes clusters with separate control planes for each cluster. Each virtual cluster operates independently, offering a high degree of isolation.

As an admin, one of the critical responsibilities is to ensure runtime security within these clusters. For this, let’s explore Falco, a CNCF graduated project that is incredibly helpful in addressing runtime security challenges.

Introducing Falco for Runtime Security

Falco, originally developed by Sysdig, is a runtime security tool designed to detect unexpected or malicious activities in Kubernetes clusters. Leveraging eBPF (Extended Berkeley Packet Filter), Falco monitors kernel events and applies custom rules to identify threats in real time.

Key Features of Falco:

• Real-Time Detection: Alerts on policy violations and suspicious activities.
• Custom Rules: Allows you to create rules tailored to your use case, while also providing a wide variety of predefined, out-of-the-box rules.
• Kubernetes Integration: Monitors Kubernetes-specific events such as pod creation, exec commands, and privilege escalations.

By combining vCluster's multi-tenancy capabilities with Falco's powerful threat detection, you can build secure and efficient multi-tenant Kubernetes environments.

Demo: Securing Multi-Tenant Kubernetes Clusters with Falco and vCluster

In this demo, we'll set up a vCluster for multi-tenancy, deploy Falco for monitoring, and simulate suspicious activity to observe Falco in action.

1. Pre requisites 

Kubernetes Cluster

Having a Kubernetes cluster, you can do this demo on any Kubernetes cluster on any provider of your choice. For this demo we are using a three node digital ocean Kubernetes cluster. 

kubectl get nodes

NAME                   STATUS   ROLES    AGE   VERSION
pool-wq5qz4aix-wt0h3   Ready    <none>   58d   v1.31.1
pool-wq5qz4aix-wt0h8   Ready    <none>   58d   v1.31.1
pool-wq5qz4aix-wt0hu   Ready    <none>   58d   v1.31.1

Don’t forget to have kubectl CLI to manage your cluster!

Install vCluster

vCluster simplifies the creation of virtual clusters. This tutorial uses a Linux AMD64 system. To install, use the command below. For more information, depending on your architecture and system type, the official documentation has a lot more ways:

curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/download/v0.21.1/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster 
/usr/local/bin && rm -f vcluster

vcluster version

2. Install Falco

Deploy Falco in the host cluster for runtime security:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm upgrade --install falco falcosecurity/falco --namespace falco --create-namespace

Output:

kubectl get pods -n falco
NAME          READY   STATUS    RESTARTS   AGE
falco-5f9jb   2/2     Running   0          5m5s
falco-s5w5m   2/2     Running   0          5m5s

3. Create a vCluster

Create a namespace for the vCluster and initialize it. Run the below commands on the host cluster.

kubectl create namespace vcluster
vcluster create ssh -n vcluster

Output:

4. Simulate Suspicious Activity

To simulate activity within the vCluster, let’s deploy a ssh server on the virtual cluster. Once the cluster is created, it should automatically switch the context that you can check using below command 

kubectl config current-context

Output:

4.1 Deploy a dummy SSH server:

helm repo add securecodebox https://charts.securecodebox.io/
helm repo update
helm install my-dummy-ssh securecodebox/dummy-ssh --version 3.14.3

4.2 Gain access to the dummy SSH server:

You can set the password of your ssh server, in this case, I am setting it to password

kubectl exec -it <podname> -- /bin/bash
passwd root
cat /etc/shadow

Output:

4.3 Forward the SSH service:

Here, you need to use two tabs: one for performing the port forward and another for SSH access to then view the sensitive file.

kubectl port-forward svc/my-dummy-ssh 5555:22
ssh -p 5555 root@127.0.0.1

Output:

5. Observe Falco Logs

Falco will detect and log suspicious activity, such as accessing /etc/shadow. You need to switch the context back to the host cluster and then run the following command on the host cluster:

kubectl config use-context kubernetes-admin@kubernetes
kubectl logs -f daemonset/falco -n falco | grep shadow

Output:

This demonstrates how Falco can detect potential threats in real time, even within isolated vClusters.

Note - You can check on which cluster the pod from the virtual cluster is created and then see the falco pod logs from that particular node.

Conclusion

By combining vCluster's multi-tenancy model with Falco's runtime security capabilities, you can build secure, efficient, and scalable Kubernetes environments. This approach ensures isolation between tenants while maintaining visibility into potential threats across the entire stack.

Learn More

• Falco GitHub Repository
• vCluster Documentation

If you’re building or managing multi-tenant Kubernetes clusters, start using vCluster with Falco today to achieve robust security without compromising efficiency! If you have further questions, don’t forget to join our Slack!

References

The dummy ssh server example here is inspired by Sysdig’s blog on honeypots with vCluster and Falco. You can find the complete article here.