Kubernetes Multi-Tenancy with Argo CD And Loft

Daniel Olaogun

July 18, 2022

11 Minute Read

This is some text inside of a div block.

Get Started Now

TL;DR:

Kubernetes multi-tenancy allows multiple users/workloads to share cluster resources in isolation.
Loft enables multi-tenancy using vClusters, creating virtual clusters within a host cluster.
Argo CD complements Loft by automating deployment of applications via GitOps.
Implementing multi-tenancy reduces operational overhead and improves application security.
The combination of Loft and Argo CD provides a robust system for managing multi-tenancy in Kubernetes clusters.

Kubernetes multi-tenancy is a mode of operation in Kubernetes where multiple users or workloads share cluster resources in isolation from each other. The users/workloads are referred to as tenants and are independent of each other while at the same time integrated into the cluster.

Managing resources amongst workloads is one of many ways multi-tenancy is useful for organizations. For example, if you host multiple related or different applications in your cluster, over time, some of these will grow bigger and start competing for resources with other applications, which will inevitably affect their overall performance.

In order to ensure fair resource usage, multi-tenancy architecture allows you to group the applications into namespaces or virtual clusters (vclusters), and to allocate them the resources they need. This, in turn, ensures the applications work effectively without being affected by other tenants in the cluster. This is just one example of where implementing multi-tenancy is important.

There are several ways of implementing multi-tenancy in your cluster, including namespacing, vclusters, and so on. Furthermore, there are also Kubernetes tools like Loft and Argo CD that make implementation and management of multi-tenancy in your Kubernetes cluster a breeze. In this article, you’re going to learn about multi-tenancy, its benefits, and how Loft and Argo CD can help you achieve multi-tenancy in your cluster.

Tutorial: Getting the most out of your Delivery Pipeline with Loft & Argo CD

Why Multi-Tenancy is Essential for Kubernetes Environments

Aside from resource allocation, there are many other reasons why you might need multi-tenancy in your cluster.

The Importance of Cost Effectiveness in Multi-Tenancy

Multi-tenancy architecture is better for cost saving compared to using multiple clusters.

When you deploy applications on multiple clusters, you spend more on compute resources compared to a single cluster with multi-tenancy because components such as master nodes require additional resources. You also save overhead costs on cluster management components because you can reuse cluster-wide resources that include load balancers, ingress controllers, and so on.

Enhancing Application Security Through Multi-Tenancy

When an application is compromised in a single-tenant cluster, it has a greater chance of compromising other applications running in the cluster, because the applications are sharing resources and they can interact with each other. Therefore, when one application is compromised it can easily spread to other applications. However, in a multi-tenant cluster, the applications are isolated from each other via namespaces; therefore, there’s a lesser chance of a compromised application affecting other applications.

Reducing Operational Overhead with Multi-Tenancy

Kubernetes multi-tenancy removes the operational overhead of managing multiple clusters. There are more complexities involved in managing multiple clusters, including setting up the clusters (which can be time-consuming), authentication, maintaining the clusters, authorization, implementing monitoring and logging tools, and so much more. However, in a multi-tenant cluster, these operational overheads are reduced; for instance, you don't need to set up a new cluster for a new application.

Leveraging Loft and Argo CD for Kubernetes Multi-Tenancy Solutions

Loft is an advanced platform for implementing self-service and multi-tenancy in your Kubernetes cluster. It implements multi-tenancy using vclusters. Traditionally, namespacing is used to achieve multi-tenancy, thereby sharing resources like ETCD storage and the API server. In contrast, Loft spins up an entirely new cluster in the host cluster, without any of the complexities of spinning up new clusters.

With Loft, each vcluster has its own API server, controller manager, and ETCD. However, it’s important to note that the vcluster resources will be running on the host cluster. As a vcluster operates like a full-blown cluster running in a host cluster, you’re not restricted in creating specific roles for the vcluster, creating namespace in the cluster, and so much more.

Loft also gives you the freedom to run custom configurations on your vclusters, which is not possible in namespacing. For instance, in namespacing multi-tenancy you can't use different cert-managers for tenants in your cluster, but each vcluster can have its own dedicated cert-manager using different versions. It’s important to note that a vcluster does not have the same overhead as a multi-cluster because it’s much easier to manage vclusters and they’re less expensive than multiple clusters.

You might be interested in our 5 part kubernetes multi-cluster series. In part 1 we talk about Kubernetes multi-cluster deployments, part 2 discusses Kubernetes cluster lifecycle, part 3 describes authentication and access control in Kubernetes, part 4 explains Kubernetes Self-service, and lastly part 5 talks about Kubernetes cost optimization.

Argo CD—a GitOps continuous delivery tool for Kubernetes under the Cloud Native Computing Foundation (CNCF) project—is a tool that complements Loft in achieving multi-tenancy in your cluster. Argo CD uses the GitOps principle in the continuous delivery and deployment of applications in your cluster. The Argo CD tool is installed in your Kubernetes cluster and then connected to a Git repository, which then watches for changes in an assigned branch. Whenever it notices any change in the branch, it pulls the code from the branch and applies it to your Kubernetes cluster.

Using Argo CD with Git also comes with some significant advantages: by utilizing the Git concept of pull requests, team members can review code/configuration changes and also restrict who can approve them in the cluster before they’re applied. This means that only approved configurations or applications can be pulled and applied to your cluster and you can also easily find out whose configuration caused certain changes, and even indirectly implement access control on your cluster.

Architecture Image of Argo CD — *Image courtesy of Argo CD*

Step-by-Step Guide to Implementing Multi-Tenancy with Loft and Argo CD

To implement multi-tenancy with Loft and Argo CD, you must first install both tools in your Kubernetes cluster. You should start with installing Loft; the process will depend on the operating system you’re using to access your Kubernetes cluster.

Installing Loft for Multi-Tenancy

In Mac Terminal and Linux Bash :

curl -s -L "https://github.com/loft-sh/loft/releases/latest" | sed -nE 's!.*"([^"]*loft-darwin-amd64)".*!https://github.com\1!p' | xargs -n 1 curl -L -o loft && chmod +x loft; sudo mv loft /usr/local/bin;

In Windows Powershell:

md -Force "$Env:APPDATA\loft"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -UseBasicParsing ((Invoke-WebRequest -URI "https://github.com/loft-sh/loft/releases/latest" -UseBasicParsing).Content -replace "(?ms).*`"([^`"]*loft-windows-amd64.exe)`".*","https://github.com/`$1") -o $Env:APPDATA\loft\loft.exe; $env:Path += ";" + $Env:APPDATA + "\loft"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User);

It’s important that you add the folder %APPDATA%\loft to the PATH environment variable after installing Loft CLI via Powershell. Once you’ve done this, reboot your computer.

Then run the next command below after installation:

loft start

The command will request your admin email. Once the command is completed, it will generate a random password that you can reset. Then, in another terminal, log in using the command below:

loft login https://localhost:9898 --insecure

For production purposes, you should configure a dedicated domain for your Loft service. For more information on this, see the Loft configuration process.

Creating a vcluster is as simple as running the following command in your terminal:

loft create vcluster [vcluster-name]

where “vcluster-name” is the name of the vcluster.

You can also use YAML file configuration to create a vcluster, by creating a YAML file that contains the following:

apiVersion: cluster.loft.sh/v1 kind: VirtualCluster metadata: name: vcluster-name # The name of your vcluster namespace: vc-host-namespace # The host namespace of the vcluster

To list all vclusters in your cluster:

# Loft loft list vcluster [vcluster-name] # Kubectl loft use space [space-name] # [space-name] is the host namespace kubectl get virtualclusters

To delete a vcluster in your cluster:

# Loft loft delete vcluster [vcluster-name] # Kubectl kubectl delete vcluster [vcluster-name]

Now that you understand the installation procedure of Loft and how to create a vcluster using Loft, the next section explains the process for installing Argo CD.

Setting Up Argo CD for Multi-Tenancy

Run the following command to install Argo CD in your cluster:

kubectl create namespace argocd kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Then, install Argo CD on your host computer, which you use to access and manage your Kubernetes cluster:

brew install argocd

You can download Argo CD for Windows by navigating to the GitHub repo and clicking "argocd-windows-amd64.exe" in the v2.3.3 section.

As with Loft, the Argo CD API server also needs to be made accessible; therefore, you have to configure a dedicated domain in production using an ingress controller of your choice.

You can also patch the API server to be changed to type Load Balancer by running the following command:

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

or via port-forwarding:

kubectl port-forward svc/argocd-server -n argocd 8080:443

You’ll need a password to log in via the CLI or the UI. Upon Argo CD installation, it autogenerates a password, which you can extract by running the following command:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

It’s important you change the password using the next command:

argocd account update-password

You can now log in via the UI or via CLI using this command:

argocd login <ARGOCD_SERVER>

where “ARGOCD_SERVER” is the dedicated domain or the load balancer IP or localhost:8080 that is made accessible via port forwarding.

Finally, in order to deploy applications to the vcluster created on your cluster through Loft, register the cluster on Argo CD so that it can recognize it.

First get all the clusters registered on your kube config, which also includes the cluster:

kubectl config get-contexts -o name

Then add the clusters:

argocd cluster add <CONTEXTNAME>

where “CONTEXTNAME” is the name of the vcluster.

Streamlining Application Deployment from Git Repositories in Your Cluster

Create a file with your application name and the following content:

#ApplicationName.yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: application-name # The name of the application namespace: argocd spec: project: default source: repoURL: https://github.com/example-project/example-app.git # The git repository containing all the application YAML file targetRevision: HEAD path: example-directory # The directory within the repo where you want Argo CD to watch destination: server: https://kubernetes.default.svc # The API server of your vclusters namespace: app_namespace # The namespace where you want the application to be developed. If this namespace is not available, Argo CD automatically creates this namespace

Then apply the file in your cluster.

Now that you’ve configured Argo CD to watch for changes in example-directory in the Git repo via ApplicationName.yaml, create a file named example-app in the example-directory using your preferred code editor and push this directory; Argo CD will automatically pull the code from the Git repo and apply it in your cluster.

Therefore, with Argo CD you can automatically create a new namespace and deploy configuration changes or applications into the cluster once it’s been approved and merged into your Git main branch.

Furthermore, for a seamless review process of code changes in your Git repository, you can implement the CODEOWNERS feature, which allows you to assign parts of your codebase to the admins in charge. This feature can be useful if you’re the main cluster administrator of a repo containing multiple directories, which signify multiple vclusters in your cluster, and you want to automatically assign reviewers to each of these directories to ensure that the right admin reviews the code changes by other users.

With the CODEOWNERS feature, you can assign different directories to their respective admins. Then, when there are code changes on those directories and a pull request is raised, the respective admins are notified to review the code.

The combination of Loft and Argo CD creates a robust and dynamic system that automates many complex tasks while also improving the security of your cluster. It ensures that application files can be thoroughly reviewed before they’re applied to the Kubernetes cluster. It also enables you to limit the amount of people with direct access to your Kubernetes API server.

How Codefresh Uses vcluster to Provide Hosted Argo CD

In Summary, Embracing Multi-Tenancy with Loft and Argo CD

In conclusion, Kubernetes multi-tenancy architecture is a good option when trying to isolate applications from each other without the overhead cost and operation of spinning up a single cluster for each application. Loft’s vclusters enable you to implement multi-tenancy in your cluster, allowing you to spin multiple "virtual" clusters in a single "host cluster."

Furthermore, complementing Loft with Argo CD, a GitOps continuous delivery Kubernetes tool, you can automate the deployment of applications into your vclusters via Git, which facilitates a robust and dynamic system.

Photo by Amira El Fohail on Unsplash