Using Istio for Multicluster Management in Kubernetes on EKS

Growing enterprises and organizations using Kubernetes often face challenges in managing multitenancy effectively. Simply relying on namespaces for resource management isn't enough, which leads them to expand from single-cluster to multicluster setups. However, this expansion can be costly and complicated, especially when it comes to configuring traffic routing.

Istio, an open source service mesh, offers advanced traffic management capabilities such as routing, load balancing, and canary releases. Its integration with vCluster, developed by Loft Labs, brings forth an efficient approach. vCluster allows you to create virtual Kubernetes clusters within an existing cluster that act as regular Kubernetes clusters and utilize the physical resources of the underlying cluster. This integration ensures increased efficiency and ease of management. Istio's capabilities enable improved service mesh and application performance, allowing for seamless communication and efficient resource utilization. Moreover, it provides enhanced security, detailed tracing, and monitoring capabilities, bolstering the overall reliability and high availability across clusters.

This tutorial demonstrates how to set up multiple Kubernetes clusters using vCluster and configure Istio for seamless multitenancy management.

Prerequisites

To complete this tutorial, you'll need a few things in your local development environment:

Note: The resources you'll set up in this tutorial will incur some costs on AWS. Make sure to destroy all resources when you're done with the tutorial.

Setting Up the Clusters

You first need to create an EKS cluster on AWS. This is the cluster where you'll deploy your virtual clusters with vCluster. Open your AWS console, navigate to the EKS service, and create a new cluster called istio-vcluster. The default settings should be sufficient, but feel free to configure the cluster depending on your needs.

Wait a few minutes for the cluster to become active. You can then add some managed node groups. Go to the Compute tab and add a new node group. This tutorial uses the name istio-vcluster-nodegroup, but you can set the name to anything you want.

Select the node IAM role:

Next, make sure the AmazonEBSCSIDriverPolicy policy is attached to the role. This is important because vCluster creates EBS volumes for each cluster you set up, and you need to set the correct IAM policies that are needed.

You can leave the remaining configurations as defaults. Now, go to the Add-ons tab of your cluster, select Get more add-ons, and select the Amazon EBS CSI Driver add-on:

Wait for your node group to be ready, then run the following command to add your cluster to your kubeconfig context, replacing <cluster-name> with the name of the EKS cluster you created:

aws eks update-kubeconfig --name <cluster-name>

Next, run the command kubectl get all. You should see an output similar to the following to verify everything is working fine:

Before you set up your virtual clusters, it's important to know that on public clouds like EKS, vCluster automatically directs requests from your computer to the virtual cluster API. However, because you'll later deploy Istio, you must expose the virtual cluster API endpoint through a load balancer.

In this tutorial, you'll use the --expose flag with vcluster to create a service of type LoadBalancer. However, if you were creating multiple vClusters, you'd need to consider the cost of setting up numerous load balancers. To solve this issue, you'd use a Kubernetes ingress.

Create a virtual cluster called cluster-one with the following command:

vcluster create cluster-one --expose

vCluster automatically switches your kubeconfig context to the new virtual cluster. Disconnect from the virtual cluster and return to the one you created in EKS by running the command vcluster disconnect. Run the command kubectl get ns, and you should see the new cluster created by vCluster:

Run vcluster create cluster-two --expose to create the second virtual cluster.

Managing the Clusters with Istio

Now that you've successfully created two virtual clusters, let's take a look at installing Istio and configuring it to work with multiple clusters.

Setting Up Istio

You first need to set up two environment variables for the kubeconfig context of the virtual clusters you configured earlier. You can find these values in the contexts section of your kubeconfig file located at ~/.kube/config:

export CTX_CLUSTER1=vcluster_cluster-one_vcluster-cluster-one_arn:aws:eks:us-east-1:<aws-account-id>:cluster/istio-vcluster
export CTX_CLUSTER2=vcluster_cluster-two_vcluster-cluster-two_arn:aws:eks:us-east-1:<aws-account-id>:cluster/istio-vcluster

Next, you need to configure certificates for your clusters. Download the latest version of Istio and add it to your path by running the following commands:

curl -L https://istio.io/downloadIstio | sh -
cd istio-1.19.0
export PATH=$PWD/bin:$PATH

Before you install Istio, you need to configure certificates for each of your virtual clusters. Run the following commands:

mkdir -p certs
pushd certs
make -f ../tools/certs/Makefile.selfsigned.mk root-ca

This creates a certs directory and creates a root certificate inside it. Create intermediate certificates for each virtual cluster by running the following commands:

make -f ../tools/certs/Makefile.selfsigned.mk cluster1-cacerts # for cluster-one
make -f ../tools/certs/Makefile.selfsigned.mk cluster2-cacerts # for cluster-two 

This should create two folders named cluster1 and cluster2 with the intermediate certificates inside. Now, connect to each cluster and create a Kubernetes secret called cacerts, whose value will be the files in each folder:

# Cluster one
vcluster connect cluster-one
kubectl create namespace istio-system
kubectl create secret generic cacerts -n istio-system \
      --from-file=cluster1/ca-cert.pem \
      --from-file=cluster1/ca-key.pem \
      --from-file=cluster1/root-cert.pem \
      --from-file=cluster1/cert-chain.pem

# Cluster two
vcluster connect cluster-two
kubectl create namespace istio-system
kubectl create secret generic cacerts -n istio-system \
      --from-file=cluster2/ca-cert.pem \
      --from-file=cluster2/ca-key.pem \
      --from-file=cluster2/root-cert.pem \
      --from-file=cluster2/cert-chain.pem
vcluster disconnect

Create a file called cluster1.yaml and input the following code:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster1
      network: network1

Create another file called cluster2.yaml and input the following code:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster2
      network: network1

Connect to cluster-one by running vcluster connect cluster-one and deploy the Istio operator by running istioctl install --context="${CTX_CLUSTER1}" -f cluster1.yaml. Repeat the same steps for cluster-two.

Run the following commands to create a secret in cluster-one that provides access to cluster-two, and vice versa:

# Cluster one
vcluster connect cluster-one
istioctl create-remote-secret \
    --context="${CTX_CLUSTER2}" \
    --name=cluster2 | \
    kubectl apply -f - --context="${CTX_CLUSTER1}"

# Cluster two
vcluster connect cluster-two
istioctl create-remote-secret \
    --context="${CTX_CLUSTER1}" \
    --name=cluster1 | \
    kubectl apply -f - --context="${CTX_CLUSTER2}"
vcluster disconnect

You've now successfully installed Istio across multiple clusters.

Testing the Istio Installation

Now, let's test if everything is working properly. Create a file called deployment-1.yaml and add the following code:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: loft-app
spec:
  selector:
    matchLabels:
      app: loft-app
  template:
    metadata:
      labels:
        app: loft-app
    spec:
      containers:
      - name: loft-app
        image: utibeabasi6/loft-app
        env:
          - name: APP_ENV
            value: vcluster-1
        resources:
          limits:
            memory: "100Mi"
            cpu: "100m"
        ports:
        - containerPort: 5000---apiVersion: v1
kind: Service
metadata:
  name: loft-app
  labels: 
    service: loft-app
spec:
  type: LoadBalancer
  selector:
    app: loft-app
  ports:
  - port: 5000
    targetPort: 5000

Similarly, create a file called deployment-2.yaml and add the following code:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: loft-app
spec:
  selector:
    matchLabels:
      app: loft-app
  template:
    metadata:
      labels:
        app: loft-app
    spec:
      containers:
      - name: loft-app
        image: utibeabasi6/loft-app
        env:
          - name: APP_ENV
            value: vcluster-2
        resources:
          limits:
            memory: "100Mi"
            cpu: "100m"
        ports:
        - containerPort: 5000---apiVersion: v1
kind: Service
metadata:
  name: loft-app
  labels:
    service: loft-app
    app: loft-app
spec:
  type: LoadBalancer
  selector:
    app: loft-app
  ports:
  - port: 5000
    targetPort: 5000

In the previous steps, you created and Dockerized a simple Flask app that reads the value of APP_ENV, which is set to vcluster-1 on the first deployment and vcluster-2 on the second deployment. The above code creates a Kubernetes deployment that runs this Docker image in your cluster.

Use the following code to label your namespaces so that Istio can discover your pods:

vcluster connect cluster-one
kubectl label namespace default \
    istio-injection=enabled
kubectl apply -f deployment-1.yaml

Running this command connects to cluster-one and adds a label to the default namespace where your resources will be deployed. This label enables Istio to inject sidecar containers into the pods within that namespace. Then, it applies the configurations specified in deployment-1.yaml.

Repeat the same step for cluster-two and deployment-2.yaml.

The following code establishes a connection to cluster-one and sets up a test pod, which sends requests to the loft-app service in each cluster:

vcluster connect cluster-one
kubectl run curl-pod --image=curlimages/curl --restart=Never --command -- sleep infinity
kubectl exec -it curl-pod -- sh

Now, send some requests to the loft-app service by running the following command:

curl -sS loft-app.default:5000

The value of APP_ENV alternates between vcluster-1 and vcluster-2.

The following image shows an APP_ENV value of vcluster-1:

In the next screenshot, the APP_ENV value is vcluster-2:

Repeat the same process for the second cluster, and you should see the same results. This means your Istio installation is working properly!

Conclusion

In this tutorial, you learned how to install and configure Istio for multicluster management in Kubernetes on EKS by deploying a sample application.

Harness the power of Istio and vCluster to enhance your Kubernetes setup with efficient resource utilization, robust isolation, advanced traffic control, and heightened security measures. Explore the benefits of multitenancy in your environment by visiting the vCluster website to see how it can enhance your Kubernetes environment management.