Using Istio for Multicluster Management in Kubernetes on EKS

Utibeabasi Umanah
Minute Read

Growing enterprises and organizations using Kubernetes often face challenges in managing multitenancy effectively. Simply relying on namespaces for resource management isn't enough, which leads them to expand from single-cluster to multicluster setups. However, this expansion can be costly and complicated, especially when it comes to configuring traffic routing.

Istio, an open source service mesh, offers advanced traffic management capabilities such as routing, load balancing, and canary releases. Its integration with vCluster, developed by Loft Labs, brings forth an efficient approach. vCluster allows you to create virtual Kubernetes clusters within an existing cluster that act as regular Kubernetes clusters and utilize the physical resources of the underlying cluster. This integration ensures increased efficiency and ease of management. Istio's capabilities enable improved service mesh and application performance, allowing for seamless communication and efficient resource utilization. Moreover, it provides enhanced security, detailed tracing, and monitoring capabilities, bolstering the overall reliability and high availability across clusters.

This tutorial demonstrates how to set up multiple Kubernetes clusters using vCluster and configure Istio for seamless multitenancy management.

Prerequisites

To complete this tutorial, you'll need a few things in your local development environment:

  • An AWS account: The cloud platform used in this tutorial is AWS. If you don't have an account, follow the instructions on the AWS website to create one.
  • The AWS CLI: You'll use the AWS CLI to manage resources in your AWS account from the command line. If you haven't installed or configured the CLI before, follow the tutorial in the official docs to install it and this tutorial to configure it.
  • kubectl: kubectl is a command line tool used to interact with the Kubernetes API to manage Kubernetes resources. You can find installation instructions on the Kubernetes website.
  • The vCluster CLI: This is needed in order to set up multiple virtual Kubernetes clusters on a single cluster. You can find the installation instructions on the vCluster website.
  • Istio: You'll use the command line to install Istio. Installation instructions can be found in the official documentation.
  • Note: The resources you'll set up in this tutorial will incur some costs on AWS. Make sure to destroy all resources when you're done with the tutorial.

    Setting Up the Clusters

    You first need to create an EKS cluster on AWS. This is the cluster where you'll deploy your virtual clusters with vCluster. Open your AWS console, navigate to the EKS service, and create a new cluster called istio-vcluster. The default settings should be sufficient, but feel free to configure the cluster depending on your needs.

    Creating a cluster

    Wait a few minutes for the cluster to become active. You can then add some managed node groups. Go to the Compute tab and add a new node group. This tutorial uses the name istio-vcluster-nodegroup, but you can set the name to anything you want.

    Select the node IAM role:

    Select the node role

    Next, make sure the AmazonEBSCSIDriverPolicy policy is attached to the role. This is important because vCluster creates EBS volumes for each cluster you set up, and you need to set the correct IAM policies that are needed.

    Check the node policy

    You can leave the remaining configurations as defaults. Now, go to the Add-ons tab of your cluster, select Get more add-ons, and select the Amazon EBS CSI Driver add-on:

    EBS add-on

    Wait for your node group to be ready, then run the following command to add your cluster to your kubeconfig context, replacing <cluster-name> with the name of the EKS cluster you created:

    aws eks update-kubeconfig --name <cluster-name>
    

    Next, run the command kubectl get all. You should see an output similar to the following to verify everything is working fine:

    Kubernetes get all output

    Before you set up your virtual clusters, it's important to know that on public clouds like EKS, vCluster automatically directs requests from your computer to the virtual cluster API. However, because you'll later deploy Istio, you must expose the virtual cluster API endpoint through a load balancer.

    In this tutorial, you'll use the --expose flag with vcluster to create a service of type LoadBalancer. However, if you were creating multiple vClusters, you'd need to consider the cost of setting up numerous load balancers. To solve this issue, you'd use a Kubernetes ingress.

    Create a virtual cluster called cluster-one with the following command:

    vcluster create cluster-one --expose
    

    cluster-one

    vCluster automatically switches your kubeconfig context to the new virtual cluster. Disconnect from the virtual cluster and return to the one you created in EKS by running the command vcluster disconnect. Run the command kubectl get ns, and you should see the new cluster created by vCluster:

    kubectl get ns

    Run vcluster create cluster-two --expose to create the second virtual cluster.

    Managing the Clusters with Istio

    Now that you've successfully created two virtual clusters, let's take a look at installing Istio and configuring it to work with multiple clusters.

    Setting Up Istio

    You first need to set up two environment variables for the kubeconfig context of the virtual clusters you configured earlier. You can find these values in the contexts section of your kubeconfig file located at ~/.kube/config:

    export CTX_CLUSTER1=vcluster_cluster-one_vcluster-cluster-one_arn:aws:eks:us-east-1:<aws-account-id>:cluster/istio-vcluster
    export CTX_CLUSTER2=vcluster_cluster-two_vcluster-cluster-two_arn:aws:eks:us-east-1:<aws-account-id>:cluster/istio-vcluster
    

    Next, you need to configure certificates for your clusters. Download the latest version of Istio and add it to your path by running the following commands:

    curl -L https://istio.io/downloadIstio | sh -
    cd istio-1.19.0
    export PATH=$PWD/bin:$PATH
    

    Before you install Istio, you need to configure certificates for each of your virtual clusters. Run the following commands:

    mkdir -p certs
    pushd certs
    make -f ../tools/certs/Makefile.selfsigned.mk root-ca
    

    This creates a certs directory and creates a root certificate inside it. Create intermediate certificates for each virtual cluster by running the following commands:

    make -f ../tools/certs/Makefile.selfsigned.mk cluster1-cacerts # for cluster-one
    make -f ../tools/certs/Makefile.selfsigned.mk cluster2-cacerts # for cluster-two 
    

    This should create two folders named cluster1 and cluster2 with the intermediate certificates inside. Now, connect to each cluster and create a Kubernetes secret called cacerts, whose value will be the files in each folder:

    # Cluster one
    vcluster connect cluster-one
    kubectl create namespace istio-system
    kubectl create secret generic cacerts -n istio-system \
          --from-file=cluster1/ca-cert.pem \
          --from-file=cluster1/ca-key.pem \
          --from-file=cluster1/root-cert.pem \
          --from-file=cluster1/cert-chain.pem
    
    # Cluster two
    vcluster connect cluster-two
    kubectl create namespace istio-system
    kubectl create secret generic cacerts -n istio-system \
          --from-file=cluster2/ca-cert.pem \
          --from-file=cluster2/ca-key.pem \
          --from-file=cluster2/root-cert.pem \
          --from-file=cluster2/cert-chain.pem
    vcluster disconnect
    

    Create a file called cluster1.yaml and input the following code:

    apiVersion: install.istio.io/v1alpha1
    kind: IstioOperator
    spec:
      values:
        global:
          meshID: mesh1
          multiCluster:
            clusterName: cluster1
          network: network1
    

    Create another file called cluster2.yaml and input the following code:

    apiVersion: install.istio.io/v1alpha1
    kind: IstioOperator
    spec:
      values:
        global:
          meshID: mesh1
          multiCluster:
            clusterName: cluster2
          network: network1
    

    Connect to cluster-one by running vcluster connect cluster-one and deploy the Istio operator by running istioctl install --context="${CTX_CLUSTER1}" -f cluster1.yaml. Repeat the same steps for cluster-two.

    Istio installation

    Run the following commands to create a secret in cluster-one that provides access to cluster-two, and vice versa:

    # Cluster one
    vcluster connect cluster-one
    istioctl create-remote-secret \
        --context="${CTX_CLUSTER2}" \
        --name=cluster2 | \
        kubectl apply -f - --context="${CTX_CLUSTER1}"
    
    # Cluster two
    vcluster connect cluster-two
    istioctl create-remote-secret \
        --context="${CTX_CLUSTER1}" \
        --name=cluster1 | \
        kubectl apply -f - --context="${CTX_CLUSTER2}"
    vcluster disconnect
    

    You've now successfully installed Istio across multiple clusters.

    Testing the Istio Installation

    Now, let's test if everything is working properly. Create a file called deployment-1.yaml and add the following code:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: loft-app
    spec:
      selector:
        matchLabels:
          app: loft-app
      template:
        metadata:
          labels:
            app: loft-app
        spec:
          containers:
          - name: loft-app
            image: utibeabasi6/loft-app
            env:
              - name: APP_ENV
                value: vcluster-1
            resources:
              limits:
                memory: "100Mi"
                cpu: "100m"
            ports:
            - containerPort: 5000---apiVersion: v1
    kind: Service
    metadata:
      name: loft-app
      labels: 
        service: loft-app
    spec:
      type: LoadBalancer
      selector:
        app: loft-app
      ports:
      - port: 5000
        targetPort: 5000
    

    Similarly, create a file called deployment-2.yaml and add the following code:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: loft-app
    spec:
      selector:
        matchLabels:
          app: loft-app
      template:
        metadata:
          labels:
            app: loft-app
        spec:
          containers:
          - name: loft-app
            image: utibeabasi6/loft-app
            env:
              - name: APP_ENV
                value: vcluster-2
            resources:
              limits:
                memory: "100Mi"
                cpu: "100m"
            ports:
            - containerPort: 5000---apiVersion: v1
    kind: Service
    metadata:
      name: loft-app
      labels:
        service: loft-app
        app: loft-app
    spec:
      type: LoadBalancer
      selector:
        app: loft-app
      ports:
      - port: 5000
        targetPort: 5000
    

    In the previous steps, you created and Dockerized a simple Flask app that reads the value of APP_ENV, which is set to vcluster-1 on the first deployment and vcluster-2 on the second deployment. The above code creates a Kubernetes deployment that runs this Docker image in your cluster.

    Use the following code to label your namespaces so that Istio can discover your pods:

    vcluster connect cluster-one
    kubectl label namespace default \
        istio-injection=enabled
    kubectl apply -f deployment-1.yaml
    

    Running this command connects to cluster-one and adds a label to the default namespace where your resources will be deployed. This label enables Istio to inject sidecar containers into the pods within that namespace. Then, it applies the configurations specified in deployment-1.yaml.

    Repeat the same step for cluster-two and deployment-2.yaml.

    The following code establishes a connection to cluster-one and sets up a test pod, which sends requests to the loft-app service in each cluster:

    vcluster connect cluster-one
    kubectl run curl-pod --image=curlimages/curl --restart=Never --command -- sleep infinity
    kubectl exec -it curl-pod -- sh
    

    Now, send some requests to the loft-app service by running the following command:

    curl -sS loft-app.default:5000
    

    The value of APP_ENV alternates between vcluster-1 and vcluster-2.

    The following image shows an APP_ENV value of vcluster-1:

    Value of `APP_ENV` is `vcluster-1`

    In the next screenshot, the APP_ENV value is vcluster-2:

    Value of `APP_ENV` is `vcluster-2`

    Repeat the same process for the second cluster, and you should see the same results. This means your Istio installation is working properly!

    Conclusion

    In this tutorial, you learned how to install and configure Istio for multicluster management in Kubernetes on EKS by deploying a sample application.

    Harness the power of Istio and vCluster to enhance your Kubernetes setup with efficient resource utilization, robust isolation, advanced traffic control, and heightened security measures. Explore the benefits of multitenancy in your environment by visiting the vCluster website to see how it can enhance your Kubernetes environment management.

    Sign up for our newsletter

    Be the first to know about new features, announcements and industry insights.