Leveraging Namespaces for Cost Optimization with Kubernetes

Kubernetes is a powerful container orchestration system that's widely used in the industry today. It has many features that make it attractive to organizations, including its ability to automatically scale containerized workloads and automate deployments. However, the ease of deploying and scaling cloud applications can lead to skyrocketing expenses if not managed correctly. So, cost optimization is an important consideration when it comes to running a Kubernetes cluster.

You can manage the costs associated with a Kubernetes cluster in several ways, for example, by using lower-cost hardware for nodes, cheaper storage options, or a lower-cost networking solution. However, these cost-saving measures inevitably affect the performance of the Kubernetes cluster. So, before downgrading your infrastructure, it's worth exploring a different alternative. Leveraging namespaces' ability to organize and manage your resources in Kubernetes is one option that can help your organization save costs.

In this article, you'll learn about the following:

Kubernetes Namespaces: What They Are and Why They're Useful for Cost Optimization

You can think of namespaces as a way to divide a Kubernetes cluster into multiple virtual clusters, each with its own set of resources. This allows you to use the same cluster for multiple teams, such as development, testing, quality assurance, or staging.

Kubernetes namespaces are implemented as a set of labels on objects in the cluster. When you create a namespace, you specify a name that identifies it and a set of labels to select the objects that belong to it.

You can use namespaces to control access to the cluster. For example, you can allow developers to access the development namespace but not the production namespace. This can be done by creating a role that has access to the development namespace and adding the developers to that role.

You can also use namespaces to control the resources that are available to the applications that run on them. This is done through resource quotas and limit ranges, two objects discussed later in this article. Setting such resource limits is invaluable in terms of cost optimization because it prevents resource wastage and thus saves money. Moreover, with proper monitoring, inactive or underused namespaces could be detected and shut down if necessary to save even more resources.

In short, you can use Kubernetes namespaces to set resource requests and limits to ensure your Kubernetes clusters have enough resources for optimal performance. This will minimize over-provisioning or under-provisioning of your applications.

Identifying Namespace Resource Usage

Before you can rightsize your applications, you must first identify namespace resource usage.

In this section, you'll learn how to inspect Kubernetes namespaces using the kubectl command line tool. Before proceeding, you'll need the following:

Inspecting Namespaces Resources Using kubectl

Start by creating a namespace called ns1:

kubectl create namespace ns1
namespace/ns1 created

Next, navigate to the root directory of the repository you just cloned and deploy the app1 application in the ns1 namespace, as shown below:

kubectl apply -f app1.yaml -n ns1
deployment.apps/app created
service/app created

app1 is a simple php-apache server based on the registry.k8s.io/hpa-example image:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app1
  labels:
    app: app1
spec:
  replicas: 5
  selector:
    matchLabels:
      app: app1
  template:
    metadata:
      name: app1
      labels:
        app: app1
    spec:
      containers:
      - name: app1
        image: registry.k8s.io/hpa-example
        ports:
        - containerPort: 80
        resources:
          limits:
            cpu: 500m
          requests:
            cpu: 200m
            memory: 8Mi---apiVersion: v1
kind: Service
metadata:
  name: app1
  labels:
    app: app1
spec:
  ports:
  - port: 80
  selector:
    app: app1

As you can see, it deploys five replicas of the application, which listens on port 80 through a service called app1.

Now, deploy the app2 application in the ns1 namespace:

kubectl apply -f app2.yaml -n ns1
deployment.apps/idle-app created

app2 is a dummy app that launches a BusyBox-based application that waits forever:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app2
  template:
    metadata:
      labels:
        app: app2
    spec:
      containers:
        - name: busybox
          image: busybox
          command:
            - /bin/sh
            - -c
            - "while true; do sleep 30; done"

You can now use the command kubectl get all to check all the resources that the ns1 namespace uses, as shown below:

kubectl get all -n ns1
NAME                        READY   STATUS    RESTARTS   AGE
pod/app1-785668c957-95kmv   1/1     Running   0          9s
pod/app1-785668c957-bnlvz   1/1     Running   0          9s
pod/app1-785668c957-d6mxt   1/1     Running   0          9s
pod/app1-785668c957-gbfvv   1/1     Running   0          9s
pod/app1-785668c957-pgrjv   1/1     Running   0          9s
pod/app2-77bd8884d6-tmplz   1/1     Running   0          5s

NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/app1   ClusterIP   10.245.27.210   <none>        80/TCP    9s

NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/app1   5/5     5            5           9s
deployment.apps/app2   1/1     1            1           9s

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/app1-785668c957   5         5         5       9s
replicaset.apps/app2-77bd8884d6   1         1         1       8s

Since you have Metrics Server installed, you can also use the top pods command to check the resource consumption of pods in the ns1 namespace, as shown below:

kubectl top pods -n ns1
NAME                    CPU(cores)   MEMORY(bytes)   
app1-785668c957-95kmv   1m           8Mi             
app1-785668c957-bnlvz   1m           8Mi             
app1-785668c957-d6mxt   1m           8Mi             
app1-785668c957-gbfvv   1m           8Mi             
app1-785668c957-pgrjv   1m           8Mi
app2-77bd8884d6-tmplz   1m           0Mi        

As you can see, by using the kubectl command line tool, you can take a quick look at the activity within the namespace, list the resources used, and get an idea of the pods' CPU cores and memory spending. Additionally, you can use the command kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace> to get an idea of how often the resources in the namespace are used:

kubectl api-resources --verbs=list --namespaced -o name \
  | xargs -n 1 kubectl get --show-kind --ignore-not-found -n ns1
NAME                         DATA   AGE
configmap/kube-root-ca.crt   1      22h
NAME             ENDPOINTS                                                   AGE
endpoints/app1   10.244.0.11:80,10.244.0.110:80,10.244.0.19:80 + 2 more...   63m
...output omitted...

41m         Normal    ScalingReplicaSet   deployment/app2              Scaled up replica set app2-774c558d94 to 1
NAME                        READY   STATUS    RESTARTS   AGE
pod/app1-788dc7b9bc-2lmc4   1/1     Running   0          63m
pod/app1-788dc7b9bc-6qzl9   1/1     Running   0          63m
pod/app1-788dc7b9bc-c2jwn   1/1     Running   0          63m
pod/app1-788dc7b9bc-pf4ds   1/1     Running   0          63m
pod/app1-788dc7b9bc-wl7xp   1/1     Running   0          63m
pod/app2-774c558d94-pt978   1/1     Running   0          41m
NAME                         TYPE                                  DATA   AGE
secret/default-token-2htgh   kubernetes.io/service-account-token   3      22h
NAME                     SECRETS   AGE
serviceaccount/default   1         22h
NAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/app1   ClusterIP   10.245.5.183   <none>        80/TCP    64m
NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/app1   5/5     5            5           64m
deployment.apps/app2   1/1     1            1           42m
NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/app1-788dc7b9bc   5         5         5       64m
replicaset.apps/app2-774c558d94   1         1         1       42m
NAME                                             ENDPOINT ID   IDENTITY ID   INGRESS ENFORCEMENT   EGRESS ENFORCEMENT   VISIBILITY POLICY   ENDPOINT STATE   IPV4           IPV6
ciliumendpoint.cilium.io/app1-788dc7b9bc-2lmc4   2723          22306                                                                        ...output omitted...
                                                              ready            10.244.0.87    
NAME                                        ADDRESSTYPE   PORTS   ENDPOINTS                                          AGE
endpointslice.discovery.k8s.io/app1-kbvj9   IPv4          80      10.244.0.110,10.244.0.11,10.244.0.29 + 2 more...   64m
LAST SEEN   TYPE      REASON              OBJECT                       MESSAGE
42m         Normal    Scheduled           pod/app2-774c558d94-pt978    Successfully assigned ...output omitted...

46m         Warning   BackOff             pod/app2-85dcc749c7-dmm2n    Back-off restarting failed container
54m         Normal    Pulled              pod/app2-85dcc749c7-dmm2n    Successfully pulled image "busybox" in 621.668342ms
52m         Normal    Pulled              pod/app2-85dcc749c7-dmm2n    Successfully pulled image "busybox" in 200.910627ms
50m         Normal    Pulled              pod/app2-85dcc749c7-dmm2n    Successfully pulled image "busybox" in 273.989882ms
56m         Normal    SuccessfulCreate    replicaset/app2-85dcc749c7   Created pod: app2-85dcc749c7-dmm2n
56m         Normal    ScalingReplicaSet   deployment/app2              Scaled up replica set app2-85dcc749c7 to 1
42m         Normal    ScalingReplicaSet   deployment/app2              Scaled up replica set app2-774c558d94 to 1
NAME                                              CPU      MEMORY   WINDOW
podmetrics.metrics.k8s.io/app1-788dc7b9bc-2lmc4   55271n   8952Ki   10.279s
podmetrics.metrics.k8s.io/app1-788dc7b9bc-6qzl9   47321n   8956Ki   16.436s
podmetrics.metrics.k8s.io/app1-788dc7b9bc-c2jwn   53688n   8972Ki   12.29s
podmetrics.metrics.k8s.io/app1-788dc7b9bc-pf4ds   57384n   9016Ki   19.875s
podmetrics.metrics.k8s.io/app1-788dc7b9bc-wl7xp   57195n   8980Ki   18.183s
podmetrics.metrics.k8s.io/app2-774c558d94-pt978   0        316Ki    16.729s

This command lists the resources in use as well as the activity time of each. It can also help detect some status messages like Back-off restarting failed container, which could indicate problems that need to be addressed. Checking the endpoint activity messages is also useful for inferring when a namespace or workload has been idle for a long time, thus identifying resources or namespaces that are no longer in use and that you can delete.

That said, other situations can also lead to wasted resources. Let's go back to the output of kubectl top pods -n ns1:

kubectl top pods -n ns1
NAME                    CPU(cores)   MEMORY(bytes)   
app1-788dc7b9bc-2lmc4   1m           8Mi             
app1-788dc7b9bc-6qzl9   1m           8Mi             
app1-788dc7b9bc-c2jwn   1m           8Mi             
app1-788dc7b9bc-pf4ds   1m           8Mi             
app1-788dc7b9bc-wl7xp   1m           8Mi             
app2-774c558d94-5mk8m   0m           0Mi   

Imagine if app2 was a new feature test that someone forgot to remove. This may not seem like much of a problem, as its CPU and memory consumption are negligible; however, left unattended, pods like this could start stacking up uncontrollably and hurt the control-plane scheduling performance. The same issue applies to app1; it consumes almost no CPU, but since it has no set memory limits, it could quickly consume resources if it starts scaling.

Fortunately, you can implement resource quotas and limit ranges in your namespaces to prevent these and other potentially costly situations.

Resource Quotas and Limit Ranges

This section explains how to use two Kubernetes objects, ResourceQuota and LimitRange, to minimize the previously mentioned negative effects of pods that have low resource utilization but the potential to fill your clusters with requests and resources that are not used by the namespace.

According to the documentation, the ResourceQuota object "provides constraints that limit aggregate resource consumption per namespace," while the LimitRange object provides "a policy to constrain the resource allocations (limits and requests) that you can specify for each applicable object kind (such as pod or PersistentVolumeClaim) in a namespace."

In other words, using these two objects, you can restrict resources both at the namespace level and at the pod and container level. To elaborate:

The best way to understand these concepts is to put them into practice.

Because both ResourceQuota and LimitRange only affect pods created after they're deployed, first delete the applications to clean up the cluster:

kubectl delete -f app1.yaml -n ns1 && kubectl delete -f app2.yaml -n ns1
deployment.apps "app1" deleted
service "app1" deleted
deployment.apps "app2" deleted

Next, create the restrictive-resource-limits policy by deploying a LimitRange resource:

kubectl apply -f restrictive-limitrange.yaml -n ns1
limitrange/restrictive-resource-limits created

The command above uses the following code:

apiVersion: "v1"
kind: "LimitRange"
metadata:
  name: "restrictive-resource-limits" 
spec:
  limits:
    -
      type: "Container"
      max:
        memory: "20Mi"
        cpu: "1" 
      min:
        memory: "10Mi"
        cpu: "1m"

As you can see, limits are set at the container level for the maximum and minimum CPU and memory usage. You can use kubectl describe to review this policy in the console:

kubectl describe limitrange restrictive-resource-limits -n ns1
Name:       restrictive-resource-limits
Namespace:  ns1
Type        Resource  Min  Max   Default Request  Default Limit  Max Limit/Request Ratio------------------
Container   cpu       1m   1     1                1              -
Container   memory    10Mi  20Mi  20Mi             20Mi           -

Now try to deploy app1 again:

kubectl apply -f app1.yaml -n ns1
deployment.apps/app1 created
service/app1 created

Then, check deployments in the ns1 namespace:

kubectl get deployment -n ns1
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
app1   0/5     0            0           1m

The policy implemented by restrictive-resource-limits prevented the pods from being created. This is because the policy requires a minimum of 10 Mi of memory per container, but app1 only requests 8 Mi. Although this is just an example, it shows how you can avoid cluttering up a namespace with tiny pods and containers.

Let's review how limit ranges and resource quotas can complement each other to achieve resource management at different levels. Before continuing, delete all resources again:

kubectl delete -f restrictive-limitrange.yaml -n ns1 && kubectl delete -f app1.yaml -n ns1

Next, deploy the permissive-limitrange.yaml and namespace-resource-quota.yaml resources:

kubectl apply -f permissive-limitrange.yaml -n ns1
kubectl apply -f namespace-resource-quota.yaml -n ns1
limitrange/permissive-resource-limits created
resourcequota/namespace-limits created

The new resource management policies should look as follows:

kubectl describe limitrange permissive-resource-limits -n ns1
kubectl describe resourcequota namespace-limits -n ns1
Name:       permissive-resource-limits
Namespace:  ns1
Type        Resource  Min  Max   Default Request  Default Limit  Max Limit/Request Ratio------------------
Container   memory    6Mi  20Mi  20Mi             20Mi           -
Container   cpu       1m   1     1                1              -
Name:            namespace-limits
Namespace:       ns1
Resource         Used  Hard-------------
limits.cpu       0     2
limits.memory    0     2Gi
pods             0     5
requests.cpu     0     1
requests.memory  0     1Gi

According to permissive-resource-limits, there should be no problem deploying app1 this time:

kubectl apply -f app1.yaml -n ns1
deployment.apps/app1 created
service/app1 created

Check the resources in the ns1 namespace:

kubectl get all -n ns1
NAME                        READY   STATUS    RESTARTS   AGE
pod/app1-5579c6cdb4-5pb2h   1/1     Running   0          11m
pod/app1-5579c6cdb4-cqtrh   1/1     Running   0          11m
pod/app1-5579c6cdb4-fgm8q   1/1     Running   0          11m
pod/app1-5579c6cdb4-s97zk   1/1     Running   0          11m

NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/app1   ClusterIP   10.245.197.52   <none>        80/TCP    11m

NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/app1   4/5     4            4           11m

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/app1-5579c6cdb4   5         4         4       11m

You may be wondering why only four out of five pods were deployed. The answer lies in the CPU limits of the resource quota. Each container requests 500 CPU millicores, and the namespace limit is two cores. To put it another way, this policy only allows you to create four pods totaling 2000 millicores (two cores).

The same principle used to prevent over-provisioning of a namespace can be used to prevent under-provisioning.

Scope of LimitRange and ResourceQuota in Resource Management

Up to this point, you've seen how to use segmentation in namespaces and LimitRange and ResourceQuota policies to optimize costs. This section addresses the other side of the coin—the limitations and pros and cons of such policies.

Limitations of LimitRange and ResourceQuota

Kubernetes documentation is very clear when it comes to the scope of LimitRange and ResourceQuota.

LimitRange policies are intended to set bounds on resources such as:

Additionally, according to the documentation, you can "enforce a ratio between request and limit for a resource in a namespace."

A ResourceQuota, on the other hand, also allows you to set minimum and maximum compute resource values, but in the context of a namespace. Moreover, it also allows you to enforce other aspects at the namespace level, such as:

As you can see, LimitRange and ResourceQuota policies help keep a large number of resources under control. That said, it's wise to explore the limitations of using such resource usage policies.

LimitRange and ResourceQuota: Pros and Cons

As powerful and flexible as LimitRange and ResourceQuota policies are, they are not without certain limitations. The following is a summary of the pros and cons of these objects from the perspective of cost optimization:

Pros

Cons

In light of these limitations, it's worth asking if there is a tool that addresses these limitations by adding new functionality to Kubernetes to optimize resource usage.

Loft's Cost Optimization Solution

Loft is a state-of-the-art managed self-service platform that offers solutions for Kubernetes in areas such as access control, multi-tenancy, and cluster management. Additionally, Loft provides advanced cost optimization features such as sleep mode and auto-delete:

Needless to say, both sleep mode and auto-delete are fully configurable, giving DevOps teams full control over when a namespace is put to sleep or deleted.

Conclusion

Kubernetes allows you to use LimitRange and ResourceQuota policies to promote efficient use of resources in namespaces and thus save costs. That said, estimating resource requirements in a production environment is challenging, which is why it's a good idea to combine the flexibility provided by namespaces and resource usage policies with state-of-the-art cost optimization solutions like Loft.

Features like sleep mode and auto-delete help keep your clusters clean, which can save your organization up to 70 percent on costs. To learn more, explore our solutions.