Kubernetes is a powerful container orchestration system that’s widely used in the industry today. It has many features that make it attractive to organizations, including its ability to automatically scale containerized workloads and automate deployments. However, the ease of deploying and scaling cloud applications can lead to skyrocketing expenses if not managed correctly. So, cost optimization is an important consideration when it comes to running a Kubernetes cluster.
You can manage the costs associated with a Kubernetes cluster in several ways, for example, by using lower-cost hardware for nodes, cheaper storage options, or a lower-cost networking solution. However, these cost-saving measures inevitably affect the performance of the Kubernetes cluster. So, before downgrading your infrastructure, it’s worth exploring a different alternative. Leveraging namespaces' ability to organize and manage your resources in Kubernetes is one option that can help your organization save costs.
In this article, you’ll learn about the following:
- Kubernetes namespaces and their role from a cost optimization perspective
- Identifying resource usage in namespaces
- Resource quotas and limit ranges
- Setting up resource quotas and limit ranges in Kubernetes
- Benefits of x-as-a-service (XaaS) solutions with built-in cost optimization features
#Kubernetes Namespaces: What They Are and Why They’re Useful for Cost Optimization
You can think of namespaces as a way to divide a Kubernetes cluster into multiple virtual clusters, each with its own set of resources. This allows you to use the same cluster for multiple teams, such as development, testing, quality assurance, or staging.
Kubernetes namespaces are implemented as a set of labels on objects in the cluster. When you create a namespace, you specify a name that identifies it and a set of labels to select the objects that belong to it.
You can use namespaces to control access to the cluster. For example, you can allow developers to access the development namespace but not the production namespace. This can be done by creating a role that has access to the development namespace and adding the developers to that role.
You can also use namespaces to control the resources that are available to the applications that run on them. This is done through resource quotas and limit ranges, two objects discussed later in this article. Setting such resource limits is invaluable in terms of cost optimization because it prevents resource wastage and thus saves money. Moreover, with proper monitoring, inactive or underused namespaces could be detected and shut down if necessary to save even more resources.
In short, you can use Kubernetes namespaces to set resource requests and limits to ensure your Kubernetes clusters have enough resources for optimal performance. This will minimize over-provisioning or under-provisioning of your applications.
#Identifying Namespace Resource Usage
Before you can rightsize your applications, you must first identify namespace resource usage.
In this section, you’ll learn how to inspect Kubernetes namespaces using the
kubectl command line tool. Before proceeding, you’ll need the following:
kubectlinstalled and configured on your local machine.
- Access to a Kubernetes cluster with Metrics Server installed. The Kubernetes Metrics Server is indispensable for collecting metrics and using the
- This repository cloned to a suitable location on your local machine.
#Inspecting Namespaces Resources Using kubectl
Start by creating a namespace called
kubectl create namespace ns1 namespace/ns1 created
Next, navigate to the root directory of the repository you just cloned and deploy the
app1 application in the
ns1 namespace, as shown below:
kubectl apply -f app1.yaml -n ns1 deployment.apps/app created service/app created
app1 is a simple php-apache server based on the
apiVersion: apps/v1 kind: Deployment metadata: name: app1 labels: app: app1 spec: replicas: 5 selector: matchLabels: app: app1 template: metadata: name: app1 labels: app: app1 spec: containers: - name: app1 image: registry.k8s.io/hpa-example ports: - containerPort: 80 resources: limits: cpu: 500m requests: cpu: 200m memory: 8Mi --- apiVersion: v1 kind: Service metadata: name: app1 labels: app: app1 spec: ports: - port: 80 selector: app: app1
As you can see, it deploys five replicas of the application, which listens on port 80 through a service called
Now, deploy the
app2 application in the
kubectl apply -f app2.yaml -n ns1 deployment.apps/idle-app created
app2 is a dummy app that launches a BusyBox-based application that waits forever:
apiVersion: apps/v1 kind: Deployment metadata: name: app2 spec: replicas: 1 selector: matchLabels: app: app2 template: metadata: labels: app: app2 spec: containers: - name: busybox image: busybox command: - /bin/sh - -c - "while true; do sleep 30; done"
You can now use the command
kubectl get all to check all the resources that the
ns1 namespace uses, as shown below:
kubectl get all -n ns1 NAME READY STATUS RESTARTS AGE pod/app1-785668c957-95kmv 1/1 Running 0 9s pod/app1-785668c957-bnlvz 1/1 Running 0 9s pod/app1-785668c957-d6mxt 1/1 Running 0 9s pod/app1-785668c957-gbfvv 1/1 Running 0 9s pod/app1-785668c957-pgrjv 1/1 Running 0 9s pod/app2-77bd8884d6-tmplz 1/1 Running 0 5s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/app1 ClusterIP 10.245.27.210 <none> 80/TCP 9s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/app1 5/5 5 5 9s deployment.apps/app2 1/1 1 1 9s NAME DESIRED CURRENT READY AGE replicaset.apps/app1-785668c957 5 5 5 9s replicaset.apps/app2-77bd8884d6 1 1 1 8s
Since you have Metrics Server installed, you can also use the
top pods command to check the resource consumption of pods in the
ns1 namespace, as shown below:
kubectl top pods -n ns1 NAME CPU(cores) MEMORY(bytes) app1-785668c957-95kmv 1m 8Mi app1-785668c957-bnlvz 1m 8Mi app1-785668c957-d6mxt 1m 8Mi app1-785668c957-gbfvv 1m 8Mi app1-785668c957-pgrjv 1m 8Mi app2-77bd8884d6-tmplz 1m 0Mi
As you can see, by using the
kubectl command line tool, you can take a quick look at the activity within the namespace, list the resources used, and get an idea of the pods' CPU cores and memory spending. Additionally, you can use the command
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace> to get an idea of how often the resources in the namespace are used:
kubectl api-resources --verbs=list --namespaced -o name \ | xargs -n 1 kubectl get --show-kind --ignore-not-found -n ns1 NAME DATA AGE configmap/kube-root-ca.crt 1 22h NAME ENDPOINTS AGE endpoints/app1 10.244.0.11:80,10.244.0.110:80,10.244.0.19:80 + 2 more... 63m ...output omitted... 41m Normal ScalingReplicaSet deployment/app2 Scaled up replica set app2-774c558d94 to 1 NAME READY STATUS RESTARTS AGE pod/app1-788dc7b9bc-2lmc4 1/1 Running 0 63m pod/app1-788dc7b9bc-6qzl9 1/1 Running 0 63m pod/app1-788dc7b9bc-c2jwn 1/1 Running 0 63m pod/app1-788dc7b9bc-pf4ds 1/1 Running 0 63m pod/app1-788dc7b9bc-wl7xp 1/1 Running 0 63m pod/app2-774c558d94-pt978 1/1 Running 0 41m NAME TYPE DATA AGE secret/default-token-2htgh kubernetes.io/service-account-token 3 22h NAME SECRETS AGE serviceaccount/default 1 22h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/app1 ClusterIP 10.245.5.183 <none> 80/TCP 64m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/app1 5/5 5 5 64m deployment.apps/app2 1/1 1 1 42m NAME DESIRED CURRENT READY AGE replicaset.apps/app1-788dc7b9bc 5 5 5 64m replicaset.apps/app2-774c558d94 1 1 1 42m NAME ENDPOINT ID IDENTITY ID INGRESS ENFORCEMENT EGRESS ENFORCEMENT VISIBILITY POLICY ENDPOINT STATE IPV4 IPV6 ciliumendpoint.cilium.io/app1-788dc7b9bc-2lmc4 2723 22306 ...output omitted... ready 10.244.0.87 NAME ADDRESSTYPE PORTS ENDPOINTS AGE endpointslice.discovery.k8s.io/app1-kbvj9 IPv4 80 10.244.0.110,10.244.0.11,10.244.0.29 + 2 more... 64m LAST SEEN TYPE REASON OBJECT MESSAGE 42m Normal Scheduled pod/app2-774c558d94-pt978 Successfully assigned ...output omitted... 46m Warning BackOff pod/app2-85dcc749c7-dmm2n Back-off restarting failed container 54m Normal Pulled pod/app2-85dcc749c7-dmm2n Successfully pulled image "busybox" in 621.668342ms 52m Normal Pulled pod/app2-85dcc749c7-dmm2n Successfully pulled image "busybox" in 200.910627ms 50m Normal Pulled pod/app2-85dcc749c7-dmm2n Successfully pulled image "busybox" in 273.989882ms 56m Normal SuccessfulCreate replicaset/app2-85dcc749c7 Created pod: app2-85dcc749c7-dmm2n 56m Normal ScalingReplicaSet deployment/app2 Scaled up replica set app2-85dcc749c7 to 1 42m Normal ScalingReplicaSet deployment/app2 Scaled up replica set app2-774c558d94 to 1 NAME CPU MEMORY WINDOW podmetrics.metrics.k8s.io/app1-788dc7b9bc-2lmc4 55271n 8952Ki 10.279s podmetrics.metrics.k8s.io/app1-788dc7b9bc-6qzl9 47321n 8956Ki 16.436s podmetrics.metrics.k8s.io/app1-788dc7b9bc-c2jwn 53688n 8972Ki 12.29s podmetrics.metrics.k8s.io/app1-788dc7b9bc-pf4ds 57384n 9016Ki 19.875s podmetrics.metrics.k8s.io/app1-788dc7b9bc-wl7xp 57195n 8980Ki 18.183s podmetrics.metrics.k8s.io/app2-774c558d94-pt978 0 316Ki 16.729s
This command lists the resources in use as well as the activity time of each. It can also help detect some status messages like
Back-off restarting failed container, which could indicate problems that need to be addressed. Checking the endpoint activity messages is also useful for inferring when a namespace or workload has been idle for a long time, thus identifying resources or namespaces that are no longer in use and that you can delete.
That said, other situations can also lead to wasted resources. Let’s go back to the output of
kubectl top pods -n ns1:
kubectl top pods -n ns1 NAME CPU(cores) MEMORY(bytes) app1-788dc7b9bc-2lmc4 1m 8Mi app1-788dc7b9bc-6qzl9 1m 8Mi app1-788dc7b9bc-c2jwn 1m 8Mi app1-788dc7b9bc-pf4ds 1m 8Mi app1-788dc7b9bc-wl7xp 1m 8Mi app2-774c558d94-5mk8m 0m 0Mi
app2 was a new feature test that someone forgot to remove. This may not seem like much of a problem, as its CPU and memory consumption are negligible; however, left unattended, pods like this could start stacking up uncontrollably and hurt the control-plane scheduling performance. The same issue applies to
app1; it consumes almost no CPU, but since it has no set memory limits, it could quickly consume resources if it starts scaling.
Fortunately, you can implement resource quotas and limit ranges in your namespaces to prevent these and other potentially costly situations.
#Resource Quotas and Limit Ranges
This section explains how to use two Kubernetes objects,
LimitRange, to minimize the previously mentioned negative effects of pods that have low resource utilization but the potential to fill your clusters with requests and resources that are not used by the namespace.
According to the documentation, the
ResourceQuota object “provides constraints that limit aggregate resource consumption per namespace,” while the
LimitRange object provides “a policy to constrain the resource allocations (limits and requests) that you can specify for each applicable object kind (such as pod or PersistentVolumeClaim) in a namespace.”
In other words, using these two objects, you can restrict resources both at the namespace level and at the pod and container level. To elaborate:
ResourceQuotaallows you to limit the total resource consumption of a namespace. For example, you can create a namespace dedicated to testing and set CPU and memory limits to ensure users don’t overspend resources. Furthermore,
ResourceQuotaalso allows you to set limits on storage resources and limits on the total number of certain objects, such as ConfigMaps, cron jobs, secrets, services, and PersistentVolumeClaims.
LimitRangeallows you to set constraints at the pod and container level instead of at the namespace level. This ensures that an application does not consume all the resources allocated via
The best way to understand these concepts is to put them into practice.
LimitRange only affect pods created after they’re deployed, first delete the applications to clean up the cluster:
kubectl delete -f app1.yaml -n ns1 && kubectl delete -f app2.yaml -n ns1 deployment.apps "app1" deleted service "app1" deleted deployment.apps "app2" deleted
Next, create the
restrictive-resource-limits policy by deploying a
kubectl apply -f restrictive-limitrange.yaml -n ns1 limitrange/restrictive-resource-limits created
The command above uses the following code:
apiVersion: "v1" kind: "LimitRange" metadata: name: "restrictive-resource-limits" spec: limits: - type: "Container" max: memory: "20Mi" cpu: "1" min: memory: "10Mi" cpu: "1m"
As you can see, limits are set at the container level for the maximum and minimum CPU and memory usage. You can use
kubectl describe to review this policy in the console:
kubectl describe limitrange restrictive-resource-limits -n ns1 Name: restrictive-resource-limits Namespace: ns1 Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ---- -------- --- --- --------------- ------------- ----------------------- Container cpu 1m 1 1 1 - Container memory 10Mi 20Mi 20Mi 20Mi -
Now try to deploy
kubectl apply -f app1.yaml -n ns1 deployment.apps/app1 created service/app1 created
Then, check deployments in the
kubectl get deployment -n ns1 NAME READY UP-TO-DATE AVAILABLE AGE app1 0/5 0 0 1m
The policy implemented by
restrictive-resource-limits prevented the pods from being created. This is because the policy requires a minimum of 10 Mi of memory per container, but
app1 only requests 8 Mi. Although this is just an example, it shows how you can avoid cluttering up a namespace with tiny pods and containers.
Let’s review how limit ranges and resource quotas can complement each other to achieve resource management at different levels. Before continuing, delete all resources again:
kubectl delete -f restrictive-limitrange.yaml -n ns1 && kubectl delete -f app1.yaml -n ns1
Next, deploy the
kubectl apply -f permissive-limitrange.yaml -n ns1 kubectl apply -f namespace-resource-quota.yaml -n ns1 limitrange/permissive-resource-limits created resourcequota/namespace-limits created
The new resource management policies should look as follows:
kubectl describe limitrange permissive-resource-limits -n ns1 kubectl describe resourcequota namespace-limits -n ns1 Name: permissive-resource-limits Namespace: ns1 Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ---- -------- --- --- --------------- ------------- ----------------------- Container memory 6Mi 20Mi 20Mi 20Mi - Container cpu 1m 1 1 1 - Name: namespace-limits Namespace: ns1 Resource Used Hard -------- ---- ---- limits.cpu 0 2 limits.memory 0 2Gi pods 0 5 requests.cpu 0 1 requests.memory 0 1Gi
permissive-resource-limits, there should be no problem deploying
app1 this time:
kubectl apply -f app1.yaml -n ns1 deployment.apps/app1 created service/app1 created
Check the resources in the
kubectl get all -n ns1 NAME READY STATUS RESTARTS AGE pod/app1-5579c6cdb4-5pb2h 1/1 Running 0 11m pod/app1-5579c6cdb4-cqtrh 1/1 Running 0 11m pod/app1-5579c6cdb4-fgm8q 1/1 Running 0 11m pod/app1-5579c6cdb4-s97zk 1/1 Running 0 11m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/app1 ClusterIP 10.245.197.52 <none> 80/TCP 11m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/app1 4/5 4 4 11m NAME DESIRED CURRENT READY AGE replicaset.apps/app1-5579c6cdb4 5 4 4 11m
You may be wondering why only four out of five pods were deployed. The answer lies in the CPU limits of the resource quota. Each container requests 500 CPU millicores, and the namespace limit is two cores. To put it another way, this policy only allows you to create four pods totaling 2000 millicores (two cores).
The same principle used to prevent over-provisioning of a namespace can be used to prevent under-provisioning.
#Scope of LimitRange and ResourceQuota in Resource Management
Up to this point, you’ve seen how to use segmentation in namespaces and
ResourceQuota policies to optimize costs. This section addresses the other side of the coin—the limitations and pros and cons of such policies.
#Limitations of LimitRange and ResourceQuota
Kubernetes documentation is very clear when it comes to the scope of
LimitRange policies are intended to set bounds on resources such as:
- Containers and pods, where you can set minimum, maximum, and default request values for memory and CPU per namespace
- PersistentVolumeClaims, where you can set minimum and maximum storage request values per namespace
Additionally, according to the documentation, you can “enforce a ratio between request and limit for a resource in a namespace.”
ResourceQuota, on the other hand, also allows you to set minimum and maximum compute resource values, but in the context of a namespace. Moreover, it also allows you to enforce other aspects at the namespace level, such as:
- The total number of PersistentVolumeClaims that can exist in the namespace
- The total space to be used in the namespace for persistent volume claims and ephemeral storage requests
- The total number of pods, ConfigMaps, ReplicationControllers,
ResourceQuotaobjects, load balancers, secrets, deployments, and cron jobs that can exist in the namespace
As you can see,
ResourceQuota policies help keep a large number of resources under control. That said, it’s wise to explore the limitations of using such resource usage policies.
#LimitRange and ResourceQuota: Pros and Cons
As powerful and flexible as
ResourceQuota policies are, they are not without certain limitations. The following is a summary of the pros and cons of these objects from the perspective of cost optimization:
- You do not have to install any third-party solutions to enforce reasonable resource usage.
- If you define your policies wisely, you can minimize the incidence of issues like CPU starvation, pod eviction, or running out of memory or storage.
- Enforcing resource limits helps lower cluster operating costs.
- Kubernetes lacks built-in mechanisms to monitor resource usage. So, whether you like it or not, you will have to use third-party solutions at some point to help your team understand workload behavior and plan accordingly.
- Policies implemented using
ResourceQuotaare static. That is, you may have to fine-tune them from time to time.
ResourceQuotacannot help you avoid resource wastage in every situation. They won’t help with services and applications that comply with the policies at the time of their creation but become inactive after a while.
- Identifying inactive namespaces is a manual and time-consuming process.
In light of these limitations, it’s worth asking if there is a tool that addresses these limitations by adding new functionality to Kubernetes to optimize resource usage.
#Loft’s Cost Optimization Solution
Loft is a state-of-the-art managed self-service platform that offers solutions for Kubernetes in areas such as access control, multi-tenancy, and cluster management. Additionally, Loft provides advanced cost optimization features such as sleep mode and auto-delete:
- Sleep mode: This powerful feature monitors the activity of workloads within a namespace and automatically puts them to sleep after a certain period of inactivity. In other words, thanks to the sleep mode, only the namespaces that are in use remain active, and the rest are put to sleep. This is no doubt the definitive solution to the costs generated by idle resources.
- Auto-delete: This feature is the perfect complement to sleep mode. While sleep mode consists of scaling down to zero pods while the namespace is inactive, auto-delete goes a step further by permanently deleting namespaces that have not been active for a certain period of time. Auto-delete is especially useful for minimizing the waste of resources caused by demo environments and projects that have been sitting idle for too long.
Needless to say, both sleep mode and auto-delete are fully configurable, giving DevOps teams full control over when a namespace is put to sleep or deleted.
Kubernetes allows you to use
ResourceQuota policies to promote efficient use of resources in namespaces and thus save costs. That said, estimating resource requirements in a production environment is challenging, which is why it’s a good idea to combine the flexibility provided by namespaces and resource usage policies with state-of-the-art cost optimization solutions like Loft.
Features like sleep mode and auto-delete help keep your clusters clean, which can save your organization up to 70 percent on costs. To learn more, explore our solutions.