Table of Contents
Kubernetes resources play a crucial role in managing and optimizing the performance of containerized applications. They are essential for ensuring your applications run smoothly and efficiently in your cluster. You can control how much of each resource a container needs, ensuring appropriate allocation and utilization, by defining resource requests and limits for CPU, memory, and other resources. The Kubernetes scheduler intelligently places pods on nodes, taking into account the specified resource requests and the available capacity of the nodes, thereby preventing resource starvation and guaranteeing the applications have the resources they need to function optimally.
You can add or change Kubernetes resources to fine-tune application performance and resource usage based on evolving requirements. Adjusting resource requests and limits can ensure that your applications remain responsive during peak-demand periods while minimizing resource wastage during low-demand periods. This dynamic resource management capability is fundamental to the scalable, resilient, and efficient operation of containerized applications in your Kubernetes environment.
This article is the third part of the "Platform Engineering Workflows" series and focuses on how to add and change resources in Kubernetes. In the first two parts of this series, you learned how to add environment variables and change configurations, as well as how to manage services and dependencies in Kubernetes.
In this article, you'll learn about the various ways to add, update, and manage Kubernetes resources effectively, along with best practices to ensure efficient resource usage.
Platform Engineering + Kubernetes Series
- Platform Engineering on Kubernetes for Accelerating Development Workflows
- Adding Environment Variables and Changing Configurations in Kubernetes
- Adding Services and Dependencies in Kubernetes
- Adding and Changing Kubernetes Resources
- Enforcing RBAC in Kubernetes
- Spinning up a New Kubernetes Environment
Adding and Updating Kubernetes Resources
This article highlights several ways to add and update resources in Kubernetes, each with its own advantages and use cases.
Before checking out some of these methods, you must have a running Kubernetes cluster. You can use minikube to create a local Kubernetes cluster on your computer.
One-Off Operations via Commands Like kubectl create
Using kubectl create to create a pod with resource limits for its container involves specifying the necessary parameters in the command.
Here's an example of creating a simple pod with a single container and resource limits:
kubectl create -f- <<EOF
apiVersion: v1
kind: Pod
metadata:
name: frontend-app
spec:
containers:
- name: frontend-app
image: nginx:1.22
resources:
limits:
cpu: "400m"
memory: "256Mi"
requests:
cpu: "100m"
memory: "128Mi"
EOF
The above command creates a pod named frontend-app using the nginx:1.22 image, and the following resource limits and requests are defined for the pod:
It's important to note that the above method is not the recommended way to manage resources in your cluster, especially when you have complex applications. For better version control and change tracking, use declarative resource management with YAML files and kubectl apply, as described in the next section.
Declarative Resource Management Using kubectl apply
Instead of applying one-off operations to manage resources in your cluster, use the declarative approach for better version control, automation, and collaboration, among many other benefits. To make the previous example declarative, create a file titled frontend-app.yaml and add the following YAML configuration to the file:
apiVersion: v1
kind: Pod
metadata:
name: frontend-app
spec:
containers:
- name: frontend-app
image: nginx:1.22
resources:
limits:
cpu: "400m"
memory: "256Mi"
requests:
cpu: "100m"
memory: "128Mi"
Then, run the following command:
kubectl apply -f frontend-app.yaml
While it's possible to run the above command using the kubectl create command, kubectl apply is more suitable because of its idempotency, which means you can run the same command multiple times and end up with the same result. This is not the case with kubectl create, which fails if a resource with the same name already exists. This idempotency also means kubectl apply is more suitable for automations and scripts.
Using the Kubernetes API
Using the Kubernetes API to manage resources can enable you to automate tasks in your cluster, such as creating and updating resources allocated to specific workloads in your cluster. Furthermore, the Kubernetes API allows you to interact with the cluster using your preferred programming language, improving your development experience by providing familiar tools and libraries.
Using the Kubernetes API to create pods and manage containers' resource limits involves making an HTTP request to the API server with the necessary pod configuration.
The following example uses curl to create a pod and also adds resource limits to the pod's container.
First, you need to get a token from your cluster, which you can do by following this tutorial. Once you have the token, assign it to the TOKEN variable in your terminal:
TOKEN=eyJhbGciOiJSUzI1NiIsImtp...
Next, get the Kubernetes API server address of your cluster using the following command:
APISERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
Now, create a JSON file named frontend-pod.json and paste in the following code:
{
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"name": "frontend-pod"
},
"spec": {
"containers": [
{
"name": "frontend-container",
"image": "nginx:1.22",
"resources": {
"limits": {
"cpu": "400m",
"memory": "256Mi"
},
"requests": {
"cpu": "100m",
"memory": "128Mi"
}
}
}
]
}
}
Make an HTTP POST request to the Kubernetes API server using curl:
curl -k -X POST -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" -d @frontend-pod.json $APISERVER/api/v1/namespaces/default/pods
The above command creates a pod with the specified resource limits and requests as defined in the frontend-pod.json file. Using the Kubernetes API gives you more programmatic control and integration with custom applications or scripts.
However, for most use cases, it's more convenient and less complex to use kubectl commands than the Kubernetes API.
Setting Up Third-Party Predefined Resources Using Tools Like Helm
Helm is a package manager for Kubernetes that simplifies the deployment and management of applications on your cluster. Helm uses charts, which are prepackaged Kubernetes resources, to automate the deployment, configuration, and management of applications. You can use Helm to manage the resources of your workloads in your cluster.
This section goes over an example of how to create workloads and manage workload resources using Helm.
If you don't have Helm installed on your local machine, you can follow the installation instructions on the official Helm website.
Once you have Helm installed, run the following command to create a directory called api-chart with the basic structure and files for a Helm chart:
helm create api-chart
Move into the templates directory of the chart:
cd api-chart/templates
Then, create a new YAML file called api-pod.yaml and paste in the following code to define a pod and its resource limits:
apiVersion: v1
kind: Pod
metadata:
name: {{ .Release.Name }}-api-pod
spec:
containers:
- name: api-container
image: nginx:1.22
resources:
limits:
cpu: {{ .Values.resources.limits.cpu }}
memory: {{ .Values.resources.limits.memory }}
requests:
cpu: {{ .Values.resources.requests.cpu }}
memory: {{ .Values.resources.requests.memory }}
The above code uses Helm templating to set resource limits and requests based on the values provided in the values.yaml file found in the api-chart directory.
Now, open the values.yaml file and update the following values for the resource limits and requests:
resources:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 500m
memory: 500Mi
Deploy the chart by running the following command from the parent directory of api-chart:
helm install api-release ./api-chart
The above command creates a new release called api-release using the chart in the api-chart directory. Helm will deploy the pod with the specified resource limits and requests.
To update the pod resources to a new set of values, all you need to do is update the values.yaml file again, then run the following command:
helm upgrade api-release ./api-chart
Helm simplifies the management of Kubernetes resources, such as pods with defined resource limits and requests, by providing a version-controlled and templated approach. This makes it easier to handle and update intricate deployments while also guaranteeing consistency across different environments.
Automating Resource Creation and Management Using Operators
Kubernetes operators are software extensions that use custom resources to manage applications and their components. Operators allow you to automate routine tasks like deploying and configuring applications, scaling based on demand, and handling failover and recovery. You can use operators to automate the process of updating your workload resources under certain conditions.
For instance, you can use the Kubernetes Vertical Pod Autoscaler operator to automatically adjust CPU and memory reservations for your pods. This ensures that your pods are allocated the right amount of resources they need to operate efficiently. The operator does this by monitoring the pod's resource usage and comparing it to the existing limit. If the operator detects any inconsistency, it can automatically adjust the resource limit and restart the pod if required. You can refer to this concise tutorial on how to utilize the Kubernetes Vertical Pod Autoscaler operator, and more information can be found in the official documentation.
Additionally, you can also create your own operator that manages and automates your workload metrics based on certain conditions in your cluster using this tutorial.
Best Practices for Managing Resources in Kubernetes
When managing resources in Kubernetes, there are some best practices you should follow to efficiently manage workload resources in your cluster.
Define Appropriate Resource Limits and Request Values for Each New Resource
You should specify each container's resource requirements in a pod, as this information helps the Kubernetes scheduler decide which node to assign it to. Requests guarantee the minimum resources a container needs, allowing Kubernetes to schedule it only on nodes that can provide those resources. Limits, on the other hand, specify the maximum resources a container can consume, preventing excessive resource usage. The following example sets some request and limit requirements:
apiVersion: v1
kind: Pod
metadata:
name: sample-pod
spec:
containers:
- name: sample-container
image: nginx:1.22
resources:
requests:
memory: "128Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "1000m"
When you specify the appropriate requests and limits, you ensure that your pods are running on nodes with enough resources, which can help prevent resource starvation and improve the stability of your workloads.
Use Resource Quotas across Namespaces
Resource quotas provide constraints that limit total resource consumption in the configured namespace. You can use resource quotas to restrict the number of objects created in a namespace based on their specific types as well as the total amount of compute resources that may be consumed by resources in that namespace.
For instance, you can set a quota to limit the total amount of memory that can be used in a namespace:
apiVersion: v1
kind: ResourceQuota
metadata:
name: mem-cpu-limit
spec:
hard:
requests.cpu: "1"
requests.memory: 2Gi
limits.cpu: "2"
limits.memory: 4Gi
Monitor Resource Usage
Monitoring your Kubernetes cluster is crucial for gaining insights into its performance and resource utilization. Prometheus and Grafana are powerful tools that can be used for this purpose.
To install Prometheus and Grafana in your cluster, you can follow these tutorials:
Monitoring is really helpful for detecting issues early and optimizing resource utilization based on actual usage data. You can detect issues such as high memory and CPU consumption, slow API response times, and application containers with a high restart rate.
Use ConfigMaps, Secrets, and Persistent Volumes for Configuration, Sensitive Data, and Data Persistence
ConfigMaps and secrets can decouple configuration artifacts and sensitive data from pod specifications. This allows you to maintain the same pod specification across different environments and versions, simplifying updates and scaling operations.
For example, you might want to apply different CPU and memory requests or limits based on the environment (development, staging, or production). You can define these in a ConfigMap and use the values in your pod specification:
apiVersion: v1
kind: ConfigMap
metadata:
name: resource-config
data:
cpu: "500m"
memory: "128Mi"
In your pod specification, you can reference the ConfigMap:
apiVersion: v1
kind: Pod
metadata:
name: frontend
spec:
containers:
- name: app
image: nginx:1.22
resources:
requests:
memory: $(MEMORY_REQUEST)
cpu: $(CPU_REQUEST)
limits:
memory: $(MEMORY_LIMIT)
cpu: $(CPU_LIMIT)
envFrom:
- configMapRef:
name: resource-config
Similarly, secrets can store sensitive information like passwords or keys. These can also be referenced in your pod specifications.
You should use persistent volumes and persistent volume claims for data that should persist across pod restarts or deployments. These resources will ensure that your data remains available and intact, even if your pods are rescheduled or redeployed.
Assess the Impact of Adding or Changing Resources on a Production Cluster
It's important that you always assess the impact of adding or updating resources on the underlying nodes before introducing any changes to your live production cluster. This involves understanding how the new resources will affect the cluster's existing load, distribution, and overall performance.
For example, if you want to add a new pod that has significant resource requirements, it's important to check if your current nodes have the capacity to handle this additional load. Similarly, if you're updating resource limits, it's necessary to evaluate if these changes could lead to resource contention or performance degradation.
You can run the following command to know the total CPU and memory requests and limits in your cluster:
kubectl get nodes --no-headers | awk '{print $1}' | xargs -I {} sh -c 'echo {}; kubectl describe node {} | grep Allocated -A 5 | grep -ve Event -ve Allocated -ve percent -ve -- ; echo'
Conclusion
This article explained the process and significance of managing resources in Kubernetes. You learned about the importance of appropriately setting resource requests and limits for containers within pods and how these values influence scheduling and the stability of the system. They help prevent scenarios where a container can monopolize resources, ensuring fair usage across all running containers.
You also learned about various methods for creating and updating resources in Kubernetes, such as kubectl commands, declarative resource management with the kubectl apply command, the Kubernetes API, Kubernetes operators, and third-party tools like Helm.
Finally, the article provided some best practices for managing resources in Kubernetes. These practices, including defining appropriate resource values, using resource quotas, setting up monitoring, using ConfigMaps and secrets, and assessing the impact of changes before applying them to production, are crucial for maintaining a healthy, scalable, and efficient Kubernetes environment.
