How to Back Up and Restore Kubernetes Clusters

Kubernetes makes building and managing containerized applications easy for organizations. Still, 46% of organizations that use it have lost revenue or customers due to security issues.

A great way to prevent these issues is to back up your Kubernetes or K8s cluster. Regular backups make it easy to restore data and prevent data loss.

This tutorial will teach you step-by-step how to back up and restore Kubernetes clusters and apps. You'll explore best practices to keep your data safe and recoverable. This will help you protect your Kubernetes backup clusters from unforeseen events.

Main Points

• With tools such as K10, data can be backed up at intervals without constantly having to be done manually.
• Because Kubernetes data is arranged in namespaces, K10 manages applications as namespaces to reduce backup processes.
• It is best to close all applications before starting the procedure to prevent any disruption when restoring the game.
• It is recommended that backups be stored in an Amazon S3 bucket or Microsoft Azure Storage space to avoid data loss due to mechanical failures on local hardware.

Kubernetes Backup 101: Defining a Backup Strategy and Tooling

Kubernetes has a resource object, VolumeSnapshot. It lets users and admins create volume snapshots of persistent volume claims. In theory, you could use VolumeSnapshot to back up your cluster. However, doing it manually is cumbersome and inefficient.

Popular backup tools like Velero, Stash, and Kasten K10 can help. They can help you define and implement a backup strategy that will meet your organization's requirements.

You can find which Kubernetes cluster and app resources to back up. Then, use any of these tools to schedule automatic snapshots. Furthermore, these tools make it easier to perform the restore process and enforce better backup practices.

This tutorial will use K10. It's easy to install, has a simple GUI, and meets all Kubernetes backup and restore requirements.

Getting Ready

To complete this tutorial, you'll need the following:

• Git, a distributed version control system. You can install git on macOS and Linux using the Homebrew command brew install git. Once installed, clone the tutorial repository (git clone https://github.com/Damaso-DD/k10-tutorial.git), as it will save you time with the rest of the prerequisites.
• jq, a lightweight command line JSON processor. This is required to run K10's preflight script. You can install jq on macOS and Linux using the command brew install jq.
• Helm, a Kubernetes package manager. This is required to install K10. You can install Helm on Linux by running the provided install-helm.sh Bash script.
• The kubectl command line tool, which is essential to manage your Kubernetes cluster. You can install kubectl on Linux by running the provided install-kubectl.sh script.
• A Kubernetes cluster. This tutorial uses a DigitalOcean Kubernetes (DOKS) managed cluster. However, you can use any managed or on-prem Kubernetes environment of your choice. Nevertheless, keep in mind that K10 suffers from some limitations when it comes to local Kubernetes distributions like K3s or kind. Last but not least, at the time of writing this tutorial, Kasten K10 supports up to version 1.27 of Kubernetes, so don't try to use the most recent version.
• A properly configured kubeconfig file, which is essential to access your Kubernetes cluster using kubectl.

Setting Up a Kubernetes Cluster

You'll need to prepare your Kubernetes cluster with a demo app that you can backup and restore using K10.

For this tutorial, you'll deploy the WordPress blogging platform to the Kubernetes cluster. A WordPress application uses resources such as services, pods, and persistent volume claims, as well as a MySQL database to store website data, making it ideal for testing K10's capabilities.

For your convenience, you can use the kustomization.yaml file included in the tutorial repository to deploy WordPress in the demo-app namespace. To do this, navigate to the repository and run the following command:

kubectl apply -k ./ 

Next, check the services in the namespace demo-app by running the command kubectl get svc -n demo-app. The output should be similar to the following:

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE wordpress LoadBalancer 10.245.91.247 138.197.52.34 80:31084/TCP 2m33s wordpress-mysql ClusterIP None <none> 3306/TCP 2m33s 

Since the tutorial uses a managed Kubernetes cluster, the load balancer is automatically provisioned with an external IP that can be used to access WordPress. If you need to learn how to manually set up a service to expose your app, you can check out the official tutorial.

Now that WordPress is running, you can open your favorite browser and navigate to the external IP address provided by the load balancer. On the WordPress configuration screen, you must provide a username and password for the administrator to access the application dashboard. For more information, read Get started with WordPress.

Once logged in to WordPress, you can create or edit a post. This is an optional step, as WordPress already includes an example post titled "Hello World." For the tutorial, the "Hello World" post was edited to look like this:

This was simply done to verify the backups later.

Installing K10

Installing Kasten K10 is straightforward. First, add the Kasten Helm repository and create the kasten-io namespace, which is used by the Helm chart by default:

helm repo add kasten https://charts.kasten.io/ kubectl create namespace kasten-io 

Next, install K10 in the namespace kasten-io:

helm install k10 kasten/k10 --namespace=kasten-io 

This can take a couple of minutes to complete. The command sets up several components required by K10's front end and back end. You can check these components by running kubectl get all -n kasten-io. In particular, the Helm chart configures a Kubernetes VolumeSnapshotClass, which can manage volume snapshots (Kubernetes backup and restore operations discussed earlier) via the Container Storage Interface (CSI).

You need to annotate VolumeSnapshotClass objects to ensure proper operation when taking volume snapshots:

kubectl annotate volumesnapshotclass do-block-storage k10.kasten.io/is-snapshot-class=true 

To verify that all components will work as expected, you can run the preflight check script provided by K10:

curl https://docs.kasten.io/tools/k10_primer.sh | bash 

The output should be similar to the following:

================================================================ Kubernetes Version Check: Valid kubernetes version (v1.26.9) - OK RBAC Check: Kubernetes RBAC is enabled - OK Aggregated Layer Check: The Kubernetes Aggregated Layer is enabled - OK CSI Capabilities Check: Using CSI GroupVersion snapshot.storage.k8s.io/v1 - OK Validating Provisioners: dobs.csi.digitalocean.com: Is a CSI Provisioner - OK Storage Classes: do-block-storage Valid Storage Class - OK do-block-storage-retain Valid Storage Class - OK do-block-storage-xfs Valid Storage Class - OK do-block-storage-xfs-retain Valid Storage Class - OK Volume Snapshot Classes: do-block-storage Has k10.kasten.io/is-snapshot-class annotation set to true - OK Has deletionPolicy 'Delete' - OK Validate Generic Volume Snapshot: Pod created successfully - OK GVS Backup command executed successfully - OK Pod deleted successfully - OK ================================================================ serviceaccount "k10-primer" deleted clusterrolebinding.rbac.authorization.k8s.io "k10-primer" deleted job.batch "k10primer" deleted 

As you can see, the script performs a thorough check. If no problems are found, you can start using K10. To do it, you must access the dashboard, either by exposing it to the internet or using kubectl port-forward. For this tutorial, you will use the latter:

kubectl --namespace kasten-io port-forward service/gateway 8080:8000 

Navigate to http://127.0.0.1:8080/k10/#/ from your local machine, and if everything is working, you'll see a screen like the following:

Provide a business email and company name, then click the button to accept the terms. This will take you to the dashboard:

That's it! Kasten K10 is ready to perform your first backup of the Kubernetes cluster.

Backup Process

You may have noticed in the previous image that the Applications widget lists two applications on the system. If you click it for more details, you'll be taken to the Applications dashboard:

K10 assumes that you use a namespace-based approach to deploying your applications and services, so "applications" are actually the namespaces of your Kubernetes cluster. This explains why the demo-app and default namespaces are listed in the screenshot.

Creating a Backup Policy

On the Applications dashboard, the STATS column lists the space used by persistent volumes (40 GiB), along with the number of configs, services, and workloads in the demo-app namespace. You'll also notice that the POLICY column indicates "Unmanaged," which means that you have not established a schedule (policy) for backing up this application. Correct that by clicking the hamburger menu on the right and selecting Create a Policy:

The next thing you'll see is the New Policy pop-up window:

By default, the name field will be automatically populated using the format NAMESPACE_NAME-backup, but you can change it to any other name you like. Optionally, you can also add comments to help explain the backup policy.

For this tutorial, the snapshot action was chosen with a daily backup frequency. However, K10 supports other intervals as well as advanced frequency options that allow you to customize the snapshot interval according to your requirements. Likewise, you can configure snapshot retention policies that fit your needs.

If you scroll down, you'll notice that there are more options:

You can enable snapshot exports for cross-cluster migration, select target applications by name or label, choose which resources will be included in the backup, and set up pre- and post-snapshot action hooks and a location profile.

For this example, accept the default settings to back up all resources in the demo-app namespace.

It's considered best practice to make frequent backups of critical components of your app (the database, for example) as well as key Kubernetes cluster configuration resources. To do this, you can take advantage of Kubernetes labels, as shown below:

Alternatively, you can filter resources by group, version, name, and so on:

Another way is to create different policies with different intervals and resources.

Now, press the Create Policy button to save the new policy, which will take you to the Policies page:

You can see a summary of the backup process and various actions on the right side.

Manual Backups

There's no better way to test a policy than to run it by pressing the run once icon. Confirm that you want to run the backup, then go to the dashboard to see the progress of the backup:

Once the process is finished, you should see an output indicating that the backup was successful:

If you click it, its details will be shown:

In short, during the backup process, you can either back up the entire namespace or choose what components you need to back up. Furthermore, you can set hourly, daily, weekly, or custom schedules based on your requirements using backup policies.

Cluster backup tools are useless if they cannot be trusted; therefore, it's best practice to frequently validate backups by restoring them in a separate environment to check Kubernetes data protection integrity. That's precisely what you will do next during the restore process.

Restore Process

Let's check that the restore process is working as expected. To do this, you could take advantage of the post that was previously created or modify it again if you wish:

Next, go to the K10 dashboard and click the Applications widget. You'll notice that only the demo-app is listed. This is because the "Compliant" filter is on, which means only apps that have a backup policy will be listed. Click the hamburger menu and select Restore:

This will take you to a screen listing all the backups available for this application. K10 calls these "restoration points." Click the only one available:

A pop-up window will appear with detailed information about the restore point, including the name of the backup policy, the kubectl command to display the restore point information, and the application name (the namespace where you want to restore the app):

Please note that it's considered best practice to stop your applications before starting the restore process to ensure data integrity. These options are available under the "Optional Restore Settings" section.

There, you can configure action settings (blueprints) that can be executed before the restoration process or after it (on success or failure). You'll also see options to restore only the volume data and not other artifacts, and settings for not waiting for workloads to be ready.

For now, accept the default settings and begin the restore process. As before, you can follow the process from the dashboard:

Once the process is complete, you can see more details by clicking the event:

That's it; the restore process is complete. However, you overwrote the existing namespace during the process. A better way would be to follow best practices and restore the backup to a different namespace. Let's see how to do so.

Restore Backups to a Different Namespace

Restoring backups to a different namespace is considered best practice, allowing you to validate backups without affecting running apps and Azure kubernetes service.

You can start this process from the Restore Point screen, where you can create a new namespace and select it as the target of the restore process:

Once the restore process is complete, you'll see both environments on the Applications screen, one "Compliant" and the other "Unmanaged" (since it does not have a backup policy):

You can now run kubectl get svc -n demo-app-backup to get the new external IP provided by the load balancer. This tests the application thoroughly to verify that everything works as it should.

The restore process consists of identifying the components you want to restore, stopping the application, and specifying the restore target. K10's GUI makes performing all of these operations convenient.

Store Backups in an Offsite Location

A best practice that cannot be overlooked is saving backups in an offsite location. In K10, you can achieve this using location profiles, as shown in the screenshot below:

You can configure object storage providers from the New Profile screen, including Amazon S3, Azure Storage, Google Cloud Storage, NFS FileStore, S3 Compatible, and Veeam Repository. Furthermore, you can configure different providers and buckets and assign them to different backup policies using blueprints.

Overview of Best Practices for Backing Up and Restoring Kubernetes Clusters

During this tutorial, you saw how K10 makes enforcing best practices for backing up and restoring Kubernetes clusters easy. To summarize, you can:

• Perform frequent backups of the database cluster configuration using backup policies
• Avoid backing up everything unless needed, and use filters and labels to take frequent backups of essential cluster and app data.
• Segregate namespaces when taking backups, something that K10 does by default.
• Periodically validate your backups to ensure they work in times of need
• Store backups in an offsite location to avoid losing them in the event of an incident
• Stop existing resources before a restore process to prevent data loss

Master the Art of Backup and Restore  Kubernetes Database: Take Action Now.

Ready to elevate your Kubernetes game with Loft? Whether you're a seasoned platform engineer or just starting, mastering the art of backup and restore is crucial to safeguarding your data and maintaining business continuity. 

With powerful tools like Kasten K10, you can automate backups, simplify restores, and protect your clusters from unexpected disasters. Don’t wait until it’s too late—start implementing these best practices now. Take control of your data's safety, and ensure you're prepared for any scenario! Ready to dive in? Follow this step-by-step guide and become the Kubernetes backup expert your team needs.

Frequently Asked Questions

How does kubernetes backup deployment work?

Kubernetes backup involves taking snapshots of the Kubernetes cluster and storing them in a safe location so that essential data can be easily recovered in the event of a data breach or attack. Doing so protects the organization from data loss.

How often should I perform an etcd backup?

Ideally, you should perform one as frequently as possible, but only during non-peak usage hours since it is a blocking action with a high I/O cost.

What applications can you build with kubernetes?

You can build all sorts of powerful applications on k8s, from simple web applications to machine learning applications and enterprise DevOps.

Can you migrate the content of one cluster to another cluster?

Yes, you can. The easiest way to do this is through data migration. Simply snapshot data from one cluster and add it to a repository, then add the repository to the other cluster and restore it there.

What backup software is the best possible option for kubernetes backups?

There are many great options, such as Kasten K10, Velero, Stash, CloudCasa, and Longhorn. Before you choose backup software, though, make sure it is a great fit for your organization and suits your needs.