The sleep mode feature in Loft puts spaces or virtual clusters to sleep - either after a specified period of inactivity or manually triggered by a user. Spaces and virtual clusters automatically wake up when a user runs the first kubectl, helm or any other command using the space or virtual clusters. Users can also manually trigger the wake up process by clicking a button in the UI. Instead of the sleep mode, you are also able to configure automatic deletion of spaces.
Technically, virtual clusters do not have a sleep mode on their own. Instead, you can enable the sleep mode for the underlying namespace that the virtual cluster runs in. For convenience and easier understanding, the Loft UI shows the sleep mode statistics for each virtual cluster but in fact the numbers reflect the sleep mode of the underlying space of each virtual cluster.
Note that when a space or virtual cluster goes to sleep, all persistent data as well as all Kubernetes objects will be preserved. Only the running containers will be terminated when the sleep mode starts. And these containers will be restarted once the space or virtual cluster wakes up again. Learn more about how sleep mode works.
Use Case & Benefits
While some of the spaces and virtual clusters that you manage with Loft may contain production workloads which must run all day every day, there are many use cases where you want to create spaces and virtual clusters for development, testing, experimentation or CI/CD. These non-production workloads are usually just required to run when engineers are actually working with them. Considering a regular 40-hour work week, engineers work about 24% of the overall time per week:
And this calculation is not even including time for meetings, sick days, holidays etc. That means you can save more than 76% of your cloud infrastructure cost for idle namespaces or virtual clusters. Some namespaces and virtual clusters may even be running without any engineer touching them for weeks because one of the biggest downsides of the cloud is that while it is easy to spin up things, hardly anyone ever shuts down anything without being forced to do so.
Sleep mode offers a fully automated solution for this problem. If your cluster is configured to horizontally auto-scale its nodes (which most public cloud providers allow you to enable with just a single click), you can save over 76% of infrastructure cost for your clusters by enabling sleep mode.
When putting a space or virtual cluster to sleep, Loft looks up and saves the
replica number for Deployments, StatefulSets, DaemonSets and other ReplicaSet based resources. Then, it scales down this replica number to 0 which triggers Kubernetes to delete all pods/containers.
With Loft, you can:
- enforce automatic sleep for all spaces of a certain cluster account after a certain period of inactivity
- enable automatic sleep for individual spaces after a certain period of inactivity
- trigger sleep mode manually for individual spaces
Automatic For Entire Account
Cluster admins can enforce the sleep mode annotation for all spaces of a certain account to ensure that all spaces go to sleep after a certain period of inactivity.
Annotations set in the account settings cannot be overridden by the user when creating a space. The default RBAC set up by Loft does not allow users to modify their namespaces after creation. If you add additional RBAC to allow users to update their namespaces, users will be able to deactivate or manually trigger sleep mode.
Automatic For Individual Spaces
When Loft wakes up a space, it starts all pods/containers it has previously removed from the namespace. Loft does this by changing the
replica number for Deployments, StatefulSets, DaemonSets and other ReplicaSet based resources.
Loft is configured to wake up spaces automatically once they are being used. Using a space means to send any request to the Kubernetes API server involving any resources within the corresponding namespace. It does not matter if this request comes from running a kubectl command, a helm command or using any other tool, e.g.:
How does it work?
Sleep mode is configured by annotations in the metadata of a Kubernetes namespace. After a certain amount of inactivity, the space starts sleeping.
How is the inactivity determined?
All requests that are made through the Loft API server count as activity in the namespace. Let's say you have created a new space via:
and then you access resources within the namespace that will count as activity in the namespace. Some examples:
As you can see in the examples above, all kubernetes requests that are routed through the Loft API server are counted as activity within the namespace and will reset the period until it sleeps or is deleted.
One exception to this rule are requests that are made in the Loft UI, since the Loft UI always sets the
X-Sleep-Mode-Ignore: true header for each request.
There are multiple annotations you can set on a namespace to change what counts as activity within that namespace:
sleepmode.loft.sh/last-activity: this is usually set automatically by Loft after an activity was detected within a namespace, however you can also change this value as you like
sleepmode.loft.sh/ignore-all: if this annotation is set to "true", then all requests will be ignored and not count as activity
sleepmode.loft.sh/ignore-vclusters: if "true", activity that occurs within a created vcluster within this namespace does not count as activity
sleepmode.loft.sh/ignore-groups: a comma separated list of user groups that do not count as activity (e.g. loft:team:admins would ignore all users in the team admins or loft:user:admin would ignore the admin user)
sleepmode.loft.sh/ignore-resources: a comma separated list of kubernetes resources that do not count as activity (e.g. pods, deployments.apps, secrets etc.)
sleepmode.loft.sh/ignore-verbs: a comma separated list of kubernetes verbs that do not count as activity (e.g. get, list, create, update, patch, delete)
sleepmode.loft.sh/ignore-resource-verbs: a comma separated list of resources and verbs that do not count as activity (format: myresource.mygroup=create update delete,myresource2.mygroup=create update)
sleepmode.loft.sh/ignore-resource-names: a comma separated list of resources and names that do not count as activity (format: myresource.mygroup=name1 name2)
sleepmode.loft.sh/ignore-active-conntections: if this annotation is set to "true", then still active kubernetes connections will be ignored for determining activity in a space
If you want to set some of this annotations automatically on space creation, you can add these to the space template in the account settings.
You can check when a namespace will start sleeping or will be deleted by hovering over its status in the Loft UI. Open connections describes how many keep alive connections are still active to that namespace (e.g. through kubectl exec).
How can I specify when a namespace should sleep or should be deleted?
The period when a namespace should start sleeping can be configured through namespace annotations:
sleepmode.loft.sh/sleep-after: tells Loft to trigger sleep mode automatically after x seconds of inactivity
sleepmode.loft.sh/delete-after: tells Loft to delete the namespace automatically after x seconds of inactivity
sleepmode.loft.sh/force: if "true" tells Loft to force sleep this namespace
sleepmode.loft.sh/force-duration: the amount of seconds after the space starts sleeping after which no automatic wake up should occur (a value of 0 means infinite). This can be useful if there is an application accessing the space that would directly wake up the space again after it started sleeping.
sleepmode.loft.sh/delete-all-pods: if "true" will delete all running pods within a namespace as soon as it starts sleeping, besides scaling down deployments, replicasets and statefulsets
Loft checks every space with an automatic sleep mode annotation in a fixed interval of 1 minute.
How to deal with long running requests
It's possible that certain requests such as
kubectl exec or
kubectl port-forward keep an active connection open to the space that prevents it from sleeping. This is in most cases wanted since the space shouldn't start sleeping when someone is still clearly using it, however there are cases where the connection might be idle (someone left the laptop open, but is not using it anymore) in which you want to terminate such connections and put the space to sleep.
There are multiple ways to approach this problem of active connections preventing a space from sleeping:
You can tell loft to timeout idle streaming connections (such as
kubectl port-forwardetc.) after a certain time period with the annotation
loft.sh/streaming-connection-idle-timeout: '3600'on a cluster. With this annotation set, loft will close connections automatically that are idle after the given seconds. By default, Loft will not timeout any connections. This will only apply to new opened connections to that cluster and not affect already running connections.
Tell loft to ignore all active connections for determining space activity via the space annotation
sleepmode.loft.sh/ignore-active-conntections: 'true'. This will put a space to sleep even though there still might be open connections such as
Configure your kubelets with the flag
--streaming-connection-idle-timeout duration(see docs). This behaves essentially as option 1, however this configuration is Loft independent.
How can I prevent certain resources within a space from sleeping?
Loft allows you to specify resources that should not sleep within a space by providing the annotation
sleepmode.loft.sh/exclude: 'true' on either a
Pod. For example the following
Deployment would not sleep if the space is sleeping: