The commands/steps listed on this page can be used to check loft related issues in your cluster. If you should experience any issues that are not mentioned here or you are stuck investigating, please feel free to contact us either via the integrated Chat on this site, via the Kubernetes Slack channels #devspace and #kiosk or write us an E-Mail to firstname.lastname@example.org. We are always happy to help you out.
You can always try to connect to Loft directly and circumvent any load balancers and ingress controllers in front of Loft by using port forwarding. You can use
After that you can access Loft at
https://localhost:8080. You can even login with the Loft CLI to this URL with:
If you can access Loft via port-forwarding this is usually an indicator that the problem lies in front of Loft and might be related to a faulty Load Balancer or Ingress Controller configuration.
DevSpace / Kubectl Error: error upgrading connection
If you receive this error with a Loft context during
kubectl exec or
devspace dev this indicates that the connection to Loft does not support upgrade requests (SPDY) that are required by Kubernetes and Loft. Example:
The error means that Loft did not receive an
Upgrade http header, which needs to be resent by each reverse proxy that sits in front of Loft, because this is a hop-by-hop header. A more in depth explanation can be found in the nginx documentation.
If you are using AWS, please make sure you do NOT use any ALB or HTTP Load Balancers as they do not support the required upgrade requests. Please use an ELB with TCP backend or NLB Load Balancer instead.
If you want to use ACM for TLS termination instead of in cluster certificate management, you can use a NLB that does TLS termination and routes the raw traffic to Loft:
If you are using the ingress nginx ingress controller, Loft should work out of the box.
Nginx Ingress Controller
If you are using the nginx ingress controller, please make sure you have the annotation
nginx.org/websocket-services: loft set on the ingress.
Unfortunately istio does not support SPDY upgrade connections with HTTP routing, which is why you cannot use TLS termination at the istio Gateway itself and must use SNI to forward TLS traffic to loft which does the termination. To configure which TLS certificate loft is using see the self signed certificate guide
An example configuration could look like this:
Other Ingress Controllers
If you are using another ingress controller, please make sure WebSocket support is enabled. Loft should work with most ingress controllers that support websockets.
Error: failed calling webhook "validate.loft.sh": Post xxx/validate?timeout=10s: context deadline exceeded (InternalError)
This error occurs if the Kubernetes master cannot contact the Loft or Kiosk pods for resource validation. Usually this error occurs in private Kubernetes clusters where the Kubernetes master and nodes are in different VPC subnets that have a firewall enforced that forbids connection between those subnets on certain ports.
Loft requires that the Kubernetes master can contact the Loft and Kiosk pods on the following ports:
- 8888 (loft api service extension)
- 8443 (kiosk api service extension)
- 9443 (kiosk webhook & loft webhook)
Especially in private GKE clusters you could see this error, because by default the communication to those ports will be blocked. To enable communication for private GKE clusters, please take a look at the Google Cloud Documenation
Connecting Rancher Clusters
If you want to connect clusters that were created with Rancher, you'll need to make sure that you are using the authorized cluster endpoints that provides a kube context that bypasses the rancher authentication. Otherwise, you will get problems with different users authenticating through Loft as the default Rancher proxy does not correctly support impersonation that is used by Loft.