Loft Troubleshooting

The commands/steps listed on this page can be used to check loft related issues in your cluster. If you should experience any issues that are not mentioned here or you are stuck investigating, please feel free to contact us either via the integrated Chat on this site, via the Kubernetes Slack channels #devspace and #kiosk or write us an E-Mail to support@loft.sh. We are always happy to help you out.

Networking

You can always try to connect to Loft directly and circumvent any load balancers and ingress controllers in front of Loft by using port forwarding. You can use

kubectl port-forward deploy/loft -n loft 8080:443

After that you can access Loft at https://localhost:8080. You can even login with the Loft CLI to this URL with:

loft login localhost:8080 --insecure

If you can access Loft via port-forwarding this is usually an indicator that the problem lies in front of Loft and might be related to a faulty Load Balancer or Ingress Controller configuration.

DevSpace / Kubectl Error: error upgrading connection

If you receive this error with a Loft context during kubectl exec or devspace dev this indicates that the connection to Loft does not support upgrade requests (SPDY) that are required by Kubernetes and Loft. Example:

$ kubectl exec -it ubuntu -- bash
Error from server (BadRequest): Upgrade request required

The error means that Loft did not receive an Upgrade http header, which needs to be resent by each reverse proxy that sits in front of Loft, because this is a hop-by-hop header. A more in depth explanation can be found in the nginx documentation.

AWS

If you are using AWS, please make sure you do NOT use any ALB or HTTP Load Balancers as they do not support the required upgrade requests. Please use an ELB with TCP backend or NLB Load Balancer instead.

If you want to use ACM for TLS termination instead of in cluster certificate management, you can use a NLB that does TLS termination and routes the raw traffic to Loft:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-west-2:xxx:certificate/xxx"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
  name: loft-loadbalancer
  namespace: loft
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 80
  selector:
    app: loft
    release: loft

Ingress Nginx

If you are using the ingress nginx ingress controller, Loft should work out of the box.

Nginx Ingress Controller

If you are using the nginx ingress controller, please make sure you have the annotation nginx.org/websocket-services: loft set on the ingress.

Istio

Unfortunately istio does not support SPDY upgrade connections with HTTP routing, which is why you cannot use TLS termination at the istio Gateway itself and must use SNI to forward TLS traffic to loft which does the termination. To configure which TLS certificate loft is using see the self signed certificate guide

An example configuration could look like this:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: loft-gateway
  namespace: loft
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - loft.my.domain.com
      port:
        name: loft-https
        number: 443
        protocol: HTTPS
      tls:
        mode: PASSTHROUGH
    - hosts:
        - loft.my.domain.com
      port:
        name: loft-http
        number: 80
        protocol: HTTP
      tls:
        httpsRedirect: true
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: loft-virtualservice
  namespace: loft
spec:
  gateways:
    - loft-gateway
  hosts:
    - loft.my.domain.com
  http:
    - match:
      - uri:
          prefix: /
      route:
      - destination:
          host: loft.loft.svc.cluster.local
          port:
            number: 80
  tls:
    - match:
      - port: 443
        sniHosts:
          - loft.my.domain.com
      route:
      - destination:
          host: loft.loft.svc.cluster.local

Other Ingress Controllers

If you are using another ingress controller, please make sure WebSocket support is enabled. Loft should work with most ingress controllers that support websockets.

Error: failed calling webhook "validate.loft.sh": Post xxx/validate?timeout=10s: context deadline exceeded (InternalError)

This error occurs if the Kubernetes master cannot contact the Loft or Kiosk pods for resource validation. Usually this error occurs in private Kubernetes clusters where the Kubernetes master and nodes are in different VPC subnets that have a firewall enforced that forbids connection between those subnets on certain ports.

Loft requires that the Kubernetes master can contact the Loft and Kiosk pods on the following ports:

• 8888 (loft api service extension)
• 8443 (kiosk api service extension)
• 9443 (kiosk webhook & loft webhook)

Especially in private GKE clusters you could see this error, because by default the communication to those ports will be blocked. To enable communication for private GKE clusters, please take a look at the Google Cloud Documenation

Connecting Rancher Clusters

If you want to connect clusters that were created with Rancher, you'll need to make sure that you are using the authorized cluster endpoints that provides a kube context that bypasses the rancher authentication. Otherwise, you will get problems with different users authenticating through Loft as the default Rancher proxy does not correctly support impersonation that is used by Loft.