Scrape Prometheus Metrics for loft
Loft exposes several prometheus style metrics that can be scraped. If you want to monitor other applications with loft take a look at the monitoring guide instead.
Create the Prometheus ServiceMonitor
The metrics can be scraped with the included ServiceMonitor
in the loft chart, which can be deployed with helm. In order for this to work, make sure you have installed a prometheus operator into your cluster. You'll also need to upgrade your loft release with the following command to create the ServiceMonitor
:
Access the metrics endpoint directly without a ServiceMonitor
Loft will expose its internal metrics at https://my-loft-subdomain.my-url.com/metrics
.
By default loft will require an Authorization
http header field with a kubernetes bearer token that has access to the non resource url /metrics
in the kubernetes cluster where loft is installed.
If the header is not provided, loft will deny the request.
Disable metrics authentication
If you wish to scrape metrics without authentication, you can disable it via the environment variable INSECURE_METRICS=true
in the loft helm chart.
If you have kubernetes service account token that has the appropriate rights, you can access the metrics via curl:
Which metrics are exposed?
Loft collects metrics from its internal components and merges them together into a single metrics endpoint at /metrics
. The following component metrics are merged together:
- API gateway metrics: loft exposes metrics for its internal api gateway such as the total amount of requests (
apigateway_ENDPOINT_request_total
), request latency (apigateway_ENDPOINT_request_duration_seconds
) and request sizes (apigateway_ENDPOINT_response_sizes
) for the different endpoints. The metrics have different labels depending on the endpoint. The following endpoints and metrics are available:- ui: all requests that target loft UI assets (metrics are
apigateway_ui_request_total
,apigateway_ui_request_duration_seconds
,apigateway_ui_response_sizes
) - auth: all requests that target loft authentication endpoints (such as login, token refresh etc.)
- kubernetes: all forwarded requests that target a kubernetes cluster. The targeted backend is provided by a label and can be either
management
(the integrated loft kubernetes api server),cluster
(a request forwarded to a connected cluster) orvcluster
(a virtual kubernetes cluster deployed by loft). Further labels are provided as well such as kubernetes resource, api version, api group, subresource etc. - grafana: all forwarded requests to a grafana instance in a connected cluster
- oidc: all requests that target the internal OIDC server
- ui: all requests that target loft UI assets (metrics are
- Integrated Kubernetes API server metrics: all internal kubernetes api server metrics are exposed. An incomplete list can be found in the kubernetes repository
- Integrated Controller manager metrics: all loft management cluster controller metrics are exposed.
- Go runtime metrics: some general metrics about the golang runtime and resource usage are exposed.