Auditing

Enterprise Feature

Auditing loft requests is an enterprise feature. Please make sure your license permits auditing before you follow this guide.

Loft auditing provides a security-relevant, chronological set of records documenting the sequence of actions in loft. Loft audits the activities generated by users and by applications that use the Loft API.

Auditing allows cluster administrators to answer the following questions:

• what happened?
• when did it happen?
• who initiated it?
• on what did it happen?
• where was it observed?
• from where was it initiated?
• to where was it going?

Auditing Loft is very similar to auditing Kubernetes clusters.

Enable Auditing

Loft auditing is configured through the Loft config in the Loft UI (Admin -> Config) or via the secret loft/loft-config.

Example Loft Auditing Configuration

Loft Restart Required

Changing the Loft auditing configuration requires a restart to take effect. You can restart Loft either through the Loft UI or via kubectl: kubectl rollout restart deploy/loft -n loft

Each request on each stage of its execution generates an audit event, which is then pre-processed according to a certain policy and written to a backend (currently only log backends are supported). The policy determines what's recorded and the backends persist the records.

Each request can be recorded with an associated stage. The defined stages are:

• RequestReceived - The stage for events generated as soon as the audit handler receives the request, and before it is delegated down the handler chain.
• ResponseComplete - The response body has been completed and no more bytes will be sent.
• Panic - Events generated when a panic occurred.

info

The audit logging feature increases the memory consumption of Loft because some context required for auditing is stored for each request. Memory consumption depends on the audit logging configuration.

Audit Policy

danger

If no policy is configured, no events are logged. Note that the rules field must be provided in the audit policy config.

Audit policy defines rules about what events should be recorded and what data they should include. When an event is processed, it's compared against the list of rules in order. The first matching rule sets the audit level of the event. The defined audit levels are:

• None - don't log events that match this rule.
• Metadata - log request metadata (requesting user, timestamp, resource, verb, etc.) but not request or response body.
• Request - log event metadata and request body but not response body. This does not apply for non-resource requests.
• RequestResponse - log event metadata, request and response bodies. This does not apply for non-resource requests.

Since Loft acts as an API Gateway, it might be useful to record only events that target the Loft management API, connected Kubernetes clusters or virtual clusters. You can define that in a policy rule by specifying the requestTarget:

• Management - requests that target the loft management API.
• Cluster - requests that target a connected Kubernetes cluster. You can select specific connected clusters via the rules[*].clusters field.
• VCluster - requests that target virtual Kubernetes clusters.

Below is an example audit policy configuration (see last section for a complete policy reference):

audit:
  # Enable auditing
  enabled: true
  # The path to the audit log (Use - to log to console)
  path: /tmp/loft-audit.log
  # The policy to use
  policy:
    # Don't generate audit events for all requests in RequestReceived stage.
    omitStages:
    - "RequestReceived"
    rules:
    # Log pod changes at RequestResponse level
    - level: RequestResponse
      resources:
      - group: ""
        # Resource "pods" doesn't match requests to any subresource of pods,
        # which is consistent with the RBAC policy.
        resources: ["pods"]
    # Log "pods/log", "pods/status" at Metadata level
    - level: Metadata
      resources:
      - group: ""
        resources: ["pods/log", "pods/status"]
      
    # Don't log requests to a configmap called "controller-leader"
    - level: None
      resources:
      - group: ""
        resources: ["configmaps"]
        resourceNames: ["controller-leader"]
    
    # Don't log watch requests by the "system:kube-proxy" on endpoints or services
    - level: None
      users: ["system:kube-proxy"]
      verbs: ["watch"]
      resources:
      - group: "" # core API group
        resources: ["endpoints", "services"]
    
    # Don't log authenticated requests to certain non-resource URL paths.
    - level: None
      userGroups: ["system:authenticated"]
      nonResourceURLs:
      - "/api*" # Wildcard matching.
      - "/version"
    
    # Log the request body of configmap changes in kube-system.
    - level: Request
      resources:
      - group: "" # core API group
        resources: ["configmaps"]
      # This rule only applies to resources in the "kube-system" namespace.
      # The empty string "" can be used to select non-namespaced resources.
      namespaces: ["kube-system"]
    
    # Log configmap and secret changes in all other namespaces at the Metadata level.
    - level: Metadata
      resources:
      - group: "" # core API group
        resources: ["secrets", "configmaps"]
    
    # Log all other resources in core and extensions at the Request level.
    - level: Request
      resources:
      - group: "" # core API group
      - group: "extensions" # Version of group should NOT be included.
      
    # Log loft management changes at RequestResponse level
    - level: RequestResponse
      requestTargets: ["Management"]
      verbs: ["create", "update", "delete", "patch"]
      
    # Log connected cluster my-connected-cluster requests at RequestResponse level
    - level: RequestResponse
      clusters: ["my-connected-cluster"]
    
    # A catch-all rule to log all other requests at the Metadata level.
    - level: Metadata
      # Long-running requests like watches that fall under this rule will not
      # generate an audit event in RequestReceived.
      omitStages:
        - "RequestReceived"

Audit Log Entries

After a policy is in place, you can observe Loft logging incoming requests into the log file at audit.path.

An example log file entry for a stage Metadata request could look like this:

{
   "level":"Metadata",
   "auditID":"e09e12af-37b0-407b-9a13-a5835b67915e",
   "stage":"ResponseComplete",
   "requestURI":"/kubernetes/cluster/loft-cluster/api/v1/namespaces/default/pods",
   "verb":"list",
   "user":{
      "username":"test",
      "uid":"9780846f-3340-4323-803d-14fdcf6bb0f8",
      "groups":[
         "system:authenticated",
         "loft:authenticated"
      ],
      "extra":{
         "loft:cluster":[
            "loft-cluster"
         ]
      }
   },
   "sourceIPs":[
      "127.0.0.1"
   ],
   "userAgent":"kubectl/v0.0.0 (darwin/amd64) kubernetes/$Format",
   "objectRef":{
      "resource":"pods",
      "namespace":"default",
      "apiVersion":"v1"
   },
   "responseStatus":{
      "metadata":{
         
      },
      "status":"Success",
      "code":200
   },
   "requestReceivedTimestamp":"2021-03-02T10:17:26.478644Z",
   "stageTimestamp":"2021-03-02T10:17:26.482245Z"
}

An example log file entry for a stage ResponseComplete request could look like this:

{
   "level":"RequestResponse",
   "auditID":"fd210696-8943-4121-a185-bed9cd718d86",
   "stage":"ResponseComplete",
   "requestURI":"/kubernetes/cluster/loft-cluster/api/v1/namespaces/default/pods",
   "verb":"create",
   "user":{
      "username":"test",
      "uid":"9780846f-3340-4323-803d-14fdcf6bb0f8",
      "groups":[
         "system:authenticated",
         "loft:authenticated"
      ],
      "extra":{
         "loft:cluster":[
            "loft-cluster"
         ]
      }
   },
   "sourceIPs":[
      "127.0.0.1"
   ],
   "userAgent":"kubectl/v0.0.0 (darwin/amd64) kubernetes/$Format",
   "objectRef":{
      "resource":"pods",
      "namespace":"default",
      "apiVersion":"v1"
   },
   "responseStatus":{
      "metadata":{
         
      },
      "status":"Success",
      "code":201
   },
   "requestObject":{
      "apiVersion":"v1",
      "kind":"Pod",
      "metadata":{
         "name":"ubuntu",
         "namespace":"default"
      },
      "spec":{
         "containers":[
            {
               "image":"alpine",
               "name":"alpine"
            }
         ]
      }
   },
   "responseObject":{
      "kind":"Pod",
      "apiVersion":"v1",
      "metadata":{
         "name":"ubuntu",
         "namespace":"default",
         "selfLink":"/api/v1/namespaces/default/pods/ubuntu",
         "uid":"966ca4bf-7b28-49f7-86db-d156308fa77b",
         "resourceVersion":"44090",
         "creationTimestamp":"2021-03-02T09:21:02Z",
         "annotations":{
            "kubernetes.io/limit-ranger":"LimitRanger plugin set: cpu, memory request for container alpine; cpu, memory limit for container alpine"
         },
      },
      "spec":{
         "volumes":[
            {
               "name":"default-token-jfnkk",
               "secret":{
                  "secretName":"default-token-jfnkk",
                  "defaultMode":420
               }
            }
         ],
         "containers":[
            {
               "name":"alpine",
               "image":"alpine",
               "resources":{
                  "limits":{
                     "cpu":"2",
                     "memory":"4Gi"
                  },
                  "requests":{
                     "cpu":"20m",
                     "memory":"64Mi"
                  }
               },
               "volumeMounts":[
                  {
                     "name":"default-token-jfnkk",
                     "readOnly":true,
                     "mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"
                  }
               ],
               "terminationMessagePath":"/dev/termination-log",
               "terminationMessagePolicy":"File",
               "imagePullPolicy":"Always"
            }
         ],
         "restartPolicy":"Always",
         "terminationGracePeriodSeconds":30,
         "dnsPolicy":"ClusterFirst",
         "serviceAccountName":"default",
         "serviceAccount":"default",
         "securityContext":{
            
         },
         "schedulerName":"default-scheduler",
         "tolerations":[
            {
               "key":"node.kubernetes.io/not-ready",
               "operator":"Exists",
               "effect":"NoExecute",
               "tolerationSeconds":300
            },
            {
               "key":"node.kubernetes.io/unreachable",
               "operator":"Exists",
               "effect":"NoExecute",
               "tolerationSeconds":300
            }
         ],
         "priority":0,
         "enableServiceLinks":true,
         "preemptionPolicy":"PreemptLowerPriority"
      },
      "status":{
         "phase":"Pending",
         "qosClass":"Burstable"
      }
   },
   "requestReceivedTimestamp":"2021-03-02T09:21:02.928405Z",
   "stageTimestamp":"2021-03-02T09:21:02.954019Z"
}

Full Audit Config Reference

# 
# Loft auditing provides a security-relevant, chronological set of records documenting the sequence
# of actions in a cluster. Every action targeting the loft management API, that is forwarded to 
# a connected kubernetes cluster or targets a virtual cluster is audited and recorded. 
# Loft audits the activities generated by users and by applications that use loft kubernetes contexts.
#
audit:
  # Whether auditing should be enabled
  enabled: true
  # The path where to save the audit log files. This is required if auditing is enabled. Backup log files will 
  # be retained in the same directory. Use - to log all messages to the console instead of log files.
  path: /tmp/audit.log
  # The audit policy to use and log requests. By default loft will not log anything. The policy structure 
  # is the same as a kubernetes audit policy. 
  # See also: https://kubernetes.io/docs/tasks/debug-application-cluster/audit/#audit-policy
  policy:
    # rules specify the audit Level a request should be recorded at.
    # A request may match multiple rules, in which case the FIRST matching rule is used.
    # The default audit level is None, but can be overridden by a catch-all rule at the end of the list.
    # PolicyRules are strictly ordered.
    rules: 
      # The Level that requests matching this rule are recorded at.
      # The available levels are: None, Metadata, Request & RequestResponse
      level: None
      # (Optional) The users (by authenticated user name) this rule applies to.
      # An empty list implies every user.
      users: []
      # (Optional) The user groups this rule applies to. A user is considered matching
      # if it is a member of any of the UserGroups.
      # An empty list implies every user group.
      userGroups: []
      # (Optional) The verbs that match this rule.
      # An empty list implies every verb.
      verbs: ["create", "update"]
      
      # Rules can apply to API resources (such as "pods" or "secrets"),
      # non-resource URL paths (such as "/api"), or neither, but not both.
      # If neither is specified, the rule is treated as a default for all URLs.

      # (Optional) Resources that this rule matches. An empty list implies all kinds in all API groups.
      resources: ["pods"]
      # (Optional) Namespaces that this rule matches.
      # The empty string "" matches non-namespaced resources.
      # An empty list implies every namespace.
      namespaces: []

      # (Optional) NonResourceURLs is a set of URL paths that should be audited.
      # *s are allowed, but only as the full, final step in the path.
      nonResourceURLs: []

      # (Optional) OmitStages is a list of stages for which no events are created. Note that this can also
      # be specified policy wide in which case the union of both are omitted.
      # An empty list means no restrictions will apply.
      omitStages: []

      # (Optional) RequestTargets is a list of request targets for which events are created. 
      # An empty list implies every request.
      requestTargets: ["Management", "Cluster", "VCluster"]
    
      # (Optional) Clusters that this rule matches. Only applies to cluster requests.
      # If this is set, no events for non-cluster requests will be created.
      # An empty list means no restrictions will apply.
      clusters: ["my-connected-cluster"]
    # omitStages is a list of stages for which no events are created. Note that this can also
    # be specified per rule in which case the union of both are omitted.
    omitStages: ["RequestReceived", "ResponseComplete", "Panic"]
  # (Optional) MaxAge is the maximum number of days to retain old log files based on the timestamp 
  # encoded in their filename.
  maxAge: 0
  # (Optional) MaxBackups is the maximum number of old log files to retain.
  maxBackups: 0
  # (Optional) MaxSize is the maximum size in megabytes of the log file before it gets rotated. 
  # It defaults to 100 megabytes.
  maxSize: 0