Auditing

This guide covers how to enable Kubernetes API auditing on a kind cluster.

Overview 🔗︎

Kubernetes auditing provides a security-relevant, chronological set of records documenting the sequence of actions in a cluster. Auditing requires a file to define the audit policy and a backend configuration to store the logged events. Auditing supports two types of backends: log (file) & webhook. The following exercise uses the log backend.

Steps:

Setup 🔗︎

Create an audit-policy.yaml file 🔗︎

The audit policy defines the level of granularity outputted by the Kubernetes API server. The example below logs all requests at the “Metadata” level. See the audit policy docs for more examples.

 
cat <<EOF > audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
EOF

Create a kind-config.yaml file. 🔗︎

To enable audit logging, use kind’s configuration file to pass additional setup instructions. Kind uses kubeadm to provision the cluster and the configuration file has the ability to pass kubeadmConfigPatches for further customization.

 
cat <<EOF > kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
        # enable auditing flags on the API server
        extraArgs:
          audit-log-path: /var/log/kubernetes/kube-apiserver-audit.log
          audit-policy-file: /etc/kubernetes/policies/audit-policy.yaml
        # mount new files / directories on the control plane
        extraVolumes:
          - name: audit-policies
            hostPath: /etc/kubernetes/policies
            mountPath: /etc/kubernetes/policies
            readOnly: true
            pathType: "DirectoryOrCreate"
          - name: "audit-logs"
            hostPath: "/var/log/kubernetes"
            mountPath: "/var/log/kubernetes"
            readOnly: false
            pathType: DirectoryOrCreate
  # mount the local file on the control plane
  extraMounts:
  - hostPath: ./audit-policy.yaml
    containerPath: /etc/kubernetes/policies/audit-policy.yaml
    readOnly: true
EOF

Launch a new cluster 🔗︎ 

kind create cluster --config kind-config.yaml

View audit logs 🔗︎

Once the cluster is running, view the log files on the control plane in /var/log/kubernetes/kube-apiserver-audit.log.

 
docker exec kind-control-plane cat /var/log/kubernetes/kube-apiserver-audit.log

Troubleshooting 🔗︎

If logs are not present, let’s ensure a few things are in place.

Is the local audit-policy file mounted in the control-plane? 🔗︎ 

docker exec kind-control-plane ls /etc/kubernetes/policies

Expected output:

audit-policy.yaml

Does the API server contain the mounts and arguments? 🔗︎ 

docker exec kind-control-plane cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep audit

Expected output:

    - --audit-log-path=/var/log/kubernetes/kube-apiserver-audit.log
    - --audit-policy-file=/etc/kubernetes/policies/audit-policy.yaml
      name: audit-logs
      name: audit-policies
    name: audit-logs
    name: audit-policies

If the control plane requires further debugging use docker exec -it kind-control-plane bash to start an interactive terminal session with the container.