Admission Webhooks, Explained

Kubernetes admission webhooks are HTTP callbacks that the API server invokes when certain operations occur on cluster resources. There are two types: validating webhooks, which can reject requests to enforce policies, and mutating webhooks, which run first and can modify objects before validation occurs. When a user creates a pod, the API server sends an AdmissionReview request to any configured webhooks that match the operation and resource type. The webhook then returns an AdmissionReview response indicating whether the request is allowed, and if mutating, including a patch that modifies the object.

sequence diagram of a user creating a pod
Figure 1. user creates a pod

It’s common to conflate admission webhooks with admission controllers, but they are distinct concepts. Controllers are control loops that continually watch cluster state and make changes to drive it toward a desired state. An admission controller is essentially an admission webhook that also performs controller duties, such as managing custom resources. Building a functional webhook does not require controller capabilities—it can simply receive requests and return responses synchronously with no side effects.

Building a Minimal Webhook in Go

For a recent project at Slack, we needed to inject tolerations into pods at creation time based on user annotations. Existing frameworks like Kubebuilder and Operator SDK are powerful but bring significant dependencies and features like CRD management that we didn’t need. Instead, we wrote a lightweight Go HTTP server that handles admission requests directly, and released it as slackhq/simple-kubernetes-webhook.

Cluster-Side Setup

The webhook runs as a regular Kubernetes deployment in the cluster. Kubernetes requires that all webhook communication be over HTTPS, so the deployment mounts a Secret containing a TLS certificate and private key. The certificate’s SubjectAltName must match the service hostname, such as simple-kubernetes-webhook.default.svc.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: simple-kubernetes-webhook
  namespace: default
spec:
  selector:
    matchLabels:
      app: simple-kubernetes-webhook
  template:
    metadata:
      labels:
        app: simple-kubernetes-webhook
    spec:
      containers:
        - image: simple-kubernetes-webhook:latest
          name: simple-kubernetes-webhook
          volumeMounts:
            - name: tls
              mountPath: "/etc/admission-webhook/tls"
      volumes:
        - name: tls
          secret:
            secretName: simple-kubernetes-webhook-tls

The API server reaches the webhook through a standard Service object:

apiVersion: v1
kind: Service
metadata:
  name: simple-kubernetes-webhook
  namespace: default
spec:
  ports:
    - port: 443
      protocol: TCP
      targetPort: 443
  selector:
    app: simple-kubernetes-webhook

The TLS material is stored as a Secret:

apiVersion: v1
kind: Secret
metadata:
  name: simple-kubernetes-webhook-tls
type: kubernetes.io/tls
data:
  tls.crt: LS0t...
  tls.key: LS0t...

To register the webhook with the API server, you apply a ValidatingWebhookConfiguration (or MutatingWebhookConfiguration) object. The rules section specifies which operations—for example, CREATE—on which resources—such as pods—should trigger the webhook. The clientConfig points to the service and the HTTPS path, such as /validate-pods. A namespaceSelector limits the webhook to namespaces carrying a specific label. The webhook itself must run in a namespace not subject to the webhook, or you’ll hit a dependency loop whenever its pods are down.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
 name: "simple-kubernetes-webhook.acme.com"
webhooks:
 - name: "simple-kubernetes-webhook.acme.com"
   namespaceSelector:
     matchLabels:
       admission-webhook: enabled
   rules:
     - apiGroups: [""]
       apiVersions: ["v1"]
       operations: ["CREATE"]
       resources: ["pods"]
       scope: "*"
   clientConfig:
     service:
       namespace: default
       name: simple-kubernetes-webhook
       path: /validate-pods
       port: 443
     caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMzREND...

Webhook Server Code

The main application is a standard Go HTTP server that handles requests at /validate-pods and /mutate-pods, matching the paths in the webhook configuration:

func main() {
	// handle our core application
	http.HandleFunc("/validate-pods", ServeValidatePods)
	http.HandleFunc("/mutate-pods", ServeMutatePods)
	http.HandleFunc("/health", ServeHealth)

	logrus.Print("Listening on port 443...")
	logrus.Fatal(http.ListenAndServeTLS(":443", cert, key, nil))
}

The validation handler parses the incoming AdmissionReview—a JSON document containing the pod object plus metadata—and passes it to a component that generates a response AdmissionReview:

// ServeValidatePods validates an admission request and then writes an admission
// review to `w`
func ServeValidatePods(w http.ResponseWriter, r *http.Request) {
	in, err := parseRequest(*r)

	adm := admission.Admitter{
		Logger:  logger,
		Request: in.Request,
	}
	out, err := adm.ValidatePodReview()

	w.Header().Set("Content-Type", "application/json")
	jout, err := json.Marshal(out)
	fmt.Fprintf(w, "%s", jout)
}

A response contains an allowed field. If true, pod creation proceeds. If false, the API server halts the operation and shows the user a customisable error, optionally including a status code and message:

{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "response": {
    "uid": "<value from request.uid>",
    "allowed": true
  }
}

{
 "kind": "AdmissionReview",
 "apiVersion": "admission.k8s.io/v1",
 "response": {
   "uid": "9e8992f7-5761-4a27-a7b0-501b0d61c7f6",
   "allowed": false,
   "status": {
     "message": "pod name contains \"offensive\"",
     "code": 403
   }
 }
}

The handler relies on an abstraction that produces admission responses via its validation and mutation methods:

// Admitter is a container for admission business
type Admitter struct {
	Logger  *logrus.Entry
	Request *admissionv1.AdmissionRequest
}

// MutatePodReview takes an admission request and validates the pod within
// it returns an admission review
func (a Admitter) ValidatePodReview() (*admissionv1.AdmissionReview, error) {
	v := validation.NewValidator(a.Logger)
	val, err := v.ValidatePod(pod)

	return reviewResponse(a.Request.UID, true, http.StatusAccepted, "valid pod"), nil
}

Validation logic is delegated to a structure whose ValidatePod method runs a list of validators implementing a simple interface. Each returns a boolean and, for rejected pods, a reason:

// ValidatePod returns true if a pod is valid
func (v *Validator) ValidatePod(pod *corev1.Pod) (validation, error) {
	// list of all validations to be applied to the pod
	validations := []podValidator{
		nameValidator{v.Logger},
	}

	// apply all validations
	for _, v := range validations {
		var err error
		vp, err := v.Validate(pod)
	}

	return validation{Valid: true, Reason: "valid pod"}, nil
}

In this implementation, the sole validator checks pod names for disallowed substrings.

Mutation with JSON Patches

The mutation path closely mirrors validation, with one key difference: the response must contain a base64-encoded JSON Patch (RFC 6902) describing the modifications to apply to the pod. An example response looks like this:

{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "response": {
    "uid": "<value from request.uid>",
    "allowed": true,
    "patchType": "JSONPatch",
    "patch": "eyJvcCI6ImFkZCIsInBhdGgiOiIvc3BlYy9jb250YWluZXJzLzAvZW52IiwidmFsdWUiOlt7Im5hbWUiOiJLVUJFIiwidmFsdWUiOiJ0cnVlIn1dfQ=="
  }
}

The patch field is decoded to a JSON Patch document:

{"op":"add","path":"/spec/containers/0/env","value":[{"name":"KUBE","value":"true"}]}

Two mutators are provided in the code. One injects a KUBE=true environment variable into the pod, and the other adds a set of tolerations based on a custom pod annotation, working with taints applied to cluster nodes:

mutations := []podMutator{ 
	minLifespanTolerations{Logger: log}, 
	injectEnv{Logger: log}, 
}

Running the Webhook in Practice

It’s worth noting that admission webhooks aren’t ordered — the Kubernetes API server invokes them randomly. Since we specifically needed the new webhook to run last, we added reinvocationPolicy: IfNeeded to the MutatingWebhookConfiguration, which often results in the webhook being called twice. This is a key reason why all mutations should be idempotent: they may be applied more than once per request.

Testing in Production-Like Environments

While a thorough unit test suite and local webhook execution greatly simplified development, they aren’t sufficient on their own. We recently caught several issues in our dev environment — one of which nearly reached production. Having a production-like environment where things can break without consequences is essential for validating new components and features before rollout.

Technical Debt and Lessons Learned

We deliberately omitted that we already had a functional webhook in place. It was built on a very old Kubebuilder version, and upgrading it to a recent release would have required such an extensive rewrite that we chose to write a new webhook from scratch instead. Our original plan was to migrate features incrementally from the old webhook to the new one so we could eventually retire the old codebase — but in practice, we ended up maintaining two webhooks for a period, and finding the time and resources to complete the migration became the remaining challenge.

Update as of February 2023: The migration is finally complete, and the old webhook has been retired.

Conclusion

The entire exercise is a good example of a common principle in systems engineering: at its core, all technical work comes down to data in, data out. This Kubernetes mutating and validating admission webhook simply receives an admission review request and returns an admission review response — with no side effects, no unnecessary complexity, and no lingering maintenance burden. Building it locally made iteration faster, but the real-world caveats around ordering, idempotency, and test environments are what make such projects genuinely instructive.