Kubectl with Cloudflare Zero Trust

Using Cloudflare Zero Trust with Kubernetes to enable kubectl without SOCKS proxies

TS

Terin StockCloudflare

·June 24, 2022

Securing Kubernetes Access with Cloudflare Zero Trust

Cloudflare runs a significant amount of engineering workloads on Kubernetes, from API backends and batch-processing jobs to CI/CD pipelines. The default attack surface—load balancers, API servers, etcd, ingresses, and pods—is substantial. To reduce that exposure, the team tightly restricts network access to clusters, adding Cloudflare Access or mutual TLS (or both) on any exposed ingress.

Those restrictions extend to the Kubernetes API server itself. Blocking it outright would have been a non-starter: engineers need kubectl for troubleshooting and velocity, even when GitOps and Continuous Deployments are the norm. The solution the team landed on is Cloudflare Zero Trust private network routing, which lets engineers reach the API server without configuring proxies or tunnels on their devices.

From VPN to Tunnels to Zero Trust

Previously, engineers connected to the Kubernetes API through a VPN appliance. That worked but dropped them onto the internal network, granting far more access than needed. When Cloudflare retired its VPN in early 2020, the Kubernetes team had to find another path.

Working with the Cloudflare Tunnels team, they added support for kubectl connections through Access and cloudflared tunnels. That solved the immediate problem but created an onboarding burden: each cluster required its own tunnel connection from the engineer's machine, and switching between clusters was tedious. SOCKS proxy support in kubectl existed, but not every Kubernetes ecosystem tool supported it.

The next step was adopting the Zero Trust agent, initially for secure DNS with 1.1.1.1 and later for broader Zero Trust features. The team now uses private network routing to reach Kubernetes APIs directly—no per-cluster tunnel setup, no kubectl configuration changes.

Configuring the Zero Trust Side

The Zero Trust configuration is managed as infrastructure-as-code, though the same settings can be applied through the Cloudflare Zero Trust dashboard. The first piece is a new tunnel that connects the Cloudflare edge to the Kubernetes API.

resource "cloudflare_argo_tunnel" "k8s_zero_trust_tunnel" {
  account_id = var.account_id
  name       = "k8s_zero_trust_tunnel"
  secret     = var.tunnel_secret
}

The tunnel_secret should be a 32-byte random number; save it temporarily because it will be reused during the Kubernetes deployment.

With the tunnel created, routes must be defined so the Cloudflare network knows which traffic to send through it.

resource "cloudflare_tunnel_route" "k8s_zero_trust_tunnel_ipv4" {
  account_id = var.account_id
  tunnel_id  = cloudflare_argo_tunnel.k8s_zero_trust_tunnel.id
  network    = "198.51.100.101/32"
  comment    = "Kubernetes API Server (IPv4)"
}
 
resource "cloudflare_tunnel_route" "k8s_zero_trust_tunnel_ipv6" {
  account_id = var.account_id
  tunnel_id  = cloudflare_argo_tunnel.k8s_zero_trust_tunnel.id
  network    = "2001:DB8::101/128"
  comment    = "Kubernetes API Server (IPv6)"
}

Both IPv4 and IPv6 are supported for API server access, so routes for both address families should be configured. If the API server is reached via hostname, these IPs should match DNS lookup results.

Next, Cloudflare Gateway settings need to align with the API servers and clients.

resource "cloudflare_teams_list" "k8s_apiserver_ips" {
  account_id = var.account_id
  name       = "Kubernetes API IPs"
  type       = "IP"
  items      = ["198.51.100.101/32", "2001:DB8::101/128"]
}
 
resource "cloudflare_teams_rule" "k8s_apiserver_zero_trust_http" {
  account_id  = var.account_id
  name        = "Don't inspect Kubernetes API"
  description = "Allow connections from kubectl to API"
  precedence  = 10000
  action      = "off"
  enabled     = true
  filters     = ["http"]
  traffic     = format("any(http.conn.dst_ip[*] in $%s)", replace(cloudflare_teams_list.k8s_apiserver_ips.id, "-", ""))
}

Since mutual TLS is used between clients and API servers, and traffic between kubectl and the API is not all HTTP, HTTP inspection is disabled for these connections. Additional Zero Trust rules—such as device attestation, session lifetimes, and user/group access policies—can be layered on for tighter security.

Deploying Tunnel Endpoints in Kubernetes

Tunnel endpoints are deployed as pods, which leverages Kubernetes deployment strategies for rolling upgrades and node failure handling. First, a ConfigMap is created with the minimal configuration for WARP routing using the tunnel ID.

apiVersion: v1
kind: ConfigMap
metadata:
  name: tunnel-zt
  namespace: example
  labels:
    tunnel: api-zt
data:
  config.yaml: |
    tunnel: 8e343b13-a087-48ea-825f-9783931ff2a5
    credentials-file: /opt/zt/creds/creds.json
    metrics: 0.0.0.0:8081
    warp-routing:
        enabled: true

The tunnel ID can be retrieved from the configuration management system, the Zero Trust dashboard, or by running:

cloudflared tunnel list

Next, a secret holds the tunnel credentials. Ideally this would come from a secret management system, but a direct creation works for simplicity.

jq -cn --arg accountTag $CF_ACCOUNT_TAG \
       --arg tunnelID $CF_TUNNEL_ID \
       --arg tunnelName $CF_TUNNEL_NAME \
       --arg tunnelSecret $CF_TUNNEL_SECRET \
   '{AccountTag: $accountTag, TunnelID: $tunnelID, TunnelName: $tunnelName, TunnelSecret: $tunnelSecret}' | \
kubectl create secret generic -n example tunnel-creds --from-file=creds.json=/dev/stdin

This creates a tunnel-creds secret in the example namespace containing the credentials file the tunnel expects. Multiple tunnel replicas are deployed to maintain availability during node drains.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    tunnel: api-zt
  name: tunnel-api-zt
  namespace: example
spec:
  replicas: 3
  selector:
    matchLabels:
      tunnel: api-zt
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
  template:
    metadata:
      labels:
        tunnel: api-zt
    spec:
      containers:
        - args:
            - tunnel
            - --config
            - /opt/zt/config/config.yaml
            - run
          env:
            - name: GOMAXPROCS
              value: "2"
            - name: TZ
              value: UTC
          image: cloudflare/cloudflared:2022.5.3
          livenessProbe:
            failureThreshold: 1
            httpGet:
              path: /ready
              port: 8081
            initialDelaySeconds: 10
            periodSeconds: 10
          name: tunnel
          ports:
            - containerPort: 8081
              name: http-metrics
          resources:
            limits:
              cpu: "1"
              memory: 100Mi
          volumeMounts:
            - mountPath: /opt/zt/config
              name: config
              readOnly: true
            - mountPath: /opt/zt/creds
              name: creds
              readOnly: true
      volumes:
        - secret:
            name: tunnel-creds
          name: creds
        - configMap:
            name: tunnel-api-zt
          name: config