Go and Proxy Servers: Part 2 - HTTPS Proxies - Eli Bendersky's website

Why HTTPS changes the proxy picture

An HTTP proxy can inspect and forward requests because the traffic is plaintext. HTTPS breaks that assumption: an HTTPS client expects a valid certificate for the exact server it is addressing. A generic forward proxy has no such certificate, so it cannot terminate the TLS session and relay the request as it would with HTTP.

Proxies handle this in three ways:

• TLS termination: the proxy acts as the server, holding valid certificates for the domain. This is the model for reverse proxies sitting in front of backend services.
• Blind tunneling: the proxy establishes a raw TCP tunnel between client and destination and forwards bytes in both directions without inspection.
• Inspection tunneling: the proxy tunnels traffic while also decrypting and potentially modifying it.

The first two are essential and covered here; the third is a more advanced scenario.

TLS-terminating reverse proxy

A reverse proxy that terminates TLS is structurally the same as a plain HTTP reverse proxy, except it listens on an HTTPS port and therefore needs a certificate trusted by the client. For local development, generate one for localhost with mkcert and point the proxy at that certificate and key pair. The proxy forwards the decrypted requests to a backend server over HTTP.

package main

import (
	"log"
	"net/http"
	"net/http/httputil"
	"net/url"
)

func main() {
	// ...
	proxy := httputil.NewSingleHostReverseProxy(&url.URL{
		Scheme: "http",
		Host:   "localhost:8080",
	})
	server := &http.Server{
		Addr:    ":8089",
		Handler: proxy,
	}
	log.Fatal(server.ListenAndServeTLS("localhost.pem", "localhost-key.pem"))
}

With the backend debug server on port 8080, requests to the proxy arrive as HTTPS and are relayed to the backend as plain HTTP. This setup is how most non-trivial services are exposed: a production deployment would have the proxy bound to port 443 with a valid certificate for the public domain, and it would forward to internal backends that may themselves speak HTTP or use private certificates. Tools like Caddy and Nginx provide this behavior out of the box.

The CONNECT method for forward proxies

Forward proxies need a different mechanism because they don't own the target domain's certificate. The IETF solution is the HTTP CONNECT method, described in RFC 7231:

The CONNECT method requests that the recipient establish a tunnel to the destination origin server identified by the request-target and, if successful, thereafter restrict its behavior to blind forwarding of packets, in both directions, until the tunnel is closed.

CONNECT is only intended for proxy requests. The sequence is:

1. The client sends CONNECT host:port HTTP/1.1 to the proxy.
2. The proxy opens a TCP connection to that destination.
3. On success, it returns HTTP/1.1 200 OK to the client.
4. From that point on, the proxy blind-forwards TCP traffic between both sides.
5. It closes the tunnel when either side closes its connection.

A Go implementation requires only a handler that distinguishes CONNECT requests from others and then manages the two raw TCP connections.

func (p *forwardProxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	if r.Method == http.MethodConnect {
		p.proxyConnect(w, r)
		return
	}
	http.Error(w, "this proxy only supports CONNECT", http.StatusMethodNotAllowed)
}

The core function reads the destination address from the request, dials a TCP connection, then hijacks the client's HTTP connection. Hijack gives access to the underlying TCP connection, which the net/http package otherwise encapsulates. Once hijacked, you are responsible for closing it yourself.

func (p *forwardProxy) proxyConnect(w http.ResponseWriter, r *http.Request) {
	dstConn, err := net.Dial("tcp", r.Host)
	if err != nil {
		http.Error(w, err.Error(), http.StatusBadRequest)
		return
	}
	srcConn, _, err := w.(http.Hijacker).Hijack()
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

	_, err = srcConn.Write([]byte("HTTP/1.1 200 OK\r\n\r\n"))
	if err != nil {
		srcConn.Close()
		dstConn.Close()
		return
	}

	tunnelConn(srcConn, dstConn)
}

The tunnel itself runs two goroutines, one per direction, copying bytes until either side closes:

func tunnelConn(src net.Conn, dst net.Conn) {
	defer src.Close()
	defer dst.Close()
	go func() {
		_, err := io.Copy(dst, src)
		if err != nil {
			return
		}
		dst.Close()
	}()
	io.Copy(src, dst)
}

Testing it with curl and https_proxy set to the proxy address produces the expected handshake: curl sends a CONNECT request, the proxy answers 200, and then TLS proceeds through the tunnel direct to the destination. Existing Go HTTP clients work with this model as well once configured with the appropriate proxy environment variable.

Intercepting HTTPS Traffic with MITM Proxies

Up to this point, we've assumed HTTPS proxies can't inspect the traffic flowing through them. That holds for modern TLS versions, but there's a critical loophole: trusted certificates. A man-in-the-middle (MITM) proxy exploits this by decrypting and even rewriting HTTPS traffic, given cooperation from the client machine's administrator.

This works through the TLS chain of trust. If an administrator installs a custom root certificate authority (CA) on a client machine, that machine will trust any certificate signed by that CA. An intercepting proxy holding the CA's private key can then mint fake certificates for any domain, allowing it to impersonate that domain to the client.

The process for a CONNECT request is straightforward:

1. The client machine trusts a custom CA (installed by an administrator or user). Browsers typically rely on built-in root CA lists, but these can be extended with system-level settings.
2. The proxy possesses the private key for that CA and can sign certificates with it.
3. When the proxy receives a CONNECT for domain.com, it generates a certificate for that domain, signed with the CA's key.
4. The proxy then communicates with the client as if it were domain.com, decrypting all traffic meant for that domain.
5. The proxy can inspect, modify, and forward traffic to the real domain.com over a separate TLS connection – and do the same for responses.

This capability isn't inherently malicious. Debugging proxies and organizational network policies are legitimate reasons to deploy them. If a company sets up such a proxy, individual users can't easily bypass it with blind tunneling. That said, the same setup means an organization could decrypt traffic to your bank's website if it chose to – the TLS is only as secure as the trusted CAs on the machine.

In some environments, this decryption isn't even done by an explicit proxy. Routers on the network path can implement the same interception trick, provided they hold private keys for a CA the client trusts. The user may never notice.

Building this kind of proxy in Go requires attention to a few details. A working example is available in the source code on GitHub. The main HTTP handler is straightforward:

// proxyConnect implements the MITM proxy for CONNECT tunnels.
func (p *mitmProxy) proxyConnect(w http.ResponseWriter, proxyReq *http.Request) {
  log.Printf("CONNECT requested to %v (from %v)", proxyReq.Host, proxyReq.RemoteAddr)

  // "Hijack" the client connection to get a TCP (or TLS) socket we can read
  // and write arbitrary data to/from.
  hj, ok := w.(http.Hijacker)
  if !ok {
    log.Fatal("http server doesn't support hijacking connection")
  }

  clientConn, _, err := hj.Hijack()
  if err != nil {
    log.Fatal("http hijacking failed")
  }

  // proxyReq.Host will hold the CONNECT target host, which will typically have
  // a port - e.g. example.org:443
  // To generate a fake certificate for example.org, we have to first split off
  // the host from the port.
  host, _, err := net.SplitHostPort(proxyReq.Host)
  if err != nil {
    log.Fatal("error splitting host/port:", err)
  }

  // Create a fake TLS certificate for the target host, signed by our CA. The
  // certificate will be valid for 10 days - this number can be changed.
  pemCert, pemKey := createCert([]string{host}, p.caCert, p.caKey, 240)
  tlsCert, err := tls.X509KeyPair(pemCert, pemKey)
  if err != nil {
    log.Fatal(err)
  }

  // Send an HTTP OK response back to the client; this initiates the CONNECT
  // tunnel. From this point on the client will assume it's connected directly
  // to the target.
  if _, err := clientConn.Write([]byte("HTTP/1.1 200 OK\r\n\r\n")); err != nil {
    log.Fatal("error writing status to client:", err)
  }

  // Configure a new TLS server, pointing it at the client connection, using
  // our certificate. This server will now pretend being the target.
  tlsConfig := &tls.Config{
    PreferServerCipherSuites: true,
    CurvePreferences:         []tls.CurveID{tls.X25519, tls.CurveP256},
    MinVersion:               tls.VersionTLS13,
    Certificates:             []tls.Certificate{tlsCert},
  }

  tlsConn := tls.Server(clientConn, tlsConfig)
  defer tlsConn.Close()

  // Create a buffered reader for the client connection; this is required to
  // use http package functions with this connection.
  connReader := bufio.NewReader(tlsConn)

  // Run the proxy in a loop until the client closes the connection.
  for {
    // Read an HTTP request from the client; the request is sent over TLS that
    // connReader is configured to serve. The read will run a TLS handshake in
    // the first invocation (we could also call tlsConn.Handshake explicitly
    // before the loop, but this isn't necessary).
    // Note that while the client believes it's talking across an encrypted
    // channel with the target, the proxy gets these requests in "plain text"
    // because of the MITM setup.
    r, err := http.ReadRequest(connReader)
    if err == io.EOF {
      break
    } else if err != nil {
      log.Fatal(err)
    }

    // We can dump the request; log it, modify it...
    if b, err := httputil.DumpRequest(r, false); err == nil {
      log.Printf("incoming request:\n%s\n", string(b))
    }

    // Take the original request and changes its destination to be forwarded
    // to the target server.
    changeRequestToTarget(r, proxyReq.Host)

    // Send the request to the target server and log the response.
    resp, err := http.DefaultClient.Do(r)
    if err != nil {
      log.Fatal("error sending request to target:", err)
    }
    if b, err := httputil.DumpResponse(resp, false); err == nil {
      log.Printf("target response:\n%s\n", string(b))
    }
    defer resp.Body.Close()

    // Send the target server's response back to the client.
    if err := resp.Write(tlsConn); err != nil {
      log.Println("error writing response back:", err)
    }
  }
}

This particular proxy handles cases where the target domain is explicitly stated in the CONNECT request. It does not implement x509 SAN extensions or SNI support, though those could be added with relative ease. For deeper discussion of other complications, the mitmproxy documentation is a useful reference.

To run the proxy, you need to provide the path to a certificate and private key for a CA that your client machine already trusts. If you've used mkcert with the -install flag, you can locate these files via mkcert -CAROOT. The invocation will look something like this (exact paths vary by system):

$ mkcert -CAROOT
/home/eliben/.local/share/mkcert

$ go run connect-mitm-proxy.go \
    -cacertfile /home/eliben/.local/share/mkcert/rootCA.pem \
    -cakeyfile /home/eliben/.local/share/mkcert/rootCA-key.pem
2022/11/08 20:49:02 loaded CA certificate and key; IsCA=true
2022/11/08 20:49:02 Starting proxy server on 127.0.0.1:9999