Summer at the Edge: Implementing New TLS Drafts in Production
In the summer of 2017, I joined Cloudflare as an intern, coming from a graduate student background in automorphic forms and computational Langlands. My academic work had touched on computer security research and publishing papers, but building and deploying new protocols to production systems was uncharted territory. The shift from solitary, long-horizon academic projects to collaborative, quarter-driven industrial work was a significant adjustment.
Cloudflare then was a far smaller operation. Products like Argo, Access, Workers, and Quicksilver were either nascent or in heavy development, and offices in Lisbon and Austin didn’t exist yet. My project sat at the intersection of three Internet drafts: external authentication for TLS, secondary certificates for HTTP/2, and the HTTP/2 ORIGIN frame.
The core idea was to let a server prove, on an already established connection, that it can serve resources for other hostnames it proxies. This would allow browsers to reuse existing connections for subresources like CSS, JavaScript, and images, avoiding the latency of establishing new ones. External authentication and secondary certificates also had broader use cases, such as enabling certificate-based authentication over HTTP/2 and TLS 1.3, so the implementation had to account for the full scope of the drafts, not just Cloudflare’s immediate needs.
The Development Environment Hurdle
My first task was getting a local development environment running. Despite promises of Mac and Linux support, only Mac was functional. It took three days of struggle before I conceded and bought one. With help from three engineers, we spent days hacking the edge dev stack into a workable state. That environment has since been replaced by one built on Kubernetes VMs, which is a significant improvement: when things work on your machine, you can now share that machine with everyone.
Building a Prototype
Since these drafts were incremental changes to massive existing stacks, I chose to build my prototype on the Go standard library for its readability and my familiarity with it. There was already a basic Firefox demo for the ORIGIN frame. In three weeks, I extended it into a demonstration server and client. This proved the specifications were implementable and revealed no blockers. However, we needed integration into real servers to uncover rare, showstopper issues—a lesson learned the hard way during TLS 1.3 development, which was delayed by months due to an incompatibility with a specific brand of printer.
Moving to Production
To collect real-world performance data that could convince browsers and other implementers, we had to integrate this into production. At the time, Cloudflare’s edge was built on customized NGINX, with business logic in Lua via OpenResty. The codebase was a monolith—TLS termination and core logic lived in the same repo even though they ran as separate processes—making it hard to track what ran when. Additionally, Lua offered no way to interact with the system on a per-request basis.
The ORIGIN frame came first. We already had logic to determine which sites hosted a page’s subresources, as it was used for server push via Link headers, so this was a straightforward build.
The CERTIFICATE frame was far more complex. NGINX processes requests in phases, and the header-processing phase prohibits network I/O to avoid blocking the event loop. But that’s exactly where we needed to parse headers, fetch a certificate, and ask Keyless to sign it. The standard workaround was to use a Lua timer callback, where network I/O is allowed, but that context lacks any request data. This required significant refactoring to let the Keyless module function outside a request’s scope.
Even after generating a signature, we faced the problem of associating the resulting CERTIFICATE frame with the original connection. The request might no longer be alive, and its state was unknown after Keyless handled it. My solution was a shared btree indexed by a number, where the request could signal it was ready to send the frame, and Keyless could deposit the frame data. Whichever side finished second would enqueue the frame for transmission.
It was a hacky integration with a legacy module that assumed access to request data, opening up potential for subtle bugs. The prototype client worked against it, but a CI configuration mistake when I pushed the commit broke every other branch’s build. The timing—Friday afternoon Pacific, with the SSL server team entirely in London—didn’t help, but the issue was fixed Monday morning. We found the root cause was a CI flaw, not my code.
Despite a side-by-side effort with a colleague from London to get a demo running on a test site, we ran out of time tracking down a production bug. My internship ended with us not able to fully integrate the feature.
Outcome
The project was ultimately shelved. The risk of such intrusive changes for an experimental feature outweighed the perceived benefits, especially with little prospect of browser support for the savings to materialize. There were also nontechnical standardization hurdles: bypassing DNS for traffic direction complicates network debugging, and there were concerns about certificate misissuance risks.
Although the project didn’t ship, it was a valuable education in large-scale software collaboration, version control, and communication with fellow implementers. The experience of turning research into practical reality confirmed my decision to pursue industrial research. I’ve since returned to Cloudflare full-time. The drafts have continued to evolve, and we’ve contributed parts of my code as a starting point for other implementers. If we knew every project would succeed, it wouldn’t be ambitious enough to warrant research.
