Refactoring an Internal Tool at Scale: What Cloudflare Learned From Project Crossbow
Cloudflare’s global network spans 200 cities across more than 90 countries. Engineers across product, technical support, and operations frequently need to debug network issues from specific locations or individual servers. Crossbow, an internal tool, was built to serve that need by letting Technical Support Engineers run diagnostic commands—like traceroutes, DNS queries, and cURL requests—and debug product features using specialized tooling.
In September 2021, Cloudflare’s Support Operations team took over ownership of Crossbow from a Product Engineering team. The tool had been a secondary focus for multiple teams and had changed hands without developing deep subject matter expertise. The Support Operations team, which builds diagnostic tooling and NLP technology for Technical Support Engineers, was deemed the right home for the project.
What followed was a large-scale refactor of the tool’s architecture, interface, and workflow. The effort produced several lessons about how to manage internal tooling transitions—some of which run counter to conventional wisdom.
Deprecating an Interface to Reduce Technical Debt
Crossbow originally had two interfaces: a CLI and a web UI embedded in Cloudflare’s internal support console. Maintaining both created significant overhead for any improvement effort. The team decided to consolidate to a single platform.
Before deprecating either interface, the team polled engineering, operations, solutions engineering, and technical support teams. The survey asked which option users preferred, which they considered necessary, and why. The responses revealed that preferences for the web UI largely stemmed from missing documentation and training. CLI users, however, considered the tool critical to their workflow—product engineering teams without routine access to the support UI needed Crossbow for their jobs, and users wanted to script commands via shell.
The technical considerations reinforced this choice. The UI was written in JavaScript with an API Gateway service that translated HTTP requests to gRPC, plus configuration for the support UI. The CLI interfaced directly with the gRPC API, making it a simpler system. Given the Support Operations team’s Systems Engineering focus and limited UI resources, deprecating the UI was the right call.
The team communicated the decision early, created new documentation, trained affected teams, rolled out an internal Crossbow user group, and retired the associated services. They also improved the CLI experience with clearer help information and easier usage.
Simplifying the Architecture with Pub/Sub
Crossbow’s original architecture was years old. A gRPC API ran commands at Cloudflare’s edge using a configuration management tool that the SRE team wanted to retire—Crossbow was its last user. During a visit to the Singapore office, the Edge SRE team reviewed the architecture and suggested refactoring ideas to improve global network stability and move away from the legacy pipeline.
The team adopted a simpler pub/sub pipeline. The edge network now exposes a gRPC daemon that listens for new jobs, executes them, and makes a callback to the API service with results, which are relayed to the client.
Authentication between the API service and the client, and between the API service and the network edge, uses JWT. For CLI users, authentication works by querying an HTTP endpoint behind Cloudflare Access using cloudflared, which provides a JWT for gRPC authentication:
- CLI makes request to authentication server using
cloudflared - Authentication server responds with signed JWT token
- CLI makes gRPC request with JWT authentication token to API service
- API service validates token using a public key
The gRPC API endpoint sits on Cloudflare Spectrum, and because users authenticate via Cloudflare Access, they no longer need to be on the company VPN. The new authentication pipeline and single interface also enabled better collection of usage metrics and logs.
Managing Risk in a Security-Sensitive Environment
For this project, the primary risk wasn't availability—Technical Support Engineers could escalate to the SRE team if the tool went down. The real concern was security of the Cloudflare network and ensuring Crossbow didn't impact other services. The team took methodical steps to improve isolation and engaged InfoSec early for specification and code reviews of the new pipeline. Any availability risks were clearly communicated to the support team and the internal user group.
Feedback Loops and Measured Improvements
The Support Operations team works in the style of Extreme Programming, with an emphasis on Test Driven Development—the "red-green-refactor" pattern: write tests that define requirements, make them pass, then refactor for code quality.
At the time of the transition, Cloudflare’s support and SRE teams were already working on Project Baton, an effort to let Technical Support Engineers handle more customer escalations without handoff to SRE. That project had produced a feature wish list for Crossbow. The Support Operations team tied JIRAs to every item, prioritized the work, and delivered features with a test-driven workflow and continuous integration.
The impact was measurable. Simple additions like support for MTR and exposing different cURL flags drove increased usage. The team also embedded Crossbow support for tools built by other teams for the network edge, letting those teams maintain their tools while exposing features to Crossbow users. An improved development environment and documentation encouraged Product Engineering teams to contribute functionality that served both their interests and those of the customer support team.
The team also folded several standalone tools into Crossbow—tools used to discover Cloudflare configuration for a given URL and for distributed performance testing—and deprecated them. A Cloudflare Workers team tool, Edge Worker Debug, was likewise rolled into Crossbow.
User analytics, implemented on December 16, 2019, showed a 4.5x increase in usage by the week ending January 22, 2020. That growth happened mostly within four weeks, as the most-wanted features reached critical mass among Technical Support Engineers.
Going forward, the team used the number of checks executed as a proxy for tool usefulness, since unique-user growth flattened once adoption matured. Even during a week with modest user growth, test execution rose 2.6x—coinciding with new high-usage functionality.
Two Counterintuitive Lessons
The first lesson: cutting functionality can drive usage, provided it's done deliberately. The web UI offered no features that the CLI lacked, yet it carried substantial maintenance overhead. Deprecating it reduced technical debt and sped up delivery of more important features. But this only worked because the decision-making process was transparent and those affected were consulted.
The second lesson: tool development often relies on anecdotal user feedback without objective measures of success. When logging is added, it's typically for security and audit purposes. But effective measurement drives future tooling decisions—usage data can ultimately be more valuable than the original feature that generated it. Feedback loops with users are essential, but they need to be paired with concrete metrics to truly guide what to build next.
