From problem statement to production in ten weeks
Cloudflare sits between customers' origin servers and their end users, which means every request to an origin runs through Cloudflare's infrastructure. When an origin server becomes unreachable and no cached version is available, visitors see a 521 error. For customers without their own monitoring systems, that failure can go unnoticed until users start complaining.
The 2019 summer internship team — three interns total, two in engineering and one in product management — turned that visibility problem into a product feature over the course of ten weeks. The result was passive origin monitoring, now available on all Cloudflare plans. When an origin becomes unreachable, Cloudflare sends the customer a notification so they can take action to restore service.
Making the feature work at scale
Analyzing data from all requests across Cloudflare's millions of Internet properties to detect unreachable origins wasn't feasible from a data volume and performance perspective. The team had to figure out what datasets were available for the error patterns they were looking for, then adapt their whiteboarded algorithm ideas to work with that data.
Alert quality for low-traffic sites
Because only a fraction of requests show up in the sampled timing and error dataset the team chose, false positives and negatives were disproportionately likely for low-traffic sites — the exact sites least likely to have sophisticated monitoring systems in place and most in need of this feature. The team analyzed patterns of failed requests across different types of Cloudflare Internet properties to set thresholds that would maximize true positive notifications without spamming customers about sporadic failures.
Notification design for a broad audience
Cloudflare serves everyone from personal blogs to large enterprises with global incident response teams. To ensure the notifications were actionable across that range of technical backgrounds, the team enabled the feature for small samples of customers and tested many variations of the origin monitoring email. Customer responses, support questions, and community forum posts all fed into refining the message for clarity and actionability.
The team frontloaded research, digging into request data and support tickets about origin unreachability, then spent the remaining weeks iterating. Passive origin monitoring was enabled for customers with time left in the internship, letting the team improve supportability, document design decisions, and work with the team taking over ownership. The work also produced smaller internal capabilities, such as notifications on origin outage events for larger sites to help account teams provide proactive support.
What made the internship work
Several factors contributed to a productive summer:
- A well-scoped project. Managers and mentors identified a problem that was meaningful but still achievable in the time available, keeping technical and product scope from creeping beyond what was realistic.
- A team of interns. The three interns worked as a dedicated team with similar goals, which helped keep the project focused and collaboration natural.
- Quick, agile cycles. The team was distributed across two offices (Champaign and San Francisco) and faced strict time constraints, so daily standups, weekly planning, and frequent feedback from customers and internal stakeholders kept things moving.
- Mentorship with room to explore. Managers challenged the team but also gave them space to develop their own work process, which encouraged ambitious goals.
After the internship
In the final week, the engineering interns visited the San Francisco office to meet the team taking over the project and present at the company all-hands — a meeting that wound up being preempted by the co-founders' announcement of the public S-1 filing.
In the months that followed, Cloudflare added a notifications page for easy configurability and announced passive origin monitoring alongside other tools to help customers monitor servers and avoid downtime.
