Origin Failures Meet an Old-School Safety Net

When an origin server goes down, visitors see errors and website owners lose traffic. Cloudflare estimates its users collectively wait through the equivalent of over four and a half years every day for unreachable origins to respond. To reduce that friction, Cloudflare is partnering with the Internet Archive to rebuild its Always Online service around the Wayback Machine's web archives.

Always Online acts as insurance for websites. If a customer's origin goes offline, times out, or otherwise fails, Cloudflare steps in to serve archived copies of pages to visitors. The Internet Archive, a nonprofit that runs the Wayback Machine, has saved snapshots of billions of web pages. By linking the two services, Cloudflare can use the Archive's stored content to respond to requests for unavailable sites, while the Archive gains new sources for its mission of preserving the web.

When a user enables Always Online in their Cloudflare dashboard, the service can share the site's hostname with the Wayback Machine so it can be archived. If the origin goes down, Cloudflare retrieves the most recent archived version from the Internet Archive and shows it to visitors.

How a Request Reaches the Edge

When someone visits a Cloudflare-protected site, their request first hits Cloudflare's edge. The edge serves cached content if available; for uncached or expired content, it fetches a fresh copy from the origin and updates the cache for future requests. If the origin can't be reached, the edge retries several times before marking the origin as down and showing an error. That error page is a last resort, and Cloudflare works to minimize how often visitors see it.

The original Always Online feature emerged ten years ago, when many Cloudflare customers ran on hosts with frequent downtime. These customers worried that a search engine crawler visiting during an outage would record the site as unreachable, hurting its search ranking. The feature was built to let archived versions of pages keep showing content during these windows.

Over the years, the service has gained one important constant: a visible banner telling visitors they're viewing an archived version because the origin is unreachable. Both site owners and visitors have responded well to that transparency. Cloudflare recently aimed to improve on the concept — keeping it seamless while expanding how much content is available and how fresh it is.

What the Internet Archive Brings

The Internet Archive traces its mission to providing universal access to all knowledge. Since 1996, the Wayback Machine has preserved much of the public web that would otherwise be lost, holding over 468 billion archived web pages — more than 45 petabytes of data.

The integration goes in both directions. Domains that opt into Always Online may not have been discovered by the Archive's normal crawler, giving the Archive a wider footprint of the web. For Cloudflare's customers, the Archive's stored content gives visitors something to read rather than an error page.

"Through our partnership with Cloudflare, we are learning about, and archiving, webpages we might not have otherwise known about, and by integrating with Cloudflare's Always Online service, archives of those pages are available to people trying to access them if they become unavailable via the live web" — Mark Graham, Director of the Internet Archive's Wayback Machine

"We are excited to work with Cloudflare and expect this partnership to bring important redundancy to the Internet and allow for us to advance our ongoing efforts to make the Internet more useful and reliable." — Brewster Kahle, Founder and Digital Librarian of the Internet Archive

How the New Service Responds to Outages

Enabling the updated Always Online in the Cloudflare dashboard allows Cloudflare to share basic site information — like hostname and popular URLs — with the Internet Archive so it can crawl and archive at regular intervals. This keeps content ready for Always Online and enriches the Archive's collection.

When an origin fails to respond, Cloudflare's edge first returns a status code in the 520-to-527 range, indicating an origin connection issue. Cloudflare then checks the local edge datacenter for stale or expired content that can still be served. If nothing is available there, it queries the Internet Archive for the most recently archived version of the page. Visitors receive that content with a banner letting them know the origin is having problems and offering a one-click check to see if the site is back online.

Some content won't benefit from the archive. Dynamic features that require live communication with the origin — shopping carts, web applications — will still show errors. But for basic pages, visitors will often find something useful.

Getting Started

For now, the original Always Online service remains available, but Cloudflare expects to make the Internet Archive-backed version the default soon. Customers can enable Always Online directly in the Cloudflare dashboard.

  • Details on how Always Online works are in the Cloudflare documentation.
  • Log in to the Cloudflare dashboard and toggle Always Online on to start using the service.
  • The Internet Archive has published its own announcement of the partnership on its blog.