Why content distribution needed a rethink
Meta's private cloud relies on distributing large objects—executables, code artifacts, AI models, and search indexes—to millions of client processes. These workloads vary along three dimensions: fanout (a handful of readers to millions), object size (roughly 1 MB to a few TB), and read patterns (bursts of seconds or spread over hours). The requirements are strict: distribution must be fast enough to be bounded by a host's network or storage write bandwidth, efficient in terms of servers, network bytes, and client resources, and reliable enough to meet latency SLAs. Operational ease matters too—engineers need visibility into the health of each client and simple knobs to restore service when things degrade. Prior to Owl, Meta had grown at least three separate distribution systems—for executables, models, and indexes—each tailored to its domain. None of them satisfied all requirements simultaneously, and two root causes emerged: the wrong balance between centralization and decentralization, and a lack of flexibility to serve Meta's diverse consumer services. ## Centralization vs. decentralization: a failed trade-off Meta's first attempt used hierarchical caching. Clients fetched from first-level caches on remote hosts, which on a miss read from further cache layers, ultimately backed by distributed storage. This was inefficient and hard to scale. The hierarchy demanded many dedicated hosts, and load spikes from hot content constantly throttled readers—quota provisioning for transient bursts was a persistent problem. The alternative was highly decentralized systems: a location-aware BitTorrent variant and a static peer-to-peer tree based on consistent hashing. Here, any process that wanted data acted as a peer, and millions did. These scaled far better, but each peer made decisions from local information, leading to poor resource efficiency and tail latency. Peers independently cached, so the collective could hold too many or too few copies of an object. Operations were equally painful: engineers couldn't get a systemwide view without aggregating state from a huge number of peers, each with limited visibility. Owl's design takes the middle path: a decentralized data plane where data streams from sources to clients via distribution trees, paired with a centralized control plane that builds those trees. The trees are ephemeral and per-chunk, lasting only as long as a chunk transfer. This embodies a mechanism-policy split. ## The mechanism-policy split Peers are intentionally simple. They provide only mechanisms: download a chunk from a source, cache or evict a chunk, serve cached data to other peers. All decisions—where to fetch from, how to retry, what to cache—are made by trackers, the handful of services that form the control plane. Borrowing BitTorrent terminology, trackers maintain complete state: which peers are downloading which chunks, what each peer caches, and each peer's location (host, rack, region). That fine-grained state lets trackers make placement decisions that minimize network hops and maximize cache hits. Centralization also transformed operations. Because a tracker with a consistent view of distribution state makes the decisions, engineers can understand exactly why availability dropped, latency spiked, or cache hit rate fell. Configuration changes propagate to trackers in seconds, and new policies roll out without updating a single peer. ## Flexibility as a first-class requirement The other shortcoming of prior systems was rigidity. Meta's clients differ wildly in available resources: some can dedicate gigabytes of memory or disk to peer caching, others none. Access patterns and scale vary, and so do objectives—low latency for some, reduced external storage load for others. Each earlier solution optimized for a subset of use cases, so any unified replacement had to avoid regressing any client's key metrics. Owl treats customization as core, not an afterthought. Trackers implement modular interfaces for caching and fetching policies, and each policy is itself configurable. Meta uses trace-driven emulation to explore the policy space and pick the best configurations per observed workload. The result: 106 distinct client types are supported, 55 with customized policies, without a one-size-fits-all compromise. ## How Owl fits together Owl's components are two: peer libraries linked into every binary that downloads data, and trackers—dedicated services that manage the control plane for a group of peers. Peer libraries provide a simple download API: a client specifies a unique object identifier and a byte range, optionally adding a deadline and integrity-check or decryption classes. Peers cache chunks in memory or on disk, sharing those caches with the client binary when data is read-only, and serve other peers' requests. Policies typically prefer peer-to-peer fetches over external sources. Trackers manage download state and are grouped by region, with 3–4 per region for scale and redundancy. They are multitenant, though binary distribution runs on a separate tracker set to guarantee performance isolation. Peers register with a random tracker via RPC and pick a new one if their current association fails. Each peer is tied to a bucket identifying its client binary type. That bucket enables per-client customization as well as individual monitoring of usage, performance, and reliability. Production adoption has been rapid: Owl has over 10 million unique clients, distributes up to 800 petabytes of data daily, and improved cache hit rate and download speeds by 2–3x over BitTorrent and prior systems. The full design and deployment lessons appear in the OSDI '22 paper, but the core principle is simple: keep the data plane decentralized for scalability, centralize the policy decisions for control, and make customization the default rather than the exception.
