A single filesystem for the data center

Facebook’s storage infrastructure used to be a patchwork of purpose-built systems. Each of those systems had its own cluster instances, which scaled to tens of petabytes at most. As the company’s storage demands grew, that arrangement became increasingly inefficient and costly to operate. Tectonic replaces that patchwork with a single distributed filesystem designed to scale to exabytes per cluster.

The move to Tectonic changes the operational model in two fundamental ways. First, resource efficiency improves because storage that would otherwise be stranded across many small clusters is now pooled into one large, shared system. Second, operations are simpler because there is one system and far fewer clusters to manage.

Three design problems

Building a single exabyte-scale filesystem required solving three problems that earlier systems could sidestep:

  • Exabyte-scale clusters. Tectonic separates filesystem metadata into independently scalable layers. Each metadata layer is hash-partitioned across a scalable shared key-value store. A linearly scalable storage node layer sits underneath, allowing one Tectonic cluster to serve the storage needs of an entire data center.
  • Performance isolation. Tectonic does not try to isolate performance between hundreds of individual applications. Instead, it groups applications with similar traffic patterns and latency needs into a smaller set of traffic groups. Resource management happens between those groups, which is a tractable problem.
  • Tenant-specific optimizations. To match the performance of the specialized systems it replaces, Tectonic uses a client-driven microservice architecture. Rich client-side configuration controls how each tenant interacts with the filesystem, enabling application-specific behavior without changing the core server design.

The operational payoff

Storage is a dependency for nearly every large cloud service, and increasing demand pressures the scalability of individual clusters. Tectonic addresses that bottleneck directly by letting one cluster grow far beyond the limits of the older systems.

Adopting Tectonic for Facebook’s data warehouse cut the number of warehouse clusters by roughly 10x, which both unstranded storage resources and reduced operational overhead. The consolidated system delivers performance comparable to, and in some cases better than, the specialized systems it replaced.

Paper: Facebook’s Tectonic Filesystem: Efficiency from Exascale (USENIX FAST ’21)