Four Operational Lessons From Running Presto at Meta

Meta has run the open source SQL query engine Presto internally for the past ten years. Over that period, the fleet has grown to span data centers in multiple regions, and the operational playbook had to evolve with it. Scaling a query engine across thousands of machines surfaces problems that small deployments never see. Here are four areas where Meta had to build new tooling and processes, plus advice for teams planning their own Presto rollout.

Rolling Out New Versions Without Dropping Queries

Meta builds and prepares a new Presto release for deployment at least once a month, sometimes twice. Early on, pushing those releases to a large cluster fleet while maintaining consistent availability was a major hurdle. The issue was most acute for interactive use cases, where a user waits on a live query. Batch workloads were less sensitive since automatic retries absorb failures.

The solution relies on the Gateway, the load balancer that sits in front of all Presto clusters and routes queries to them. Before a cluster is updated, it is marked as drained so the Gateway stops sending it new queries. Automated tooling then waits a predetermined period for in-flight queries to complete before updating the cluster. Once the cluster is back online, it is re-registered with the Gateway.

Availability during the rollout is handled the same way: automation verifies that every data center retains a sufficient number of active clusters throughout the process. The remaining trade-off is tuning how many clusters are taken down simultaneously, since too many at once risks availability while too few makes deployments drag.

Automating Cluster Lifecycle Management

The data warehouse's distribution across regions is continuously shifting, which means Presto clusters are regularly created and decommissioned. When the fleet was small, this was a manual process. That became unmanageable as the footprint grew, so Meta standardized the configurations first, establishing base templates per use case. Individual clusters then carry only a minimal set of overrides on top of the base config.

Spin-up is now fully automated: the system generates configurations from templates, integrates with company-wide infrastructure services, and runs test queries against the new cluster once it comes online. Only after verifying those queries succeed does automation register the cluster with the Gateway.

Decommissioning is the reverse process: a cluster is de-registered, running queries finish, and the Presto processes are shut down and configs deleted. The entire flow, from hardware arriving at a data center to queries being served, runs without manual intervention. Meta credits this automation with saving person-hours, reducing hardware idle time, and eliminating a class of human errors.

Detecting and Remediating Host-Level Failures

Given the fleet's size, tooling to support the oncall engineer is essential. Meta built a set of "analyzers" that trigger when monitoring systems detect breaches of customer-facing SLAs. These analyzers pull data from multiple sources, including operational data stores, Scuba events, and host-level logs, then apply custom logic to identify the probable root cause. Often the oncall gets root cause analysis directly, with a path to mitigation options. In some cases, both the diagnosis and remediation are fully automated.

A concrete example is bad host detection. Meta noticed that certain hosts generated a disproportionate share of query failures. Investigations surfaced two primary causes: hardware problems not yet caught by fleet-wide monitoring, and obscure JVM bugs leading to a steady drip of errors. Now the system monitors query failures and attributes each one to the responsible host where possible. Alerts fire when a host's failure count spikes, and automation drains the affected host from the fleet automatically.

Queueing issues also benefit from the analyzer approach. Presto clusters queue queries once they hit maximum concurrency, which depends on use case, hardware, and query size. Routing decisions involve multiple systems beyond Presto that weigh queue depth, hardware distribution across data centers, and table data locality. The combination makes manual debugging difficult, so analyzers pull all those signals together and produce a clear picture of the route cause.

Making the Gateway Resilient to Traffic Spikes

The Gateway routes every Presto query at Meta, so its failure means Presto is down for everyone. Early versions of the Gateway were simple, and as internal usage scaled up, it buckled under heavy load. A notable failure scenario was one misbehaving service bombarding the Gateway with millions of queries in a short window, crashing the processes and halting routing entirely.

Meta responded with two main enhancements. The first is a throttling feature that rejects queries when the Gateway is under heavy load. Throttling can be keyed per user, per source, per IP, or globally. The second is autoscaling, which uses a Meta-wide service to add Gateway instances dynamically under load, preventing CPU and memory exhaustion that led to the earlier crashes. Together, these changes allow the Gateway to absorb unpredictable, DDoS-style traffic patterns.

Advice for Scaling Presto

Teams planning to scale their own Presto deployment should consider the following:

  • Define clean SLAs early. Customer-facing SLAs should track user pain points with obvious metrics like queueing time and failure rate. Without well-defined SLAs, incident impact is ambiguous and mitigation efforts get bogged down in confusion when the user base is large.
  • Invest in monitoring and automated debugging. Thorough monitoring catches production issues before they spread, but manual investigation does not scale. Automated analyzers are the only sane way to determine root cause quickly once incidents are customer-impacting.
  • Get load balancing right. The load balancer in front of Presto clusters matters more as the fleet grows. Small inefficiencies become outsized problems at massive query volumes.
  • Plan configuration management. A large fleet makes configuration drift painful. Hot-reloadable configs let engineering adjust settings without restarting instances, which would cause query failures and unhappy users.