Vetting Trino Changes Without Grinding the Cluster to a Halt

Shopify’s data scientists run queries against a Trino cluster that handles over 500 Gbps of data, with a target of p95 query results in five seconds or less. The infrastructure behind that is a fork of Trino scaled to hundreds of nodes and tens of thousands of virtual CPUs. Because the team runs a fork, they’re sometimes the first large organization to test new code "in the field," especially when contributing code back upstream.

Every change to that cluster—whether an optimization experiment, a feature update, or a security patch—carries risk. A bad change stalls data scientists and forces engineers to manually roll back Trino to a working version. The Data Reliability team decided that unstructured vetting was no longer sustainable and built custom tooling to minimize the risk of changes at scale.

Two Problems, One Library

Cluster management at this scale comes down to two concerns: optimizations and software updates. Both require constant vetting, and both are made more complicated by running a fork. The team iterated on several simulation prototypes before landing on a solution:

  • A tool to replay previously run queries from logs in an isolated environment.
  • A tool that replicates traffic from a previous work day.
  • Exploration of an official benchmarking framework for controlled simulations.

What was missing was structure. Manual checks lived in engineers’ heads, were undocumented, and some caused significant toil to complete. The team consolidated everything into a single lightweight Python framework with three use cases: verification, benchmarking, and profiling.

The core library is a set of classes for Trino query orchestration—essentially an API shared across all testing needs. The higher-level Library class handles connections, query states, and multithreaded execution or cancellation of queries. The Query class manages lower-level concerns like query annotations, safety checks, and fetching individual results. The library builds on the open source trino-python-client, which implements the Python Database API Specification for Trino.

Verification: PyTest as a Trino Test Harness

Verification means simple checks that Trino still works as expected after a change. The team wanted to accelerate the deployment cycle, and since future users are data platform engineers who likely know Python, they built a PyTest interface on top of the query orchestration library. That interface abstracts away Trino complications into test fixtures that initialize each test repeatably and handle cleanup.

Tests are marked so users can run groups like "correctness" or "performance," or exclude "production_only" tests. Correctness tests expect an exact set of rows back from a given query. A developer writes a familiar pattern:

@pytest.mark.correctness
def test_something(candidate_cluster):
    result = candidate_cluster.run_query("SELECT ...")
    assert result == expected_rows

Here, the candidate_cluster fixture creates the connection, executes the query, fetches results, and closes the connection—so developers can focus on the logic of the test itself.

For performance verification, the pattern relies on a multi_cluster fixture that runs the same query on two separate clusters at the same time: the candidate and a standby control. Assertions compare simple thresholds like execution time, which is enough to verify that performance isn’t negatively impacted. This pattern was used to verify internal Trino User Defined Functions (used in domains such as finance and privacy) and to assess a candidate cluster’s proposed storage caching layer. In the caching case, the suite couldn’t assert on performance automatically yet, so an engineer evaluated results with heuristics after kicking off tests.

Containerization and scheduled runs are on the roadmap, which would let the team run verification at regular intervals and make decisions from the results.

Benchmarking: TPC-DS Without a Separate System

Benchmarking evaluates a change under standardized conditions. Rather than standing up a full benchmarking infrastructure, the team keeps a lightweight suite that runs TPC-DS queries on sample datasets relevant to their business. Trino generates these datasets deterministically and makes them accessible via its TPCDS connector.

Benchmarking queries are parametrized to run at multiple scale factors (sf). Repeating the same test on larger datasets applies more load: sf10 is a 10 GB database, while sf1000 is 1 TB. PyTest works well for this case too:

@pytest.mark.performance
@pytest.mark.parametrize("sf", [10, 100, 1000])
def test_tpcds_query(candidate_cluster, sf):
    result = candidate_cluster.run_query(f"SELECT ... FROM ... WHERE ...", scale_factor=sf)
    assert result.query_time <= threshold

This replaces improvised methodologies—some engineers used Spark or Jupyter Notebooks, others manually executed queries with SQL consoles. The resulting inconsistencies defeated the purpose of benchmarking. The team acknowledges that more advanced benchmarking frameworks exist, but their architectures are more time-consuming to set up. For a limited set of simple performance checks, the lightweight suite suffices.

Profiling: Background Queries for Scale

Profiling goes deeper, instrumenting specific scenarios to optimize how Trino handles them. The core library supports this with what the team calls background queries—queries executed without fetching individual results. Launching hundreds of parallel queries that return millions of rows would quickly exhaust memory on laptops or external clients. Background queries let the team push the cluster into overdrive and profile at a much larger scale.

The prototype simulation code was formalized into library functions. generate_traffic is called with a custom profile to target specific behavior, while replay_queries plays back queries in real time against a modified cluster. These methodologies cover edge cases that standard benchmark tests miss.

This profiling approach was used to evaluate an auto-scaling configuration for cloud resources during peak and off-hours. Most queries happen between 9-5 PM EST even though data scientists are distributed globally, so the cluster is overprovisioned outside those hours. An engineer experimented with Kubernetes’ horizontal pod autoscaling, generating simulated traffic to watch how the count of Trino workers adjusted to different load patterns like "high to low" and "low to high."

Outcome: A Stable Upgrade, No Spin-Out

The tooling is a platform effort supported by multiple teams. The Data Foundations team used the framework to write an extensive series of correctness tests in preparation for the next Trino upgrade. Those tests surfaced issues that were resolved before rollout, and the upgrade was successful. P95 query execution time remained stable through the upgrade window—maintaining speed while avoiding the crash that a poorly vetted change could have caused.

Lessons from Building Performance Tooling

The custom verification suite gave the team a clear view of which experimental changes were worth shipping. Storage caching and traffic-based autoscaling, for example, could be evaluated with real evidence rather than intuition. But building the tooling also made it clear that performance testing has its own set of pitfalls.

  1. A solid statistics foundation is crucial. Teams need a shared vocabulary for interpreting numbers and calculating service level indicators, otherwise reports become points of debate rather than decision-making tools.
  2. Environment nuances can skew results. Differences between production and development setups, along with usage patterns, can unintentionally influence measurements. Deep system knowledge is required to identify and account for these variables.
  3. Relevant data is rarely available immediately. Resource usage metrics often arrive late, making automation, containerization, and scheduling necessary for collecting complete datasets.

The team ultimately narrowed the project's scope to verification, deferring more advanced profiling and benchmarking ambitions. The framework was designed to be extensible and the supporting library modular, leaving room for those interfaces to be added later without reworking the foundation.

Planned Next Steps

The tool is scheduled for gameday exercises ahead of Black Friday and Cyber Monday, when the Data Reliability team needs business-critical metrics on demand. Those scenarios will be the impetus for formalizing repeatable load and stress tests, similar to how Shopify tests the broader platform.

The team is also evaluating whether the suite can be open-sourced to benefit the wider community.