Scaling Muse: How Netflix Powers Data-Driven Creative Insights at Trillion-Row Scale

Rethinking the Serving Layer for Deeper Creative Analytics

Muse is Netflix’s internal platform for giving creative strategists and launch managers the data they need to judge promotional media — which artwork or clips resonate with audiences, and which assets risk reading as misleading. On the surface it sounds like a straightforward dashboard workload, but the underlying requirements have pushed Muse’s architecture through multiple generations. The original design paired batch Spark pipelines with a single Druid cluster delivering aggregate metrics. As feature demands grew — outlier detection, notifications, media playback, granular audience filters — the system was stretched to its limits, particularly around volatile multi-dimensional data.

The inflection point came with “audience” affinities: algorithmically derived labels grouping viewers by taste. Supporting queries like “Does this key art perform better with Character Drama fans than Pop Culture enthusiasts?” meant adding many-to-many audience relationships to an already massive impression and playback dataset. The combinatorial growth of that model was untenable, so the team re-architected the data serving layer around three pillars: HyperLogLog sketches for distinct counting, the Hollow in-memory store for precomputed aggregates, and a set of Druid tuning measures.

Distinct Counts Without the Full Dataset

Impressions (times an asset was shown) and qualified plays (playbacks past a duration threshold tied back to an impression) both require distinct user counts. In a distributed environment, exact distinct counting over long time ranges means reconciling profile IDs across days, months, even years of data — an escalating resource drain. The Apache Datasketches library provides a way out, returning estimates within roughly 1–2% error. Muse uses the logK precision parameter at 17, which yields approximately 0.8% error.

Sketches are built in two locations. At ingest, Druid applies the HLLSketchBuild aggregator with rollup enabled to shrink data before storage. Independently, the Spark ETL persists pre-aggregated all-time metrics as HLL sketches, daily merging new data using Spark’s hll_union and hll_union_agg functions. The strategy pays off broadly — across common OLAP query patterns the team reports roughly 50% latency reduction.

Still, approximate counts over large date ranges for high-traffic titles can exhaust Druid’s thread pool during concurrency peaks. To keep query volume off the cluster and reserve threads for interactive work, the team leaned on Hollow.

Caching Precomputed Aggregates In Memory

Hollow is Netflix’s framework for building compressed, in-memory read-only key/value data stores. The architecture is straightforward: producer servers listen for updates to Iceberg tables and push changes downstream. Consumer Spring Boot applications pick up announcements from producers and refresh their in-memory caches automatically.

In Muse’s data service, Hollow serves as the fast path for metrics that can be computed ahead of time — all-time impressions per asset, for instance — while Druid stays reserved for queries that require on-the-fly grouping or filtering that precomputation can’t satisfy.

Keeping Accuracy in View

Because HLL trades an exact count for an approximation, the migration wasn’t just a performance swap. To validate that pre- and post-change metrics aligned within acceptable bounds, the team relied on internal debugging tooling as part of the rollout, verifying that reported values stayed within the expected error envelopes.

The combination of sketch-backed ingest, lower-latency serving through Hollow, and tuned Druid query paths has let Muse support far more expensive analytical patterns without sacrificing the responsiveness users expect from a dashboard interacting with data on the order of trillions of rows per year.

Pre-computed Paths and Druid Tuning

The shift toward pre-computed aggregates opened up patterns that were previously hard to serve from the live Druid cluster. One example: fetching distinct dimension values per asset. A licensed title may only be available in specific countries, and retrieving those values used to mean a SELECT DISTINCT query against Druid. Now Muse maintains a feed of distinct dimension values that supports stream operations directly on a cached dataset.

/**
 * Returns the possible filter values for a dimension such as countries
 */
public List<Dimension> getDimensions(long movieId, String dimensionId) {
    // Access in-memory Hollow feed with near instant query time
    Map<String, List<Dimension>> dimensions = dimensionsHollowConsumer.lookup(movieId);
    return dimensions.getOrDefault(dimensionId, List.of()).stream()
        .sorted(Comparator.comparing(Dimension::getName))
        .toList();
}

This approach adds complexity — more intricate request routing and a higher memory footprint — but the stability and performance gains have been substantial. Query times for distinct dimensions dropped from hundreds of milliseconds to tens of milliseconds, and high-concurrency loads moved off Druid entirely. The same pre-computed aggregates also power recently launched title retrieval, all-time asset metrics, and several title metadata features.

Getting More From Druid

Keeping Druid performant remains an ongoing effort, aided by the presence of multiple Apache Druid PMC members on the Netflix engineering team. The optimizations that made the biggest difference:

• Broker-to-historical ratio: Increasing the broker count relative to historical nodes, aiming for the recommended 1:15 ratio to improve query throughput.
• Segment sizing: Targeting the 300–700 MB range via tuningConfig.targetRowsPerSegment during ingestion so each segment scanned by a single historical thread stays manageable.
• Lookups over joins: Using Druid lookups at query time for key column enrichment since joins are prohibitively expensive.
• Physical search predicates: Ensuring all search predicates run on physical rather than virtual columns, creating needed columns during ingestion with transformSpec.transforms.
• Lean ingestion: Applying filters in transformSpec.filter and removing unused columns via dimensionsSpec.dimensions to improve rollup yield.
• Multi-value dimensions: Leveraging this Druid feature to resolve the many-to-many combinatorial problem when combining audience filtering with grouping functionality.

Together these changes cut p99 Druid latencies by roughly half.

Validation and Controlled Rollout

Migrating the metrics system demanded careful attention to data integrity and user trust. The core strategy was a parallel deployment: the legacy and new metric stacks ran side-by-side within the Muse Data microservice, enabling continuous data quality checks, real-world performance monitoring, and fallback at any point.

Validation ran on two tracks. First, automated offline validation used Jupyter Notebooks to sample and compare key metrics across both stacks. The sampling set deliberately included recently accessed titles, high-profile launches, and edge-case titles with unusual handling requirements. This caught subtle discrepancies early and guided fixes such as tuning the HLL logK parameter and benchmarking end-to-end latency. Second, an in-app comparison tool displayed data from both stacks side-by-side for developers, automatically flagging significant differences for rapid triage.

Release risk was managed with several practices:

• Staggered implementation: The new stack was built and deployed by application segment — first by asset type like artwork or video, then by CEE phase (Explore, Exploit) — so issues could be isolated and resolved piece by piece.
• Shadow testing: Before end-user exposure, production traffic was mirrored asynchronously to the new implementation to validate real-world latency and catch faults without affecting the user experience.
• Granular feature flags: Fine-grained flags controlled exposure within each segment, allowing rollback or scope adjustment for specific user groups or titles at a moment's notice.

What the Migration Taught Us

The work pushed limits across the ETL layer, Druid, and the serving layer. Some choices — like building on Netflix’s in-house Hollow infrastructure — reflected what was available internally, but the transferable lessons are simpler: offload query volume where possible, filter rows and columns before Druid rollup, optimize search predicates, and apply HLL sketches where cardinality approximations are acceptable. Engineering discipline mattered just as much: producing side-by-side implementations and backwards-compatible changes allowed steady rollout without sacrificing validation rigor.

The foundation is now in place to expand support to content types like Live and Games, incorporate synopsis data, model how assets interact to influence member choices, and add metrics that distinguish effective promotional assets from authentic ones — all in service of helping members find content that resonates.