Netflix Stratum Media Processing: Automated Container Right-Sizing

Why manual sizing fails at Stratum's scale

Stratum schedules its functions as batch jobs on Titus, Netflix's container management platform. Every day it manages hundreds of thousands of containers across bursty, high-throughput and latency-sensitive workloads, spanning encoding, packaging, inspection and quality scoring. Function owners declare CPU, memory, disk and network limits in YAML, and those numbers tend to be wrong.

functionName: VideoEncoder.encodeV1
resources:
  numCpus: 8
  memoryInMB: 18000
  diskSizeInMB: 50000
  networkInMbps: 1000

Engineers usually copy a resource YAML from an existing function or size it for the worst case, then leave it alone until something breaks. Several forces push those stale values out of alignment: cloud hardware generations change, media standards and encoding algorithms advance (AV1, AV2, spatial audio), new media types such as video podcasts, sports, AI-adapted ad formats and broadcaster integrations like TF1 in France shift processing footprints unpredictably, and rising server and DRAM prices make idle container memory costlier to waste. Manual tuning also carries real friction — downsizing demands days of metric watching and trial-and-error, with production incidents as the penalty for cutting too far. At Netflix scale, small over-provisioning compounds across hundreds of thousands of daily executions.

The Stratum Resource Tuner

The Stratum Resource Tuner (SRT) was built to eliminate manual resource specification. It began with a handful of high-throughput workloads with relaxed latency requirements, which account for 70%–80% of resource usage. In SRT's vocabulary, right-sizing and tuning are interchangeable, downsizing means lowering resource limits, and upsizing means raising them; because the goal is cost, the work centers on downsizing.

Enrollment is manual and per resource type — CPU, memory, network or disk. Once a function owner opts in, SRT runs with no human in the loop:

1. Collect resource usage metrics across all executions of the function.
2. Generate a recommendation per resource type.
3. Verify the recommendation through experiments.
4. Apply or reject it based on experiment results.
5. Monitor post-apply metrics for anomalies.

The recommendation is SRT's primary unit of operation, promoted through five states:

• CREATED — a scheduled workflow generated it.
• EVALUATING — verification is running and results are being analyzed.
• APPLIED — all experiments passed and the configuration was rolled out automatically.
• REJECTED — at least one experiment failed; the function configuration is untouched.
• MONITORED — a post-apply soak period confirms stability.

All thresholds and safety limits described below are configurable per function.

Collection: execution summaries and Iceberg history

Every Stratum container reports function execution metadata, infrastructure dimensions and CPU, memory, disk and network utilization at sub-minute resolution. The metrics flow through Keystone, Netflix's real-time event streaming platform, into two datastores:

• Execution summaries store — execution-level summaries indexed in Elasticsearch for fast point lookups; the short-term store for inspecting individual runs.
• High-resolution long-term store — raw usage metrics in Iceberg tables; slower to query, but holds months of history for trend analysis.

One record is written per execution, represented as a span. For a single VideoEncoder.encodeV1 span, Elasticsearch holds the completed-span summary keyed by tags.stratum.spanId:

{
  "tags.stratum.spanId": "38536313f65f060a",
  "tags.functionName": "VideoEncoder.encodeV1",
  "tags.stratum.phase": "run",
  "tags.stratum.success": true,
  "tags.stratum.retriedWithMoreResources": false,
  "tags.stratum.function.execDurationSecs": 26,
  "tags.cpuCount": 8,
  "tags.memoryInMB": 18000,
  "tags.diskInMB": 50000,
  "tags.networkInMbps": 1000,
  "tags.titusTaskId": "06e1a7c0-2502-4a54-905f-b13a1e786107",
  "tags.instanceType": "r7a.24xlarge",
  "tags.nf.region": "us-east-1",
  "ts": 1780743028000
}

The Iceberg record carries the same metadata plus the sub-minute utilization timeseries tracked during that span:

{
  "span_id": "38536313f65f060a",
  "function_name": "VideoEncoder.encodeV1",
  "phase": "run",
  "success": true,
  "start_epoch_secs": 1780743002,
  "end_epoch_secs": 1780743028,
  "runtime_secs": 26,
  "cpu_requested": 8,
  "memory_mb_requested": 18000,
  "disk_mb_requested": 50000,
  "network_mbps_requested": 1000,
  "titusTaskId": "06e1a7c0-2502-4a54-905f-b13a1e786107",
  "instance_type": "r7a.24xlarge",
  "tags.stratum.retriedWithMoreResources": false,
  "env": "prod",
  "resource_usage": [ 
   { "cpu_usage": 1.13, "memory_mb": 3701.37, "disk_mb": 20120.4, "net_in_mb": 221.09, "net_out_mb": 0.12 },
   { "cpu_usage": 2.25, "memory_mb": 3584.09, "disk_mb": 25410.1, "net_in_mb": 0.02, "net_out_mb": 0.05 },
   { "cpu_usage": 4.16, "memory_mb": 6750.79, "disk_mb": 30400.5, "net_in_mb": 165.90, "net_out_mb": 0.38 },
   ... 
  ]
}

Generation: p99 for CPU and network, p99.9 for memory and disk

A Conductor workflow runs daily, reads the previous month of usage from Iceberg, and generates recommendations for every function and resource type — including functions that have not opted in, so owners can preview potential savings. Each recommendation lands in the recommendations datastore and can be queried and filtered by function, resource type or status.

Per function and resource type, the generation logic applies four rules:

• Filter by workload type: only high-throughput workloads are tuned, concentrating effort where fleet-wide savings are largest.
• Aggressively tune CPU and network: target p99 usage with no headroom. These are compressible resources, so occasional limit hits only slow a function slightly.
• Conservatively tune memory and disk: target p99.9 usage with a safety buffer, because exhausting either causes hard failures — OOM or OOD crashes.
• Safeguard changes: discard recommendations whose absolute reduction is too small to justify the effort, and clamp memory and disk reductions to at most 15% per cycle, a limit introduced after early production runs.

New recommendations enter the CREATED state. If one already sits in that state for a function and resource type, SRT overwrites it with fresh calculations. Once past CREATED, the recommended_value and stats fields become immutable, and later daily runs create a new recommendation with a unique ID instead. For VideoEncoder.encodeV1, the initial CPU and memory recommendations look like this:

• CPU recommendation

{

  "function_name": "VideoEncoder.encodeV1",
  "env": "prod", "resource_type": "CPU",
  "current_value": 8, "recommended_value": 4,
  "status": "CREATED",
  "resource_cost_usd_per_hour": ***,
  "stats": {
    "usage": { "p99": 4.16, "p999": 4.57 },
    "execution": {
      "total_fn_executions": 748762182,
      "success_fn_executions_pct": 99.75,
      "total_fn_exec_runtime_hr": 88102723
    },
    "window": { "start": "2026-05-05T00:00:00Z", "end": "2026-06-05T00:00:00Z" }
  },
  "experiment_inputs": null,
  "experiment_workflow_execution_id": null,
  "outcome_message": null,
  "finalized_at": null
}

• Memory recommendation (15% reduction in one step)

{
  "id": "2edb734a-6184-4f5e-8ffa-23004bd665d7",
  "function_name": "VideoEncoder.encodeV1",
  "env": "prod", "resource_type": "MEMORY_MB",
  "current_value": 18000, "recommended_value": 15300,
  "status": "CREATED",
  "resource_cost_usd_per_hour": ***,

  "stats": {
    "usage": { "avg": 5296.43, "p99": 6750.8, "p999": 7820.3 },
    "execution": {
      "total_fn_executions": 748762182,
      "success_fn_executions_pct": 99.75,
      "total_fn_exec_runtime_hr": 88102723
    },
    "window": { "start": "2026-05-05T00:00:00Z", "end": "2026-06-05T00:00:00Z" }
  },
  "experiment_inputs": null,
  "experiment_workflow_execution_id": null,
  "outcome_message": null,
  "finalized_at": null
}

Moving to verification, SRT sets the status to EVALUATING, records the target experiment settings, and links the workflow execution tracking the experiment. When the runs finish, the status becomes APPLIED or REJECTED with outcome details and a completion timestamp.

Retry with more resources

Recommendations are based on historical usage, so they cannot anticipate sudden spikes or novel workload profiles. The fail-safe is a watcher process in each tuned container that tracks memory and disk usage during execution. Exceeding pre-set thresholds — 85% for non-cached memory, 95% for disk — flags the execution as high risk, cancels it, and re-queues the task on a larger container:

new_container_size = original_container_size * 1.5

The 1.5 multiplier applies only to memory and disk.

This mechanism lets day-to-day configurations stay tightly optimized instead of being sized for rare, costly outliers; the infrequent retries cost extra compute and latency. For VideoEncoder.encodeV1, a heavy run crossing 85% memory (15,300 MB) inside its 18,000 MB container is aborted and rerun on a 27,000 MB container. True OOM/OOD event detection would be preferable to threshold-based triggering, but current architectural constraints make it difficult; thresholds are the reliable stopgap. SRT enables retry automatically for any function opted into memory or disk right-sizing, and retry events flow into Elasticsearch and Iceberg alongside the standard telemetry.

Verification: synthetic first, then production

Every recommendation must clear a two-stage verification process run by a Conductor workflow before rollout:

1. Synthetic experiment — a low-cost, low-risk test that catches obvious regressions.
2. Production experiment — a live evaluation across diverse production workloads.

A recommendation reaches production testing only after passing the synthetic stage, and when a function has multiple pending candidates, SRT evaluates the ones with the highest estimated savings first.

Synthetic experiment:

• Runs in a test environment isolated from production traffic.
• Sets up two clusters, one on the current config and one on the recommended config.
• Replays identical request sets across both clusters simultaneously, without splitting traffic, so each configuration sees 100% of test requests.
• Typically runs for 1–2 hours.
• Runs to completion without early termination checks.

Production experiment:

• Runs in production on real user traffic, split three ways: roughly 90% on the main cluster (current config), roughly 5% on a baseline cluster (current config), and roughly 5% on an experimental cluster (recommendation config).
• Compares the recommendation against the similarly sized baseline cluster rather than the main cluster, preserving comparable traffic profiles.
• Typically runs one day to one week, depending on traffic volume.
• Fails fast, cancelling early when regressions appear.

Other Stratum experiments run concurrently, so SRT experiments run at lower priority and yield resources to active higher-priority experiments. The pipeline also tolerates traffic arriving in irregular bursts separated by multi-day or multi-week lulls. For the VideoEncoder.encodeV1 memory recommendation (18,000 MB → 15,300 MB), verification is triggered with:

{
  "recommendationId": "2edb734a-6184-4f5e-8ffa-23004bd665d7",
  "syntheticExperiment": { "numberOfExecutions": 3000 },
  "productionExperiment": {
    "enabled": true,
    "maxContainers": 20, // max containers shift to baseline & experimental clusters
    "upGateTimeoutMinutes": 10080, // time to wait for traffic to start flowing
    "maxRunDurationMinutes": 1240, // max time to run the experiment once traffic starts
    "minSamples": 500 // min executions needed on both clusters before evaluating
  }
}

Both stages apply two sequential checks and terminate immediately if either fails:

• Retries with more resources: memory and disk downsizes are capped at a 1% retry threshold over the experiment duration.
• Execution time: median runtimes are compared to avoid outlier distortion, permitting up to a 10–15% slowdown versus baseline depending on workload, configurable per function.

Passing both stages moves the recommendation to APPLIED and rolls the configuration out to the main production cluster. Failing any check marks it REJECTED and preserves the existing configuration. SRT notifies on both outcomes. For VideoEncoder.encodeV1, both stages passed cleanly:

{
  "outcome": "APPLIED",
  "syntheticExperiment": {
    "result": "SUCCESS",
    "samples": { "baseline": 3000, "experimental": 3000 },
    "medianExecutionSecs": { "baseline": 59.0, "experimental": 59.0 },
    "slownessIncreasePct": 0.0,
    "retryRatePct": { "baseline": 0.0, "experimental": 0.0 }
  },
  "productionExperiment": {
    "result": "SUCCESS",
    "samples": { "baseline": 50361, "experimental": 49790 },
    "medianExecutionSecs": { "baseline": 196.0, "experimental": 198.0 },
    "slownessIncreasePct": 1.0,
    "retryRatePct": { "baseline": 0.0, "experimental": 0.0 }
  }
}

Memory was permanently set to 15,300 MB. The CPU recommendation was equally clean, cutting allocation from 8 cores to 4.

Post-apply monitoring and automated rollback

Verification is not the end of the lifecycle. Updated function code, modified input parameters, or changes in the underlying Titus/EC2 compute environment can alter production behavior. SRT therefore monitors applied changes for 90 days across a 30-minute fast window and a 24-hour slow window. A 3% retry rate over 30 minutes or 2% over 24 hours triggers an automated notification. Steady-state workloads show baseline retry rates near 0.1%, with 2%–3% considered acceptable variance; most applied recommendations have remained stable over the past three months.

VideoEncoder.encodeV1 has run for months on its tuned configuration without breaching thresholds. VideoEncoder.encodeV2 was the difficult case: a 50% memory reduction passed every verification check and ran stably for two weeks before an unexpected traffic shift drove retry rates above 60%.

Early versions had no automated rollback, so engineers triaged exception cases by hand — the incident that prompted self-healing rollback. Today the two auto-revert windows are:

• Last 30 minutes: 3% retries warns; 10% triggers an auto-revert.
• Last 24 hours: a 2% retry rate both alerts and reverts automatically.

Breaching a threshold immediately restores the pre-recommendation baseline.

Revert logic is deliberately simple — restore the last known stable configuration rather than attempt dynamic adjustment mid-incident. The VideoEncoder.encodeV2 incident also reshaped the generation rules: instead of applying full target reductions at once, SRT now makes incremental 15% steps and advances only after confirming stability at each step. A global kill switch can disable automated right-sizing across all functions, and a mandatory cooldown period, typically one week, is enforced before new experiments on a function, allowing performance to be observed across varied production traffic patterns first.

Results and roadmap

The initial focus was high-throughput video encoding, targeting a 5% reduction in overall Stratum compute costs. VideoEncoder.encodeV1 achieved a 50% CPU reduction and 40% memory reduction without hurting performance or retry rates. The VideoEncoder.encodeV2 edge case showed that large single-step reductions hide risk and that manual incident response slows recovery; under the stricter guardrails — 15% per-cycle caps and automated rollbacks at retry thresholds — re-tuning VideoEncoder.encodeV2 produced a stable 20% memory reduction with zero subsequent incidents.

Priorities going forward:

• Expand SRT beyond high-throughput encoding to more function and workload types.
• Tune resources in regions with constrained resource availability.
• Move from reactive alerts to proactive, automated upsizing for under-provisioned functions.
• Experiment with intentional upsizing for latency-sensitive workloads to evaluate performance gains and reduce execution time.
• Upgrade OOM/OOD failure signals from lagging to leading indicators to improve tuning precision.

Credits

Special thanks to Olof Johansson, Jakub Karczewski, Adam Mazur, Ethan Mathieu, Rosanna Lee, Ameya Vasani, Jean Barkhuysen, Paul Hieromnimon and Adithya Prakash.