Introducing Netflix’s TimeSeries Data Abstraction Layer

Why Netflix Built a TimeSeries Abstraction Layer

Netflix’s expansion into areas like Video on Demand and Gaming has made the ability to ingest and store massive volumes of temporal data — often petabytes — with millisecond access latency a core requirement. Building on the earlier Key-Value Data Abstraction Layer and the Data Gateway Platform, the engineering team created the TimeSeries Abstraction: a scalable solution for storing and querying large volumes of immutable temporal event data with low latency and cost efficiency.

It’s important to clarify what this system is not. Despite its name, it is not a general-purpose time series database. Use cases like metrics, histograms, and timers are well served by Netflix’s Atlas telemetry platform. The TimeSeries Abstraction focuses instead on the challenge of managing extremely high-throughput, immutable event data in a low-latency, cost-efficient manner.

The Challenges of Temporal Data at Scale

Temporal data at Netflix comes from diverse sources: user interactions like video-play events, asset impressions, and micro-service network activities. Managing this data at scale presents several specific challenges:

• High Throughput: Handling up to 10 million writes per second while maintaining high availability.
• Efficient Querying: Storing petabytes of data while ensuring primary key reads return results within low double-digit milliseconds, plus supporting searches and aggregations across multiple secondary attributes.
• Global Operations: Supporting reads and writes from anywhere in the world with adjustable consistency models.
• Flexible Configuration: Partitioning datasets in either single-tenant or multi-tenant datastores, with tunable dataset options like retention and consistency.
• Handling Bursty Traffic: Managing significant traffic spikes during high-demand events such as new content launches or regional failovers.
• Cost Efficiency: Reducing the cost per byte and per operation to optimize long-term retention, with infrastructure expenses reaching millions of dollars.

Core Design Principles

The TimeSeries Abstraction was built around several guiding principles to meet these requirements:

• Partitioned Data: A temporal partitioning strategy combined with event bucketing efficiently handles bursty workloads and streamlines queries.
• Flexible Storage: The service integrates with different storage backends, including Apache Cassandra and Elasticsearch, allowing customization based on use case requirements.
• Configurability: Tunable options at the dataset level accommodate a wide array of use cases.
• Scalability: Horizontal and vertical scaling lets the system handle increasing throughput and data volumes as Netflix grows.
• Sharded Infrastructure: The Data Gateway Platform enables deployment of single-tenant or multi-tenant infrastructure with the necessary access and traffic isolation.

Data Model

The event data model captures everything needed to store event data while supporting efficient queries, working from the smallest unit upward:

• Event Item: A key-value pair for storing data for a given event, such as {“device_type”: “ios”}.
• Event: A structured collection of one or more event items, occurring at a specific point in time and identified by a client-generated timestamp plus an event identifier (like a UUID). The combination of event_time and event_id forms part of the unique idempotency key for the event, enabling safe retries.
• Time Series ID: A collection of one or more events over the dataset’s retention period. For example, a device_id would store all events occurring for that device over the retention period. Events are immutable; the service only ever appends events to a given time series ID.
• Namespace: A collection of time series IDs and event data, representing the complete TimeSeries dataset. Users can create one or more namespaces per use case, with tunable options applied at the namespace level through the service’s control plane.

API Overview

The abstraction exposes several endpoints for interacting with event data:

WriteEventRecordsSync: Writes a batch of events with a durability acknowledgement to the client, for cases requiring guaranteed durability.

WriteEventRecords: The fire-and-forget version that enqueues events without a durability acknowledgement, suited for logging or tracing where throughput matters more and small data loss is acceptable.

ReadEventRecords: Given a namespace, timeSeriesId, timeInterval, and optional eventFilters, returns all matching events sorted descending by event_time with low millisecond latency.

SearchEventRecords: Returns all matching events based on a search criteria and time interval, for use cases tolerant of eventually consistent reads.

AggregateEventRecords: Performs aggregations (e.g., DistinctAggregation) within a time interval based on search criteria. Users can tolerate eventual consistency and potentially higher latency measured in seconds.

Storage Layer Design

The storage layer consists of a primary data store and an optional index data store. The primary data store guarantees data durability during writes and supports primary read operations; the index data store handles search and aggregate operations. Netflix typically uses Apache Cassandra for durable high-throughput storage and Elasticsearch for indexing. However, the storage layer is not tightly coupled to these databases — the abstraction defines storage API contracts instead, keeping the flexibility to swap underlying data stores as needs evolve.

Storing Time-Series Data in Cassandra

At Netflix's scale, continuous event data can overwhelm traditional databases. Temporal partitioning solves this by dividing data into manageable chunks based on time intervals — hourly, daily, or monthly windows. This approach allows efficient querying of specific time ranges without scanning entire datasets, and it enables archiving, compression, or deletion of older data with less effort. The strategy also mitigates performance issues associated with wide partitions in Cassandra, allowing much higher disk utilization since less space must be reserved for compactions.

Time slices. A time slice is the unit of data retention and maps directly to a Cassandra table. Multiple time slices are created, each covering a specific interval of time, and an event lands in the slice determined by its event_time. Slices are joined with no time gaps, with operations being start-inclusive and end-exclusive, so all data finds a home. Retention is implemented by dropping entire tables, which reduces storage and costs.

Why not row-based TTL? Applying TTL to individual events generates a significant number of tombstones in Cassandra, degrading performance, especially during range scans. Discrete time slices avoid the tombstone issue entirely. The tradeoff is that data may be retained slightly longer than necessary, since an entire table's time range must fall outside the retention window before it can be dropped. TTLs are also difficult to adjust later, whereas TimeSeries can extend retention instantly through a single control plane operation.

Time buckets. Within a time slice, data is further partitioned into time buckets. This enables efficient range scans by targeting only the relevant buckets for a query range. If a user needs the entire range over a large period, more partitions must be scanned; these are read in parallel and aggregated at the end, mitigating latency. In most cases, targeting smaller data subsets outweighs the read amplification from scatter-gather operations, since users typically read narrow ranges rather than full retention windows.

Event buckets. To manage high-throughput writes that can burst for a given time series within a short period, time buckets are further divided into event buckets. This prevents overloading a single partition for a given time range and reduces partition sizes, at the cost of slightly more read amplification.

Note: With Cassandra 4.x onward, scanning wide partitions has substantially improved. See Future Enhancements for the Dynamic Event bucketing work that leverages this.

Storage Tables

Two types of tables are used:

• Data tables: time slices that store the actual event data.
• Metadata table: stores per-namespace configuration for each time slice.

Data Tables

The partition key splits events for a time_series_id over a range of time_bucket(s) and event_bucket(s), preventing hot partitions. The clustering key keeps data sorted on disk in the order it is typically read. The value_metadata column records metadata for the event_item_value, such as compression settings.

Writes land in a given time slice, time bucket, and event bucket based on the event_time attached to the event, as dictated by the control plane configuration for the namespace. For example:

During this process, the writer decides how to handle the data before writing — whether to compress it, for instance. The value_metadata column records any such post-processing so the reader can interpret data correctly.

Reads work via a scatter-gather pattern, fetching from multiple partitions and joining the results before returning the final response:

Metadata Table

The metadata table stores time-slice configuration for each namespace:

• No time gaps: The end_time of one time slice overlaps with the start of the next, ensuring every event finds a home.
• Retention: The status marks which tables fall inside or outside the retention window.
• Flexible: Configuration can be adjusted per time slice, allowing partition settings for future slices to be tuned based on patterns in the current one.

The metadata column can hold more information, such as compaction settings, though only partition settings are shown here for brevity.

Indexing for Secondary Access Patterns

To support access via non-primary key attributes, data is indexed into Elasticsearch. Users configure a list of attributes per namespace for searching or aggregating. The service extracts these fields from events as they stream in and indexes documents into Elasticsearch. Depending on throughput, Elasticsearch is used either as a reverse index (with full data fetched from Cassandra) or as the primary store for the entire source data.

Users are never directly exposed to Elasticsearch, just as they are not exposed to Cassandra. They interact with Search and Aggregate API endpoints that translate queries to the appropriate underlying datastore.

Control Plane and Configuration

The data plane executes read and write operations, while the control plane configures every aspect of a namespace's behavior. The data plane communicates with the TimeSeries control stack, which in turn interacts with a sharded Data Gateway Platform Control Plane that oversees control configurations for all abstractions and namespaces.

Separating data plane and control plane responsibilities helps maintain high availability in the data plane, since the control plane handles tasks that may require schema consensus from the underlying data stores.

The namespace configuration snippet below shows the flexibility per namespace:

"persistence_configuration": [
  {
    "id": "PRIMARY_STORAGE",
    "physical_storage": {
      "type": "CASSANDRA",                  // type of primary storage
      "cluster": "cass_dgw_ts_tracing",     // physical cluster name
      "dataset": "tracing_default"          // maps to the keyspace
    },
    "config": {
      "timePartition": {
        "secondsPerTimeSlice": "129600",    // width of a time slice
        "secondPerTimeBucket": "3600",      // width of a time bucket
        "eventBuckets": 4                   // how many event buckets within
      },
      "queueBuffering": {
        "coalesce": "1s",                   // how long to coalesce writes
        "bufferCapacity": 4194304           // queue capacity in bytes
      },
      "consistencyScope": "LOCAL",          // single-region/multi-region
      "consistencyTarget": "EVENTUAL",      // read/write consistency
      "acceptLimit": "129600s"              // how far back writes are allowed
    },
    "lifecycleConfigs": {
      "lifecycleConfig": [                  // Primary store data retention
        {
          "type": "retention",
          "config": {
            "close_after": "1296000s",      // close for reads/writes
            "delete_after": "1382400s"      // drop time slice
          }
        }
      ]
    }
  },
  {
    "id": "INDEX_STORAGE",
    "physicalStorage": {
      "type": "ELASTICSEARCH",              // type of index storage
      "cluster": "es_dgw_ts_tracing",       // ES cluster name
      "dataset": "tracing_default_useast1"  // base index name
    },
    "config": {
      "timePartition": {
        "secondsPerSlice": "129600"         // width of the index slice
      },
      "consistencyScope": "LOCAL",
      "consistencyTarget": "EVENTUAL",      // how should we read/write data
      "acceptLimit": "129600s",             // how far back writes are allowed
      "indexConfig": {
        "fieldMapping": {                   // fields to extract to index
          "tags.nf.app": "KEYWORD",
          "tags.duration": "INTEGER",
          "tags.enabled": "BOOLEAN"
        },
        "refreshInterval": "60s"            // Index related settings
      }
    },
    "lifecycleConfigs": {
      "lifecycleConfig": [
        {
          "type": "retention",              // Index retention settings
          "config": {
            "close_after": "1296000s",
            "delete_after": "1382400s"
          }
        }
      ]
    }
  }
]

Provisioning and Scaling

With so many tunable parameters, automated provisioning workflows deduce settings for a given workload. Users specify workload desires when creating namespaces, and automation translates these into infrastructure and control plane configuration. More detail on this approach is available in an ApacheCon talk by Joey Lynch.

Once initial infrastructure is provisioned, the system scales in response to actual workload:

• Horizontal scaling: TimeSeries server instances auto-scale per attached policies. Storage server capacity is recomputed using the capacity planner.
• Vertical scaling: Server or storage instances can be scaled up for greater CPU, RAM, or attached storage.
• Scaling disk: EBS volumes may be attached when lower-cost storage is preferred over SSD latency; jobs scale the volume when disk usage crosses a percentage threshold.
• Re-partitioning: When workload estimates prove inaccurate, the control plane adjusts partitioning for upcoming time slices based on observed partition histograms. Re-partitioning older data and dynamic partitioning of current data are planned for the future.

Operational Design Principles

TimeSeries applies several techniques to improve performance and strengthen operational guarantees across its read and write paths.

Idempotent Mutations and Hedging

All mutation endpoints are designed to be idempotent so that clients can safely retry or hedge requests. Hedging—sending an identical competing request when the original exceeds an expected response time—keeps tail latencies low, but is only safe when mutations are idempotent. For a given time_series_id event, the combination of event_time, event_id, and event_item_key forms the idempotency key.

Each endpoint is assigned Service Level Objective (SLO) targets per namespace. If a response does not arrive within the configured time, the client hedges the request.

"slos": {
  "read": {               // SLOs per endpoint
    "latency": {
      "target": "0.5s",   // hedge around this number
      "max": "1s"         // time-out around this number
    }
  },
  "write": {
    "latency": {
      "target": "0.01s",
      "max": "0.05s"
    }
  }
}

Partial Returns and Adaptive Pagination

Latency-sensitive clients can opt for partial result sets. Real-time frequency capping is a good example: precision matters less than speed, since a delayed response is effectively useless. The TimeSeries client supports partial returns around SLOs while still preserving the latest ordering of events in the partial fetch.

Reads begin with a default fanout factor of 8 partition buckets scanned in parallel. When the service determines a dataset is dense—most reads are satisfied by the first few buckets—it dynamically reduces the fanout factor to lower read amplification on the underlying datastore. Sparse datasets may trigger an increased fanout, within a reasonable upper bound.

Write Window and Buffering

The active write range is typically smaller than the read range. A configurable acceptLimit prevents writes of events older than now() — acceptLimit; for instance, a 4-hour limit blocks events older than 4 hours. The limit can be raised temporarily for backfills, but is tuned down for normal operations. Once a time range is immutable, it becomes safe to cache, compress, and compact that data for reads.

To handle bursty workloads without overwhelming the datastore, events are coalesced over short durations (usually seconds) in per-instance in-memory queues. Dedicated consumers drain these queues steadily, grouping events by partition key and batching writes to the datastore.

Queues are tailored per datastore because operational characteristics differ. For example, batch sizes for Cassandra writes are significantly smaller than for Elasticsearch indexing, which affects drain rates and batch sizes.

In-memory queues increase JVM garbage collection pressure, but upgrading to JDK 21 with ZGC has delivered an 86% reduction in tail latencies:

Because in-memory queues risk losing events during an instance crash, they are only used for use cases that tolerate some data loss, such as tracing or logging. Use cases requiring durability or read-after-write consistency effectively disable these queues, flushing writes to the datastore nearly immediately.

Dynamic Compaction

Once a time slice leaves the active write window, the immutability of the data enables read optimizations. This includes re-compacting data with optimal strategies, dynamically shrinking or splitting shards to manage resources, and other techniques to maintain fast, reliable performance.

Production Performance

The service writes data in the low single-digit millisecond range:

Point-read latencies remain stable throughout:

At peak globally, the service processes nearly 15 million events/second across all datasets.

Use Cases at Netflix

• Tracing and Insights: Logs traces across all Netflix apps and micro-services to understand service-to-service communication, debug issues, and answer support requests.
• User Interaction Tracking: Tracks millions of interactions—video playbacks, searches, content engagement—feeding real-time insights into recommendation algorithms and user experience.
• Feature Rollout and Performance Analysis: Measures how users engage with new product features, powering data-driven decisions.
• Asset Impression Tracking and Optimization: Tracks asset impressions for efficient content delivery with real-time optimization feedback.
• Billing and Subscription Management: Stores historical billing and subscription data, supporting transaction accuracy and customer service.

Planned Enhancements

• Tiered Storage: Move older, less-accessed data to cheaper object storage with higher time-to-first-byte, potentially saving millions of dollars.
• Dynamic Event Bucketing: Partition keys into optimally-sized partitions in real time as events stream in, rather than relying on static configuration at namespace provisioning. This avoids partitioning time_series_ids that do not need it, reducing read amplification. Improvements in Cassandra 4.x for reading subsets of wide partitions could also reduce the need for aggressive upfront partitioning.
• Caching: Leverage data immutability to cache discrete time ranges intelligently.
• Count and Aggregations: Support users who only need event counts within a time interval, rather than fetching all event data.