The problem with petabyte-scale hot storage
Netflix's TimeSeries Abstraction, a high-throughput, low-latency platform for temporal event data, has grown to back some of the company's largest datasets, now spanning multiple petabytes. Keeping all of that in hot storage tuned for low-millisecond access is expensive, and a set of compounding pressures makes the status quo hard to sustain:
- Retention demands: users push for longer retention to serve more use cases, raising storage costs.
- Cost: dense clusters kept around for rarely accessed historical data are prohibitively expensive.
- Sparse access: most queries hit recent events, but older data must stay available for audits and investigations.
- Operational strain: high storage density on hot nodes complicates scaling, deployment and cluster stability.
- Latency: historical data sometimes still needs sub-second performance for internal tooling.
The opportunity is to keep recent data on fast storage and move older, rarely read data to cheaper object storage behind a consistent API. Two designs did that: a Kafka and Flink streaming pipeline that wrote to S3 in real time, and a later approach that reads Cassandra's own S3 backups directly.
Recap: Time Slices and query routing
TimeSeries partitions its data into multiple tables, each covering a window of time called a Time Slice. Once a table crosses the write boundary it becomes immutable, which unlocks further optimizations.
Slice information lives in a dedicated metadata table:
The TimeSeries control plane owns creation of new Time Slices and their metadata; the data plane consumes it to route reads and writes to the right tables.
Solution 1: streaming to S3 with Kafka and Flink
The first approach was a conventional streaming pattern — data written to Cassandra while simultaneously streamed to S3.
Making the S3 writes efficient
Time Series IDs are hash-partitioned into buckets, which splits the stream so dedicated Flink tasks can own each bucket and batch writes to reduce S3 throughput. Flink workers batch events into Parquet files grouped by event time within the hour. That format suits high-throughput streaming writes but not low-latency reads, which is where a compaction job comes in.
That job runs daily and merges small Parquet files into one optimized daily Parquet file per namespace, tuning it in three ways:
- Small row groups (4–8 MB) to minimize lookup cost, read via range reads through the Parquet Java library. This is deliberately below Parquet's recommended row group floor, because reads here are dominated by point and range lookups rather than full scans.
- Event sorting within a row group by time, removing brute-force scans for time-range queries.
- Bloom filters and file metadata cached in EVCache, so the TimeSeries query planner can skip unnecessary S3 reads.
When compaction finishes, the compacted files per partition are recorded in the control plane and the hot storage slices can be truncated. On the largest datasets the job takes roughly 12 hours — lower compute cost is prioritized over execution speed.
Serving the cold reads
The TimeSeries data plane fans a query out across hot and cold tiers and merges the results, while users keep the same query interface.
The cold tier data plane uses compaction metadata for aggressive planning and pruning, which sharply cuts S3 fan-out. A full day of data requires at most three sequential S3 reads, even when an ID spans two row groups. That yields roughly 150 ms at p50 and ~500 ms at p99 per read. At the time of writing, the streaming cold tier was serving up to 1,250 reads per second globally from cold storage.
Why the streaming tier wasn't the endpoint
The pipeline did cut Cassandra's EBS footprint and the Cassandra backup footprint in S3, but the streaming infrastructure itself is costly and its extra moving pieces widen the team's on-call scope.
Two questions followed: can we avoid streaming data into the object store and the data transfer costs that come with it, and can we reduce the compute and operational overhead of running new components? That led to a more radical one — can we read directly from the Cassandra backups already sitting in S3?
Solution 2: reading Cassandra backups in place
Cassandra data is already backed up to S3 at regular intervals for disaster recovery. Tiering on top of those backups means reading from an existing copy instead of paying a separate transfer cost to create one. Once a hot Time Slice is marked ready for cold storage, Cassandra can be customized to read its backups directly and the on-disk copy deleted — with minimal changes to the TimeSeries read path.
Making that production-viable meant answering a chain of design questions.
Reading backups efficiently: ReadOnlyCompactionStrategy
The first is a new Cassandra compaction strategy, ReadOnlyCompactionStrategy (ROCS). It relies on immutability: once a Time Slice crosses the user-configured cold-tier boundary, TimeSeries stops accepting writes to it, which makes optimization safe. Standard strategies such as LCS and TWCS often produce SSTables that overlap in token space, so a single partition key can exist across multiple files; because a read must touch every SSTable covering the key's token range, that overlap turns one point read into multiple lookups and raises latency on every tier. ROCS re-compacts all SSTables once, aligning them precisely with token ranges.
ROCS applies broadly across TimeSeries use cases, reducing SSTable fan-out during reads and driving tail latency down on all storage tiers — not just the cold one.
What the backups contain
Cassandra backs up every node in the cluster, organizing each backup on S3 as a declarative manifest plus the SSTables backing the keyspace's tables.
{bucket}/
└── {prefix}/ ← cluster identity (env_backup/hash_app)
└── {token}/ ← node identity (ring token)
├── META_V2/ ← manifests, one per snapshot
│ ├── {snapshotInstantMs}.json
│ ├── {snapshotInstantMs}.json
│ └── ...
└── SST_V2/ ← actual SSTable files
└── {backupTimestamp}/ ← snapshot timestamp
└── {keyspace}/
└── {table-with-uuid}/
└── {compression}/
└── {encryption}/
├── {component-file}
└── ...
The backup manifest orchestrates the backup by listing the paths to all of its SSTables.
manifest.json
├── info
│ ├── appName "cass-cluster"
│ ├── region "us-east-1"
│ ├── snapshotInstantMs 1748908800000
│ ├── backupPathPrefix "prod_backup/1234_cass/token"
└── data[]
└── [0]
├── keyspaceName "ks"
├── columnfamily "tbl-a1b2..."
└── sstables[]
└── [0]
└── components[]
├── na-1-big-Data.db
├── na-1-big-Index.db
├── na-1-big-Filter.db
├── na-1-big-Statistics.db
├── na-1-big-Summary.db
├── na-1-big-CompressionInfo.db
Alongside the large Data.db file, a backup carries a compact set of index, metadata and integrity files. That lets the system locate, validate and restore specific partitions or data ranges without scanning the whole dataset, and because the manifest catalogs every S3 path, most components other than Data.db can be pulled down to the Cassandra node where they take up negligible space. Having those components locally enables query planning before any data is fetched from S3.
Routing reads through a pluggable Memtable
Cassandra normally reads from the Memtable, which holds recent unflushed writes, and from flushed SSTables on durable storage. As of Cassandra 4.1, pluggable Memtables allow reads on "cold" tables to be redirected straight to backup storage in S3.
-- create table for a given Time Slice
CREATE TABLE data_20260616_00_04 (
...
...,
PRIMARY KEY ((time_series_id, time_bucket, event_bucket), event_time, event_id, key)
) WITH CLUSTERING ORDER BY (event_time DESC, event_id DESC, key DESC)
...
AND compaction = ...
AND compression = ...
AND memtable = 'backupmemtable:timestamp=1781953200000' ← Custom memtable
...
The timestamp marks the point at which the backup is guaranteed to cover the full table and the table is guaranteed to be immutable. Opening a backup Memtable starts a state-machine initialization that pulls index data from S3 onto the node once.
Several mechanisms keep that initialization reliable:
- Retries and backoff: all S3 operations and file-system cache writes retry with exponential backoff.
- Resource isolation: initialization runs on a dedicated thread pool, separate from normal Cassandra operations.
- Partial download protection: a component whose downloaded size doesn't match the manifest is deleted and re-downloaded.
- DDL change protection: a .timestamp file marks completed initialization and ties it to the backup timestamp; if that timestamp changes, the file is removed and initialization restarts.
- Read protection: initialization is asynchronous, and reads against an initializing memtable block for at most half the read RPC timeout.
Once every SSTable component except Data.db is on local disk, the Memtable is ready to serve reads.
Latency is minimized by parallelizing S3 reads across requests, SSTables and individual chunks, with a shared cache that deduplicates fetches and prefetches data so chunks are ready before needed. Cluster and table settings further cut the number of S3 calls:
- Larger compression chunks: fewer, bigger chunks per partition mean fewer S3 calls, accepting some over-fetch on small reads.
- Read repair disabled: immutable, pre-repaired backup data makes repair traffic unnecessary and expensive.
- Speculative retries disabled: S3's higher latency would otherwise trigger redundant, overlapping reads.
Verifying safety before pointing at a backup
Before a Memtable can point at a backup, two conditions must hold across the whole ring, checked through Cassandra Virtual Tables: that ROCS re-compaction succeeded, and that backups were fully captured (denoted by uploaded=True).
cqlsh> SELECT * FROM netflix_views.cluster_view
WHERE keyspace_name = 'system_views' AND table_name = 'sstable_tasks';
keyspace_name | table_name | host | value
--------------+---------------+----------+------------------------------------------------
system_views | sstable_tasks | 10.0.0.1 | [{"table_name":"<.>","kind":"compaction", ...}]
system_views | sstable_tasks | 10.0.0.2 | [] <---- no active compactions
system_views | sstable_tasks | 10.0.0.3 | []
cqlsh> SELECT * FROM netflix_views.complete_backups
WHERE keyspace_name = 'my_ks' AND table_name = 'my_table';
keyspace_name | table_name | timestamp | num_tokens | num_uploaded | uploaded
--------------+------------+---------------+------------+--------------+----------
my_ks | my_table | 1765211400000 | 3 | 3 | True
my_ks | my_table | 1765213200000 | 3 | 2 | False
Lifecycle automation and load isolation
The TimeSeries control plane automates the transition as slices cross the immutable boundary; a slice can be in any lifecycle stage at any time without affecting availability, and the system tolerates delayed backups or lagging compactions.
Reading from S3 is far more CPU-intensive than reading local storage, so a spike in cold read requests could overwhelm a cluster and degrade the hot path, turning a cost-saving feature into a reliability risk. Cold reads can therefore optionally run on a separate Cassandra cluster whose backup Memtables point at the hot cluster's backups. That cold cluster must match the original in topology but need not match it in hardware — it can run on much smaller, cheaper, storage-light instances.
Measured results
Cassandra Cold Tier was rolled out to the most storage-intensive TimeSeries clusters. It now holds over 15 PB of compressed data in cold storage and has allowed attached storage to be cut significantly across all affected clusters.
It serves thousands of reads per second from cold tier storage in any given region.
Against the previous cold storage implementation, it improves p90 latency by roughly 30% through more efficient S3 access patterns and smaller read sizes, while keeping p99 latency under about 500 ms. Cassandra already absorbs bursts of cold reads through its chunk cache, and this design preserves that resilience for the S3-backed tier.
