Why Fixed-Size Chunks Fell Short

Astra, Slack’s in-house log search engine, ingests more than 6 million log messages per second — over 10 GB of data every second. To make that volume searchable, Astra groups logs by time into storage units called chunks. Design assumptions early on treated all chunks as equal in size, which simplified the codebase but created real inefficiencies in how cache node disk space was allocated.

Cache nodes were assigned a fixed number of slots, each expected to hold a chunk of a predetermined size — for example, 200 slots on a 3 TB node, each for a 15 GB chunk. But chunks frequently came in undersized (say 10 GB instead of 15 GB), leaving the extra 5 GB per slot allocated yet unused. Across thousands of chunks, that wasted space added up to a meaningful percentage of cluster capacity.

The inverse problem also occurred. Recovery tasks, which are generated based on how many messages Astra is behind rather than the byte size of that backlog, could produce chunks larger than the assumed slot size. An oversized chunk on an undersized allocation is worse than unused space: it means the system simply isn’t providing enough room.

Reworking the Cache Node Lifecycle

The fix required changes to two core parts of Astra: the cache layer and the Cluster Manager.

In the old design, a cache node coming online advertised the number of slots it could hold in Zookeeper, Astra’s coordination store. The manager then assigned a chunk to each slot, and the node downloaded and served the assigned chunks. Slots were ephemeral — when a node went offline, its slots vanished, prompting the manager to reassign those chunks elsewhere.

With dynamic chunks, a node can no longer predict how many chunks it will be assigned, only its total disk capacity. That broke the ephemeral slot model, so the team replaced it with two persistent metadata types in Zookeeper:

  • Cache Node Assignment: maps each chunk ID to a cache node.
  • Cache Node Metadata: records per-node details such as capacity and hostname.

The revised flow:

  1. The cache node comes online and advertises its disk space rather than a fixed slot count.
  2. The manager evaluates available chunk assignments and node capacities, applying bin packing to minimize the number of nodes in use.
  3. Each cache node reads its assignments and downloads the listed chunks.

Manager Changes and Bin Packing

The Cluster Manager’s assignment logic was also rewritten to use the new metadata. The previous approach was straightforward zipping: grab all slots, grab all chunks, pair them off in order. The new approach:

  • Collect the list of chunks needing assignment.
  • Collect the list of available cache nodes with their capacities.
  • For each chunk, run a first-fit bin packing pass to pick a destination node, then persist the node-to-chunk mapping.

First-fit bin packing was chosen for its execution speed and implementation simplicity, fitting chunks into nodes as tightly as possible.

for each chunk
  for each cache node
    if the current chunk fits into the cache node:
      assign the chunk
    else:
      move on to the next cache node
  if there aren’t any cache nodes left and the chunk hasn’t been assigned:
    create a new cache node

Safe Rollout Strategy

Because the change effectively rewrote all chunk assignment and download logic, the rollout was deliberately conservative. Two safeguards were used:

  1. Dual replicas: Astra hosted two copies of the same data. The team could deploy the change to one replica and observe behavior while the other continued serving traffic uninterrupted.
  2. Feature flag: All new code paths were hidden behind a flag, allowing early merges to master without any runtime effect until explicitly enabled.

The flag also made it possible to stage the rollout, starting with small clusters and advancing to larger ones only after verification passed at each step.

Measured Impact

The result: clusters with many undersized chunks required up to 50% fewer cache nodes. Overall, cache node infrastructure costs dropped 20%, a significant reduction in the cost of operating Astra.