Guaranteeing Order with Kinesis Bulk Puts

The Ordering Problem in Kinesis Bulk Writes

Kinesis offers two ways to push records into a stream. The single-record PutRecord API accepts an optional SequenceNumberForOrdering parameter: pass in the sequence number from your previous write, and the service guarantees the new record receives a larger sequence number, preserving your intended order. The bulk PutRecords API, meanwhile, lets you send up to 1,000 records or 1 MB in a single request. That throughput advantage comes with a catch: order is only best-effort. If one record in the batch fails, the rest succeed as normal, and the producer is responsible for retrying the failed record later—which almost certainly means it lands in the stream after later records from the original batch.

REQUEST 1 [Producer]
PutRecords
  - record1
  - record2
  - record3

RESPONSE 1 [Kinesis]
  - OK. SequenceNumber="123".
  - FAILED.
  - OK. SequenceNumber="124".

REQUEST 2 (retry failed) [Producer]
PutRecords
  - record2

RESPONSE 2 [Kinesis]
  - OK. SequenceNumber="125".

So the trade-off seems stark: PutRecord gives you ordering per record but at the cost of many round trips; PutRecords gives you throughput but with no ordering guarantees across retries. To get both, you first have to accept the fundamental constraint of Kinesis: ordering exists only within a shard, not across a stream with multiple shards.

Scaling via Shards, Ordering via Partition Keys

A Kinesis stream is divided into shards, each with its own write and read throughput limits (1 MB/s and 2 MB/s, respectively). As data volume grows, you add shards. Records on a single shard are streamed to consumers in the order they were appended; there is no inter-shard ordering. Producers choose a shard implicitly by supplying a partition key with each record. That key is hashed to determine a shard deterministically, and as long as the number of shards is unchanged, the same key always resolves to the same shard.

The practical consequence: a global, cross-shard ordering guarantee is impossible. The design goal should shift to maximizing throughput while ensuring per-shard ordering for the records that actually need it—namely, events that share a logical subject and must therefore be processed in order.

Batching by Partition Key, Sequentially Per Key

The production strategy that solves this is straightforward:

while records = more_records()
  records_to_post = records.
    group_by { |record| record.partition_key }.
    map { |_, partition_group| partition_group.first }
  kinesis.put_records(records_to_post)
end

Most of the time, all pending records can be posted in one bulk call. The exception is when multiple pending records share a partition key. In that case, only the earliest one is posted in the current batch; the rest wait for subsequent cycles. Because the batch will contain at most one record per partition key, a partial failure—and later retry—can’t reorder records that need to be ordered relative to each other.

In practice, if records are staged in a relational database before streaming, this logic can be expressed in SQL. Given a table tracking partition_key and a sequential id per record, you can partition the pending set by key and pull only the first record from each partition:

=> SELECT partition_key, record_data FROM kinesis_records
WHERE id IN (
  SELECT MIN(id)
  FROM kinesis_records
  GROUP BY partition_key
)
ORDER BY id;

            partition_key             |              record_data
--------------------------------------+---------------------------------------
 8a9e7a19-9fe1-49b2-9b42-591520784449 | {"resource":"app","action":"create"}
 d0d97986-0c90-404f-bccd-9ac6c27f9235 | {"resource":"app","action":"create"}
 b20d88bc-ba68-41e3-87cb-3a93cc619833 | {"resource":"app","action":"update"}
(3 rows)

That query yields a single record per partition key, all of which are safe to post together in one bulk request. The next pass picks up the next record for each still-pending key, and so on until the table is drained. Each batch therefore contains unique partition keys, and even if a record fails and must be retried, it remains the earliest unsent record for its key.

Choosing a Semantically Correct Partition Key

This approach transfers a great deal of responsibility to partition key selection. The key is the only lever you have to influence ordering, and poor choices will produce incorrect behavior at the consumer even when every producer works exactly as designed.

Consider a GitHub-like service streaming repository events. Three event types need to be handled: repository creation, repository destruction, and commits. A consumer maintaining a cache must see the create and destroy events for a given repository in the correct relative order, or it will end up with a stale cache entry for a deleted repository. Partitioning those events by the repository identifier (for example, brandur/my-project) routes them to the same shard, preserving order.

A tempting but wrong choice would be to partition commits by their SHA hash (c0ab1e5c). That would scatter commits for the same repository across different shards, making it possible for a consumer to see a commit for a repository it has already processed a destroy event for. The correct approach is to partition a commit by the identifier of its parent repository, not by the commit itself, so that all events touching that repository share a shard and an ordering context.

In the end, this design accepts that global ordering across the stream is unattainable. It invests the available ordering guarantees where they matter—per logical entity—and allows the stream to be sharded aggressively without breaking consumers, since every consumer sees a consistent sequence for each partition key. The result is throughput scaling paired with correctness for the relationships that require it.