A Kinesis stream's capacity is expressed through the hash key ranges its shards own. Working through a split and a merge on a real stream makes the mechanics easier to grasp than the high-level developer guide treatment alone.

The starting point is a stream named split-merge-test with a single shard, online and ACTIVE:

$ aws kinesis describe-stream --stream-name split-merge-test
{
    "StreamDescription": {
        "StreamStatus": "ACTIVE",
        "StreamName": "split-merge-test",
        "StreamARN": "arn:aws:kinesis:us-east-1:551639669466:stream/split-merge-test",
        "Shards": [
            {
                "ShardId": "shardId-000000000000",
                "HashKeyRange": {
                    "EndingHashKey": "340282366920938463463374607431768211455",
                    "StartingHashKey": "0"
                },
                "SequenceNumberRange": {
                    "StartingSequenceNumber": "49548859072970256769156879668947671610661756289899560962"
                }
            }
        ]
    }
}

That shard's HashKeyRange runs from zero to 340282366920938463463374607431768211455. A record's partition key is run through a basic hash function, and the result decides which shard receives it according to the range each shard handles. Raising a stream's total capacity means subdividing the hash key range of an existing shard so that it maps to a greater number of shards.

Dividing a shard's hash space

Splitting is manual: an operator chooses how to divide the shard's total hash space among the new shards. Splitting evenly across two children requires only basic arithmetic on the EndingHashKey to locate the midpoint between it and zero:

$ irb
irb(main):001:0> 340282366920938463463374607431768211455 / 2
=> 170141183460469231731687303715884105727

The split is then issued:

$ aws kinesis split-shard --stream-name split-merge-test \
  --shard-to-split shardId-000000000000 \
  --new-starting-hash-key 170141183460469231731687303715884105727

While the work is in progress the stream reports UPDATING, and the shards still look unchanged:

$ aws kinesis describe-stream --stream-name split-merge-test
{
    "StreamDescription": {
        "StreamStatus": "UPDATING",
        "StreamName": "split-merge-test",
        "StreamARN": "arn:aws:kinesis:us-east-1:551639669466:stream/split-merge-test",
        "Shards": [
            {
                "ShardId": "shardId-000000000000",
                "HashKeyRange": {
                    "EndingHashKey": "340282366920938463463374607431768211455",
                    "StartingHashKey": "0"
                },
                "SequenceNumberRange": {
                    "StartingSequenceNumber": "49548859072970256769156879668947671610661756289899560962"
                }
            }
        ]
    }
}

Seconds later the changes are visible:

  • Shard hash key ranges are immutable. The parent survives after a split but moves into a CLOSED state — shardId-000000000000 here — and its full range is taken over by the children shardId-000000000001 and shardId-000000000002. The presence of an EndingSequenceNumber marks a shard as CLOSED.
  • With the update finished, the stream is ACTIVE again.
  • The children carry a ParentShardId pointer back to the shard they split from, preserving some history.
  • The stream's sequence number jumps substantially during the split — roughly 10^48. Normal record insertions already move it by about 10^24, which puts the split jump in perspective, but it is still considerably larger.
$ aws kinesis describe-stream --stream-name split-merge-test
{
    "StreamDescription": {
        "StreamStatus": "ACTIVE",
        "StreamName": "split-merge-test",
        "StreamARN": "arn:aws:kinesis:us-east-1:551639669466:stream/split-merge-test",
        "Shards": [
            {
                "ShardId": "shardId-000000000000",
                "HashKeyRange": {
                    "EndingHashKey": "340282366920938463463374607431768211455",
                    "StartingHashKey": "0"
                },
                "SequenceNumberRange": {
                    "EndingSequenceNumber": "49548859072981407141756144980517230543978492779512725506",
                    "StartingSequenceNumber": "49548859072970256769156879668947671610661756289899560962"
                }
            },
            {
                "ShardId": "shardId-000000000001",
                "HashKeyRange": {
                    "EndingHashKey": "170141183460469231731687303715884105726",
                    "StartingHashKey": "0"
                },
                "ParentShardId": "shardId-000000000000",
                "SequenceNumberRange": {
                    "StartingSequenceNumber": "49548859213219643322715968606065803827347328807764754450"
                }
            },
            {
                "ShardId": "shardId-000000000002",
                "HashKeyRange": {
                    "EndingHashKey": "340282366920938463463374607431768211455",
                    "StartingHashKey": "170141183460469231731687303715884105727"
                },
                "ParentShardId": "shardId-000000000000",
                "SequenceNumberRange": {
                    "StartingSequenceNumber": "49548859213241944067914499229207339545619977169270734882"
                }
            }
        ]
    }
}

A CLOSED shard's story isn't over yet. Once the last records it holds pass out of Kinesis' retention window it moves from CLOSED to EXPIRED, and after that no records can ever be read from it again.

Recombining adjacent shards

A merge takes two parameters: the main shard to merge, and the adjacent shard to mix into it. "Adjacent" is deliberate wording — because of how shards handle hash key ranges, only two shards whose ranges are contiguous can be merged.

$ aws kinesis merge-shards --stream-name split-merge-test \
  --shard-to-merge shardId-000000000001 \
  --adjacent-shard-to-merge shardId-000000000002

Once again the stream enters UPDATING without yet reflecting the change:

$ aws kinesis describe-stream --stream-name split-merge-test
{
    "StreamDescription": {
        "StreamStatus": "UPDATING",
        "StreamName": "split-merge-test",
        "StreamARN": "arn:aws:kinesis:us-east-1:551639669466:stream/split-merge-test",
        "Shards": [
            {
                "ShardId": "shardId-000000000000",
                "HashKeyRange": {
                    "EndingHashKey": "340282366920938463463374607431768211455",
                    "StartingHashKey": "0"
                },
                "SequenceNumberRange": {
                    "EndingSequenceNumber": "49548859072981407141756144980517230543978492779512725506",
                    "StartingSequenceNumber": "49548859072970256769156879668947671610661756289899560962"
                }
            },
            {
                "ShardId": "shardId-000000000001",
                "HashKeyRange": {
                    "EndingHashKey": "170141183460469231731687303715884105726",
                    "StartingHashKey": "0"
                },
                "ParentShardId": "shardId-000000000000",
                "SequenceNumberRange": {
                    "StartingSequenceNumber": "49548859213219643322715968606065803827347328807764754450"
                }
            },
            {
                "ShardId": "shardId-000000000002",
                "HashKeyRange": {
                    "EndingHashKey": "340282366920938463463374607431768211455",
                    "StartingHashKey": "170141183460469231731687303715884105727"
                },
                "ParentShardId": "shardId-000000000000",
                "SequenceNumberRange": {
                    "StartingSequenceNumber": "49548859213241944067914499229207339545619977169270734882"
                }
            }
        ]
    }
}

The stream then returns to ACTIVE with the merged shard in place:

  • As with the split, the closed shards shardId-000000000001 and shardId-000000000002 still exist, each now carrying an EndingSequenceNumber.
  • The new shard shardId-000000000003 retains its history, pointing to its ParentShardId and to the AdjacentParentShardID that also contributed to it.
$ aws kinesis describe-stream --stream-name split-merge-test
{
    "StreamDescription": {
        "StreamStatus": "ACTIVE",
        "StreamName": "split-merge-test",
        "StreamARN": "arn:aws:kinesis:us-east-1:551639669466:stream/split-merge-test",
        "Shards": [
            {
                "ShardId": "shardId-000000000000",
                "HashKeyRange": {
                    "EndingHashKey": "340282366920938463463374607431768211455",
                    "StartingHashKey": "0"
                },
                "SequenceNumberRange": {
                    "EndingSequenceNumber": "49548859072981407141756144980517230543978492779512725506",
                    "StartingSequenceNumber": "49548859072970256769156879668947671610661756289899560962"
                }
            },
            {
                "ShardId": "shardId-000000000001",
                "HashKeyRange": {
                    "EndingHashKey": "170141183460469231731687303715884105726",
                    "StartingHashKey": "0"
                },
                "ParentShardId": "shardId-000000000000",
                "SequenceNumberRange": {
                    "EndingSequenceNumber": "49548859213230793695315233917635362760664090379986927634",
                    "StartingSequenceNumber": "49548859213219643322715968606065803827347328807764754450"
                }
            },
            {
                "ShardId": "shardId-000000000002",
                "HashKeyRange": {
                    "EndingHashKey": "340282366920938463463374607431768211455",
                    "StartingHashKey": "170141183460469231731687303715884105727"
                },
                "ParentShardId": "shardId-000000000000",
                "SequenceNumberRange": {
                    "EndingSequenceNumber": "49548859213253094440513764540776898478936738741492908066",
                    "StartingSequenceNumber": "49548859213241944067914499229207339545619977169270734882"
                }
            },
            {
                "ShardId": "shardId-000000000003",
                "HashKeyRange": {
                    "EndingHashKey": "340282366920938463463374607431768211455",
                    "StartingHashKey": "0"
                },
                "ParentShardId": "shardId-000000000001",
                "AdjacentParentShardId": "shardId-000000000002",
                "SequenceNumberRange": {
                    "StartingSequenceNumber": "49548859483727682580892427312894066474572005964670566450"
                }
            }
        ]
    }
}

Every later split and merge repeats the same pattern, leaving a trail of dead shards as a historical record of the stream's lifecycle. The immutable hash range of a shard is what makes it possible to guarantee in-order record consumption across a merge or split — a subject worth its own detailed treatment.