Kinesis After 30 Days in Production
We’ve been running a production component on Amazon Kinesis for just over a month. The load we’re putting on our stream is moderate — we haven’t approached the service’s upper bounds — but the experience has been revealing enough to share a few concrete observations on performance, limitations, and whether I’d make the same choice again.
Kinesis is a data store for real-time processing: records are written in on one end and read out on the other. Unlike a queue, every record is meant to be consumed multiple times, by independent consumers, so records persist for a 24-hour sliding window. A stream is the basic resource, subdivided into shards, each handling 1 MB/s of writes and 2 MB/s of reads. Records are routed to shards via a user-supplied partition key, which Kinesis hashes to a consistent shard mapping. That design puts scaling decisions in the user’s hands — a simple and powerful trade-off.
Performance and stability
Our metrics come from six producer nodes in the same AWS region as the stream, using the bulk put records API with payloads of roughly 1 to 10 events per call. The API runs over HTTP, and our code reuses open connections wherever possible.
Write latency has been steady: P50 sits around 35 ms, P95 around 100 ms, and P99 close to 200 ms. We don’t see 300 ms broken in these numbers.
Read latency is higher: P50 is closer to 150 ms, with P95 around 1 second and P99 just over 2 seconds. Reads are bulk fetches — consumers pull as many unconsumed records as are available.
End-to-end latency — from record inception on the producer to delivery to a consumer — shows a P50 of about 1.5 seconds, with P95 and P99 around 5 seconds. That total includes some time our records spend waiting in our own application before being dispatched to Kinesis, which may mirror similar setups for other real-world users.
These are acceptable numbers for a distributed system — especially considering that Kinesis synchronously replicates each record across three availability zones within a region during a write, which carries a real performance cost. Accuracy matters more than ultra-low latency for our use case.
Qualitatively, we haven’t observed a serious operational problem in any stream during the time they’ve been live. That doesn’t tell us much about behavior during a major outage, but it does suggest the infrastructure is solid.
The real limitations
The limits we found are few but meaningful. Two stand out.
Five reads per second per shard
Each shard supports a maximum of 5 reads per second — a number on the limits page, but one whose consequences the docs never spell out. Since every consuming application must read every shard of a stream, 5 reads per second per shard caps the total number of concurrent consumers. With 5 applications, each can read the stream once per second. With 10 applications, each must throttle back to once every two seconds. Each read pulls up to 10 MB, so keeping up with the stream isn’t an issue — but scaling the number of consumers forces you to sacrifice read latency, and you can’t have more than 5 consumers that each read a shard once per second.
I had hoped to build infrastructure with 100 or more applications reading the same stream. At the current limit, each would only be able to read once every 20 seconds on average — too slow even for our relatively latency-insensitive use case.
Given that each shard already has hard input/output limits, the aggressive per-shard read cap is puzzling. I’ve spoken with Kinesis staff but haven’t gotten a clear justification. The suggested workaround — chaining streams together to achieve sufficient fanout — wasn’t something I was willing to adopt.
Here’s what “throughput exceeded” errors looked like over a 30-minute window with only three consumers hitting a low-throughput stream once per second:
Vanishing history
Kinesis shards are immutable: splitting or merging one closes it and creates new shards in its place. This design is elegant for consumers trying to preserve read order across splits and merges. The DescribeStream API lets consumers examine shard ancestry — each open shard’s parentage and the range of sequence numbers each closed shard handled over its lifetime.
However, DescribeStream prunes its list regularly, removing closed shards on a schedule that makes it impossible for a consumer that comes back online after a split — outside the pruning window — to determine whether it has fully consumed the stream. There’s no API parameter to request a complete list.
Aggressively pruning these few hundred bytes of JSON from an infrequently-called endpoint is a strange design choice, and as with the read limit, staff discussions didn’t yield insight into the rationale.
Kafka comparison
I’ve only run Kafka for a few months and without deep analysis, so I’ll keep this brief. What I appreciated about Kinesis is that it removes the concept of a topic — a logical channel within a Kafka cluster that lets one cluster serve multiple applications. Topics reduce operational moving parts in Kafka, but as a developer, they’re not something I care to think about. With Kinesis, each record type gets its own stream, appearing as isolated services that scale independently with their own rate limits and throughput capacity. For a hosted service, that’s a significant win.
Verdict
Overall, Kinesis behaves like a well-designed infrastructure component: it performs, and it stays out of the way. We’ll continue to use it moving forward. But the 5-reads-per-second-per-shard limit will cap what we can build on it — and had I fully understood the consequences earlier, I might have pushed harder for Kafka. It’s not worth re-implementing what we have now.
For prospective users, Kinesis is worth recommending to avoid running your own Kafka cluster — with one caveat: if the read limit sounds problematic for your requirements, investigate all alternatives before committing.
