Why Kafka Exists
Apache Kafka began as an internal LinkedIn project in 2011, born from a familiar problem: as microservices proliferate, the number of direct connections between services and data stores grows combinatorially. LinkedIn wanted a single platform that could act as the central nervous system for all data flowing through the company.
The solution was to build a distributed system that serves as a source of truth for streaming data. Data flows in, gets processed or transformed, and is consumed by downstream systems like data warehouses, indices, and microservices. Kafka is optimized to handle massive throughput — millions of messages per second — while storing terabytes of data.
The Log at the Core
Topics are the logical containers for data in Kafka, and every topic is built on a log — a simple ordered structure that stores records sequentially. The log's properties define Kafka's fundamental behavior.
Because the log is immutable with O(1) reads and writes from its head or tail, access speed stays constant regardless of how large the log grows. That immutability also makes concurrent reads efficient. But the log's real advantage is that it maps perfectly to HDD performance: hard drives excel at linear reads and writes, which is precisely what the log structure demands.
Kafka's architecture was deliberately built around the cost economics of storage. HDDs have become dramatically cheaper per byte over time, and by optimizing for linear disk operations, Kafka can store large volumes of data cost-effectively while staying fast.
How Kafka Achieves Its Performance
A well-tuned on-premise Kafka deployment typically saturates the network before anything else, scaling to gigabytes per second of throughput. Several optimizations work together to reach this level.
Disk-First Design
Kafka writes every record to disk — it does not keep data in memory. The protocol batches messages together, reducing network overhead, and the broker persists those chunks in single linear writes. Consumers fetch large contiguous chunks as well.
The operating system amplifies these linear operations. Read-ahead prefetches large blocks into memory so subsequent reads don't touch the disk. Write-behind groups small logical writes into larger physical writes. Kafka skips fsync, allowing writes to land on disk asynchronously.
Zero-Copy and Pagecache
Because Kafka stores messages in a standardized binary format that remains unmodified from producer to broker to consumer, it can use the zero-copy optimization. The OS copies data directly from the pagecache to the socket, bypassing Kafka's JVM entirely and eliminating several copies plus user/kernel mode switches.
Zero-copy matters less than it might seem, though. CPU is rarely the bottleneck in a well-optimized Kafka deployment, and encryption via SSL/TLS — standard in production — modifies the message in transit, which prevents zero-copy from being used at all.
Brokers, Replicas, and Partitions
The nodes in a Kafka cluster are called brokers. Each topic splits into partitions, and each partition is replicated N times according to the configured replication factor, creating multiple replicas for durability and availability. A replica is nothing more than a set of files on disk, each embodying the log structure; records within get a monotonically increasing offset.
Replication is leader-based: one broker leads a partition at a time. A partition's replica set includes both in-sync replicas — those that have the latest data — and out-of-sync ones that lag behind.
Writes and Durability Configuration
Producers are clients that write data, and writes go only to the partition leader, which asynchronously replicates to followers. Producers choose their durability guarantee with the acks property:
acks=0— the producer considers the write successful immediately, without waiting for any broker response.acks=1— the leader acknowledges once it persists the record to disk.acks=all— the default — the producer gets a response only after all in-sync replicas have persisted the record.
To prevent a degraded situation where only one in-sync replica exists, the min.insync.replicas setting defines the minimum number of in-sync replicas that must acknowledge a write configured with acks=all.
Reads and Consumer Groups
Consumers are clients that read data and process it. They can read from any replica — typically the closest one in the network topology.
Consumers organize into consumer groups: logically grouped clients that coordinate through the broker rather than connecting directly to each other. Each consumer group persists its offset progress in a special Kafka topic called __consumer_offsets. The broker leading that topic's partition acts as the Group Coordinator, maintaining membership and liveness for the group.
Record order is guaranteed per partition, and the consumer group protocol ensures no two consumers within a group read from the same partition, preserving that ordering guarantee.
Multiple consumer groups can independently read from the same topic. This decoupling of producer and consumer was a key factor in Kafka's adoption over traditional message buses. Older systems deleted messages upon consumption, creating tight coupling — slow consumers could exhaust memory and hurt producers. Kafka's disk persistence avoids that failure mode entirely.
Controller and Leader Election
One broker in a Kafka cluster always acts as the active Controller. It handles administrative operations requiring a single source of truth: creating and deleting topics, adding partitions, and reassigning replicas. Most importantly, the Controller manages partition leader election, deciding when and to which broker leadership moves — particularly during broker failover or graceful shutdown.
From ZooKeeper to KRaft
Choosing one controller at any given time is a distributed consensus problem. Kafka historically delegated that responsibility to ZooKeeper: brokers raced to register the /controller zNode at startup, and ZooKeeper hosted metadata like the set of alive brokers, topic partition counts, and partition assignments. ZooKeeper's watch mechanism also notified subscribers of metadata changes.
Kafka has spent the past several years migrating away from ZooKeeper toward its own consensus protocol, KRaft. A dialect of Raft influenced by Kafka's existing replication protocol, KRaft builds on a key insight: cluster metadata can be expressed as an ordered log of events. Brokers replay those events to reconstruct the system's state.
In the KRaft model, a quorum of controllers (typically three) hosts a special single-partition topic called __cluster_metadata. Its partition leader is elected by Raft, and that leader becomes the active Controller. The other controllers stay hot with the latest metadata in memory. Regular brokers replicate this topic themselves, updating their metadata asynchronously instead of querying the Controller directly.
KRaft supports two deployment modes. In combined mode — similar to the ZooKeeper model — a single node serves as both a regular broker and a controller. In isolated mode, dedicated controller nodes handle nothing else.
Production-ready KRaft shipped with Kafka 3.3 in October 2022. ZooKeeper is scheduled for full removal in Kafka 4.0, expected around Q3 2024.
Moving Data Out of Broker Storage
Kafka’s original design kept all data on broker-local disks—a sound approach for on-premise clusters, but one that grows problematic at cloud scale. Brokers holding several terabytes of history are common; with a default replication factor of three, a 3TB logical partition set means 9TB of physical storage per broker. That volume of local data creates several failure-mode headaches.
Log recovery after an ungraceful shutdown requires rebuilding index files for every local partition. On a 10TB disk, that process can take hours or even days. Historical reads present another challenge: Kafka’s performance assumes consumers read from the log tail, which is served from page cache. Fetching older data forces disk reads, and HDDs have historically been stuck around 120 IOPS. Historical consumers therefore compete with producers for IOPS, and when that resource is exhausted, cluster performance degrades sharply.
Hardware failures amplify the issue. A broker with a failed disk must replicate its full 10TB from peers upon restart, subjecting those peers to sustained historical reads throughout the recovery—which can take up to a day. A full availability-zone failure multiplies that load across many brokers. Partition reassignment has similar consequences: moving a replica set from [0,1,2] to include new brokers requires the new follower to read the entire partition from the leader before it can become an in-sync replica.
Adding nodes to a cluster forces some reassignment or the new brokers remain empty. Each reassignment copies all data for the affected replicas, consuming IOPS and wall-clock time in rough proportion to partition size.
Tiered Storage, currently in Early Access, addresses these problems by separating storage into two tiers: hot local storage and cold remote storage (e.g., S3). Leader brokers handle tiering data into the object store; once data is tiered, both leaders and followers can serve historical reads from the remote tier. Brokers no longer need to copy massive datasets locally, and historical reads stop exhausting IOPS. Development tests showed a 43% producer performance improvement when historical consumers were present. Cost may also drop, since replication and durability guarantees shift to the object store.
Rebalancing and Cluster Management
Partition reassignment is essential in any Kafka cluster with meaningful usage. Hot spots and uneven resource distribution emerge naturally as client workloads shift. Kafka exposes a low-level API for reassigning partitions, but deciding what to move where is the hard part—it is essentially the NP-hard bin-packing problem.
Cruise Control, open-sourced from LinkedIn, automates this decision process. It reads broker metrics from a Kafka topic, builds an in-memory model of the cluster, and runs a greedy heuristic bin-packing algorithm to find an improved layout. The component then incrementally applies that layout via Kafka’s low-level reassignment API. Its logic is organized into configurable Goals, each with an assigned priority and a specific resource to balance.
Cruise Control continuously monitors metrics and triggers rebalances automatically when values drift outside configured thresholds. It also exposes APIs for adding or removing brokers, operations that necessarily involve moving replicas because Kafka brokers remain stateful even with Tiered Storage.
Kafka Connect
An event-driven architecture built around Kafka usually needs data flowing in from many systems (sources) and out to many others (sinks)—ElasticSearch, Snowflake, PostgreSQL, BigQuery, MySQL, and similar. Kafka Connect, part of the Apache project, provides a generic plug-and-play framework for these integrations.
The Connect runtime has two deployment modes:
- Standalone Mode—a single node, suited for development, testing, or small-scale loading.
- Distributed Mode—a cluster of nodes sharing the ingestion load.
Each Connect node is a Connect Worker, essentially a container executing plugin code. Community-developed plugins, called Connectors, handle fault tolerance, exactly-once processing, ordering, and other invariants that would be burdensome to implement from scratch. Workers use internal Kafka topics for configuration, status, and offset checkpointing, and they rely on Kafka’s Consumer Group protocol for failure handling and task assignment.
Users install Connectors on workers and manage them via a REST API. Each Connector creates tasks that move data in parallel, shielding users from the underlying exchange details. Two flavors exist:
- Source Connector—imports data from an external system into Kafka.
- Sink Connector—exports Kafka data to an external system.
In the diagram above, two Source connectors run in separate Connect clusters, each with its own workers, ingesting MongoDB/PostgreSQL data into Kafka. A separate Connect cluster with Sink connectors then moves that data from Kafka into Snowflake.
Kafka Streams
Stream processing involves reading continuous data from input topics, transforming it, and producing results to output topics or external services. Simple processing is possible with the producer/consumer APIs, but joins and other complex transformations call for the integrated Streams API library.
Kafka Streams, also part of Apache Kafka, is a client library offering a high-level API for real-time processing, transformation, and enrichment. It runs inside your application as a regular Kafka client—not on a broker—so it requires no separate cluster or complex deployment strategy. It scales out across multiple application instances similarly to consumer groups. When both input and output are Kafka topics, it supports exactly-once processing semantics.
The Road Ahead
Kafka first shipped in 2011, but development continues actively. The trend points toward standardization on the Kafka API with competition happening at the implementation layer. Confluent, founded by Kafka’s original creators, has built a cloud-native engine called Kora. RedPanda rewrote Kafka in C++, and WarpStream introduced an architecture that leans heavily on S3, avoiding replication and broker statefulness entirely.
Cloud offerings vary widely in approach: some vendors provide a proper serverless SaaS experience, while others still expect users to understand system internals and manage significant portions themselves. Kafka remains mature, widely adopted, and open source, with a community still innovating strongly more than a decade into its lifecycle.
