Shopify uses Apache Flink as its standard stateful streaming engine for a range of data-heavy use cases. Following an earlier set of optimization notes for large stateful applications, the engineering team has published another round of practical guidance covering parallelism, sink bottlenecks, and mixed-source reads.

Parallelism Starts With One Setting

A Flink job is broken into parallel instances of its tasks, which include transformations, sources, and sinks. The number of those instances is the parallelism, and scaling it is a primary lever on job performance: more instances can use more task slots, raising throughput.

Parallelism can be set at several levels:

  • Operator level
  • Execution environment level
  • Client level
  • System level

Which one you choose depends on the application. When a specific operator is a known bottleneck, raising parallelism for just that operator makes sense. As a default, though, Shopify recommends starting with one execution-environment-level value and increasing it only when needed. Task slot sharing means a single value lets I/O-bound subtasks coexist efficiently with CPU-bound ones on the same task manager.

A practical rule for sizing: the number of task managers multiplied by slots per manager should equal, or slightly exceed, the highest parallelism in use. For a parallelism of 100 with four slots per manager, that means 25 task managers.

Keep Sinks Clear of Bottlenecks

Writing to an external destination—Bigtable, Kafka, and others—is frequently where Flink jobs slow down. A target system under heavy load, such as a Bigtable instance with high CPU utilization, can push backpressure through the entire pipeline without raising any exceptions. The Flink UI will show the effect on upstream operators.

Since backpressure from a sink propagates to all upstream dependencies, the sink should never be the limiting factor. Where higher latency is acceptable, batch writes are a useful countermeasure: collecting events into a bundle before submitting avoids per-event request overhead, improving compression, reducing network usage, and lowering CPU cost on the destination. Kafka's batch.size producer property and Bigtable's bulk mutations are examples of this pattern.

Data skew is the other common sink problem. Flink partitions keyed streams across operators according to a key, and a poorly distributed key leaves some task managers busy while others sit idle. Shopify's shop ID is an example of an unevenly distributed key because merchant traffic varies significantly. Low-cardinality keys (under roughly 100 distinct values) are just as problematic because they cannot spread evenly across task managers.

When you must keep a skewed key, a bucketing technique can help:

  1. Pick a maximum bucket number, starting at or below operator parallelism.
  2. Generate a random value between zero and that maximum.
  3. Append the value to the key before calling keyBy.

That spreads processing across more buckets per key. Downstream, once the volume per bucket is reduced, re-key by the original key to aggregate results, or combine at query time if the query engine supports it.

Reading History and Live Data as One Stream

Many Shopify Flink jobs read from and write to Kafka. To control storage cost, retention policies and expiration are enforced per topic. Some topics support archival instead: data is copied to cloud object storage before expiry, preserving it indefinitely.

Applications that need both the history and the live feed would normally require two sources, but that introduces two simultaneous event-time points and forces the job logic to handle ordering across them. Shopify's alternative is Flink's HybridSource, which presents both the archived object-store data and the live Kafka topic as a single logical source. The reader consumes the archive first, exhausting it, and Flink then switches to the real-time topic automatically. To the developer, this appears as one uninterrupted DataStream covering the topic's full history.

That approach has a throughput benefit, too. Archival datasets are typically partitioned across thousands of segments per day, far more partitions than a live Kafka topic uses for real-time load. When paired with sufficient task managers, HybridSource lets Flink scan historical data considerably faster than reading the same volume from a less-partitioned Kafka topic. Shopify wraps this logic in a KafkaBackfillSource construct so the archive source is inferred from topic and cluster, keeping the stream abstraction clean. If a Flink application must consume heterogeneous sources in order, HybridSource is worth evaluating.