Serving High-Intent Buyer Context in Shopify Inbox
Shopify Inbox is the business chat app where merchants manage customer conversations across channels like Online Store Chat, Facebook Messenger, and Apple Business Chat. In 2020, over four million conversations flowed through Inbox, and 70 percent of them involved customers actively making a purchasing decision. The Shopify Data team wanted to help merchants recognize and convert those high-intent conversations.
The result is a real-time buyer signal data pipeline that surfaces cart activity and order completion details to merchants while they chat. With these signals, a merchant can see where a buyer is in the shopping journey — from browsing products to placing an order — without leaving the conversation. That context helps merchants ask sharper questions, answer accurately, and prioritize chats most likely to close.
What Signals We Surface
When a buyer is mid-conversation with a merchant, the pipeline currently emits two kinds of conversation events:
- Cart action event: details on the buyer’s latest cart actions, product information, and cart status.
- Order completion event: recent purchase information, including a link to the order in the Shopify admin.
These events are inserted chronologically into the conversation message flow so they appear naturally without overwhelming the merchant. Events are aggregated and shared under two time windows:
- Pre-conversation events: up to 14 days before a conversation starts.
- Post-conversation events: the seven-day active life of the conversation, after which state is expired.
Pipeline Architecture and Key Technologies
Two system requirements drove the design: low latency and high reliability. The pipeline is built on three core technologies:
- Apache Kafka for message queues
- Apache Beam for unified stream and batch processing
- Google Cloud Dataflow for managed execution
Message Queues: Monorail and CDC
Kafka carries two forms of events. Monorail is Shopify’s internal abstraction layer that adds a defined schema and versioning to raw Kafka events. If an upstream schema changes, the event is produced under the new version while the Kafka topic stays the same — making schema evolution safe and maintaining data integrity.
Change Data Capture (CDC) is used for inputs that aren’t naturally streaming. CDC reads MySQL binlogs via Debezium and converts changed rows into a stream of events, allowing the pipeline to handle large record delivery and ingest transactional database changes in real time.
Streaming with Apache Beam and Dataflow
Apache Beam unifies batch and streaming so we avoid maintaining separate systems for historical aggregation and live traffic. For transactional event data, accuracy matters: results must be correct despite late or out-of-order events. Beam provides windowing, timers, and stateful processing to handle these behaviors deterministically.
Running Beam on Google Dataflow offloads the physical orchestration of parallel processing to a managed service, letting the team focus on the logical composition of the data processing job.
System Design
The pipeline ingests from both CDC and Monorail topics; its sink writes only to a Monorail topic as the standardized handoff to downstream services. The consumer that serves Shopify Inbox listens to those produced Monorail events, structures them, and delivers them to the merchant.
The pipeline has two main components:
- Event filtering jobs: Cart and checkout sources emit tens of thousands of snapshot events per second, even off-peak. Filtering keeps only mission-critical transactional events for Inbox users, trimming workload and resource usage.
- Customer events aggregation job: This job contains the core logic. It maintains the latest snapshot of a buyer’s store activity — current carts, completed orders, and recent conversations. Stateful processing with timers in a global window keeps that state queryable at any moment; the rule to emit a signal triggers when a buyer initiates a conversation.
The aggregation job is where most of the engineering effort sits. As the next sections explain, the team tackled common streaming pain points there: transactional consistency, out-of-order data, and state expiration.
How the Aggregation Job Works
The customer events aggregation job ingests three input collections: filtered conversation, checkout, and cart events. Each element is keyed by the buyer's unique identifier on the store via the Apache Beam CoGroupByKey operator, which groups all inputs into a single Tuple collection for downstream processing.
To maintain historical context, the pipeline leverages Beam's ValueState, which stores values per key and window. Because state expires when a window ends, the job keeps events in a Global Window—unbounded and single-windowed—so state remains accessible at any time. Three separate states are maintained, one for each event stream: conversation, checkout, and cart. When new events arrive, a processing-time trigger emits the current window data as a pane. A PTransform then combines state from last-seen events with new pane data applying defined logic.
At this stage, processing answers three questions:
Does the buyer have an active conversation? The pipeline only emits output when a buyer has started a conversation with the merchant through Shopify Inbox. Cart and checkout events are otherwise processed and stored to state without producing signals.
Do events occur before or after a conversation? Aggregation depends on the event's timing relative to the conversation. Pre-conversation events are not shown to merchants. Post-conversation events—such as a cart addition or order completion after the buyer has messaged—are displayed as transactional context in the merchant's inbox.
What is the buyer's latest interaction? Signals must reflect the most recent activity, staying relevant to the conversation at hand. This is the core design constraint of the pipeline, and it introduces several engineering challenges.
Handling Dependency and Disorder
Cart and checkout events arrive from different Monorail sources and depend on each other. For instance, when an order is placed and the buyer returns, the cart should appear empty. A single PTransform can access all mutable states and perform cross-state logic—such as clearing the cart state upon receiving a checkout event for the same user token—to model the real purchase flow.
Output events are cumulative (total cart value, for example), so ordering matters. A removal must always follow an addition. But streaming sources do not guarantee event order across data streams, and the cart action is never declared explicitly. The action must be inferred by comparing quantity changes between transactional events.
This is resolved with stateful processing. Beam's state acts as a mutable buffer per key and window, evolving with time as new elements arrive. The job compares timestamps between new events and stored state to detect out-of-order arrivals and suppress outdated signals before they reach the merchant.
State Lifecycle and Cleanup
Beam's Timer with the event_time domain is used to expire irrelevant per-key-and-window state values manually. This accommodates the variable lifespan of a cart, ensuring that stale entries do not accumulate and burden the job.
Conversation and cart cookies have different life spans, and event characteristics can shift over time. A cart event shared as part of an active conversation may become a pre-conversation event once that conversation expires. To handle this, the pipeline maintains a dynamic tag in state indicating whether an event has already been shared. When the conversation-state timer fires, it resets this tag on the cart and checkout states so the context is accurately reclassified.
Measuring the Impact
Shopify validated the feature with a controlled A/B test. Merchants using Shopify Inbox were split evenly into control and treatment groups, the latter seeing buyer signal events in real time. Three metrics were tracked:
- Response rate—the percentage of buyer conversations receiving a merchant reply, which saw a significant increase of two percentage points.
- Response time—the interval between the first buyer message and first merchant response, which showed no significant change despite the response-rate increase.
- Conversion rate—the percentage of conversations attributed to a sale, which rose by a significant 0.7 percentage points.
The results demonstrated that having buyer context in real time let merchants answer queries more effectively and prioritize conversations with buyers already in the checkout flow. The feature had a positive impact across all metrics measured.
Key Takeaways
- Apache Beam is well suited to transactional use cases such as cart tracking due to its state management and timer functionality.
- Handling out-of-order events is essential to correctness, and that requires robust per-key state.
- Controlled experiments are a reliable method for evaluating the true effect of feature changes on user behavior.
