The Evolution of Kit: Automating Marketing Using Machine Learning - Shopify

From Hardcoded Rules to Learned Recommendations

Kit is Shopify’s virtual marketing assistant, reachable through Shopify Ping and SMS, that helps merchants create Facebook and Instagram ads, run email automations, and connect with other marketing apps. The original version guided users through campaign creation using hardcoded heuristic rules: it limited product recommendations to a few popular categories, offered a fixed set of budget ranges, and defaulted every other configuration to best practices. That approach reduced the cognitive load of the Facebook Ads Manager, but it treated every business the same.

The team behind Kit decided to replace those static rules with a machine learning system that predicts the right next marketing move for each merchant. The shift meant answering two questions for every recommendation: how much is the merchant willing to spend, and will that budget be enough to generate sales given their store’s current traffic and order history?

Budget Prediction as Two ML Problems

The bucket of budgeting was the first thing to go. Instead of asking merchants to pick from a preset list, Kit would predict the amount they are likely to spend and, from the predicted range, find the figure with the best chance of producing sales.

That single proactive recommendation embodies two machine learning tasks:

• Regression — from the merchant’s historical ad spending, estimate a budget range they are comfortable with.
• Classification — given the predicted budget and store attributes such as recent visitors and orders, estimate the probability of a sale.

The years of rule-based interactions provided the dataset necessary to train those models. Every Facebook campaign created through Kit contributed a record of spend against store performance at the time of creation.

A Two-Flow Architecture on Google Cloud

Replacing the app’s hardcoded logic required a new system design. The architecture in the second version of Kit splits into a training flow and a prediction flow, running on the Google Cloud Platform.

Training flow builds and validates the regression and classification models:

1. Aggregate features — collect historical Facebook campaigns created through Shopify, plus the store’s state (traffic, sales) when that campaign ran.
2. Feature engineering — derive useful predictors from the raw data: features like average ad spend over the past 30 days, marketing sales over the past 30 days, unique visitors over the past 30 days, and total orders over the past 30 days. Apache Spark’s distributed computation handles Shopify’s large data volume.
3. Train models — use Google Cloud ML Engine, which supports popular frameworks such as scikit-learn and TensorFlow.
4. Monitor model metrics — compare predicted values against ground truth to catch broken feature pipelines or shift underlaying data distribution. Alerting uses thresholds plus z-score-based outlier detection when metrics deviate from their normal distribution.
5. Persist the trained models — store them for the prediction path.

Prediction flow turns the trained models into merchant-facing recommendations:

1. Generate a marketing recommendation by predicting the ideal budget and the likelihood of a sale given store state.
2. Deliver the recommendation to Kit via Apache Kafka.

Shopify’s data platform engineering team maintains the data services underpinning both flows, which lets the Kit product team focus on the domain-specific machine learning pipeline. The result is a one-step conversation: rather than choosing products and budget, the merchant sees a recommendation like “spend about $40 to market your new products” — with the budget explained by their past behavior and the sale likelihood by the current state of their store.

Real-Time Recommendations for Audience Building

When a merchant like Cheryl has no existing store traffic, the first marketing step is acquiring new visitors who can later be used to build lookalike audiences — groups of prospects that resemble the merchant’s current customers. For these cases, Kit must solve a different problem: it has to estimate how many new visitors are required to create a meaningful lookalike audience, what budget is appropriate to acquire them, and whether that budget will actually be sufficient given the merchant’s current state.

The second and third decisions reuse the same machine learning architecture described for standard campaign budgets. But the first introduces fresh complexity: store traffic changes constantly, so the prediction must be computed at the moment the recommendation is delivered, not in advance.

Why the Batch Approach Broke Down

The original Spark-based prediction flow processed recommendations in scheduled batches. That meant the model ran on a fixed schedule rather than being invoked when Kit actually needed an answer for a merchant. By the time a budget suggestion reached the merchant, it could already be outdated. The fix required a dedicated real-time prediction service that Kit could call directly when generating a recommendation.

The real-time flow works as follows:

1. Based on the merchant’s store state, Kit determines that the marketing objective is audience building and sends a request to the prediction service. The request hits an HTTP API exposed through the web container component.
2. The web container produces marketing recommendations using the same features and models prepared in the training flow, but with several new design considerations:
   • Features generated during the feature engineering stage are immediately loaded into a key value store backed by Google Cloud’s Bigtable. This ensures efficient access to minimize prediction latency.
   • Since complex model prediction can be computationally expensive, the team uses Google’s TensorFlow Serving — a high-performance serving system for machine learning models in production environments. It integrates directly with TensorFlow models generated during the training flow with minimal configuration.
   • By offloading the CPU/GPU-heavy prediction operations to TensorFlow Serving, the web container stays lightweight, holding only the business logic for generating recommendations. The team chose Tornado, a Python web framework with non-blocking network I/O that scales to tens of thousands of open connections for model predictions.
3. Model predictions are delegated to the TensorFlow Serving container.
4. That container preloads the models created during training and performs predictions when requests arrive.

Scaling to One-Third of Kit’s Campaigns

Kit began as a rules-based application that applied common marketing best practices to simplify and automate campaigns for Shopify merchants. Machine learning recommendations were introduced gradually — first via a Spark-based prediction process that was already well supported within Shopify. This established the value of machine learning in improving user engagement and marketing outcomes while giving the team an opportunity to productionalize an end-to-end pipeline (training and prediction) serving tens of thousands of merchants.

Monitoring proved essential to maintaining the integrity of the system. The move to a real-time prediction architecture solved the use cases that demanded time-sensitive recommendations, despite adding two new containers (web and TensorFlow Serving) to operate. The heavy model prediction work was delegated to TensorFlow Serving, which is well supported by Google and integrated easily with Shopify’s existing cloud infrastructure. That left the team free to focus on defining and implementing the core business logic for marketing recommendations in the web container.

The results speak to the architecture’s value: machine learning driven recommendations now power one third of all Kit marketing campaigns. Kit continues to refine its automation capabilities, optimizing for different marketing tactics and objectives to accommodate the diverse needs of merchants.