Shipping Data Products Under Deadline Pressure
Launching a new product is difficult under any circumstances; under a tight deadline it becomes a serious challenge. That holds just as true for data-centric products—forecasts, classification tools, dashboards—as for anything else. Shopify’s Data Science team routinely builds data products for merchants, from onboarding through first sale and beyond. Because commerce moves quickly, the team has had to develop a repeatable approach for shipping fast without cutting corners. The following four practices have proven effective.
Run a Timeboxed Design Sprint
Investing upfront in a design sprint pays off when deadlines loom. The Google Ventures design sprint methodology—structured time for answering critical business questions through design, prototyping, and customer testing—is well suited to getting data products off the ground quickly because it forces a team to focus on a single problem for a defined period. Shopify’s data science teams use sprints regularly, especially under time pressure. Their sprint setup follows four steps:
- Choose an impactful problem. Start by asking what problem needs solving and why. Good questions uncover merchant pain points, point toward the right technical solution, and ultimately drive impact.
- Keep the team small. A sprint team should have no more than six or seven highly motivated people. Smaller groups maintain better communication and alignment, which makes moving fast easier.
- Name a sprint champion. One person should own the project direction and make judgement calls when choices arise. A designated champion removes ambiguity so the rest of the team can focus on execution.
- Set fixed dates. Timeboxing is what makes sprints effective. A typical sprint runs up to five days; a three-day sprint is often enough to build a first version of something like a COVID-19 impact dashboard for an acquisition funnel.
During the sprint, keep an open mind about solutions. Sometimes a data-centered problem is solved more simply without a complex data approach—for example, asking a user for their preferred location instead of inferring it with a heuristic.
Prototype Before You Polish
First iterations of a new product go through many changes. Prototyping enables quick, inexpensive learning cycles and helps avoid the sunk cost fallacy, where past investment becomes the rationale for continuing down a wrong path.
For data work, spreadsheets are a surprisingly good prototyping tool. They are often underused by data scientists and engineers, but are designed to make sense of messy data quickly. Built-in spreadsheet functions handle the most common needs:
- rapid manual data cleaning
- chart generation
- basic ranking index computation
- output formatting
A robust production system is tempting, but building one takes time. When releasing a brand new product on a tight timeline, the priority needs to be fast prototype development.
Speed does not mean sloppiness, though. The first iteration of Shopify’s Marketing attribution tool for the Revenue team was a handful of SQL queries automated by a bash script, with output formatted in a spreadsheet. That lightweight setup made it easy to iterate and compare against off-the-shelf tools. It also meant no wasted effort on dashboards or production code, and no emotional attachment to work that might be discarded—which made it easier for the best solution to win.
Skip Machine Learning for the First Pass
When a data product is supposed to be “smart,” the temptation to spend weeks on a flashy machine learning algorithm is strong. That is precisely the wrong instinct for a first iteration. A lead scoring system for Sales Representatives is a case in point: the team spent 80% of the sprint gathering features and training a model, leaving almost no time to integrate with the existing customer relationship management (CRM) infrastructure, polish the experience, or solicit feedback. A simple ranking based on a proxy metric would have shipped much sooner, and the freed-up time could have gone toward understanding how users engaged with the tool.
The lesson stuck. When Shopify built a sales forecasting tool next, the team started with a linear regression using just two input variables, and had a prototype ready in a couple of hours. That simple model shipped fast and quickly demonstrated whether it solved the user’s problem. Only after confirming the direction did they invest in a more complex machine learning model.
The general rule: build models that solve problems and can ship quickly. Once the product has proven its impact, then invest in more complex modeling.
Talk to Users — With Purpose
Shipping fast also means shipping the right product. Gathering feedback before, during, and after each build iteration is essential. Shadowing users, or doing the task yourself, is a strong return on investment. But feedback collection is a skill, and two practices in particular improved the quality of what Shopify received:
- Ask specific questions. “Do you have any feedback?” gives users no direction. Asking, for example, “How do you feel about the speed at which the dashboard loads?” or “Are you able to locate the insights you need on this dashboard to report on top of funnel performance?” yields much richer responses.
- Get a diverse user group. If a dashboard serves three regional teams, ask one person from each team for feedback rather than five people from one.
Applying those tips to the sales forecasting tool surfaced a real usability issue: users found a numerical score from 0 to 100 confusing, since the difference between scores was unclear. When the team switched to letter grades (A, B, C), users interpreted the output far more quickly, and the overall experience improved.
Following these practices gives a clear path to launching new data products under tight deadlines, and helps avoid the common traps: endless design cycles, overengineered machine learning systems, and data products that no one actually uses. When the clock is ticking on a new data product, the checklist is short: run a focused design sprint, prototype early and often, resist machine learning on the first pass, and keep conversations with users specific and diverse.
