
There’s a way to build a model for your life and the other demands on you. It doesn’t have to be 100 hours a week. It might not even be 40 hours a week. Whatever sustainable model you come up with, prioritize fiercely and communicate expectations accordingly. I’ve worked this way for three decades through different phases of my life. The hours have changed, but the basic model and principles remai

Have you ever been handed a dataset and then been asked to describe it? When I was first starting out in data science, this question confused me. My first thought was “What do you mean?” followed by “Can you be more specific?” The reality is that exploratory data analysis (EDA) is a critical tool in every data scientist’s kit, and the results are invaluable for answering important business questio

Resiliency is important at Shopify and although we have many tools to help us during incidents, some problems need to be fixed at the database level. At Shopify, we use ProxySQL to manage connections between applications and our MySQL databases
SEShopify Engineering·April 20, 2021Databases 
We had to determine how to leverage React Native while still using iOS and Android native features, and when and how we write native code.
SEShopify Engineering·April 16, 2021Frontend 
In 2019 we set out to rewrite the Shopify storefront implementation. Our goal was to make it faster. We talked about the strategies we used to achieve that goal in a previous post about optimizing server-side rendering and implementing efficient caching. To build on that post, I’ll go into detail on how the Storefront Renderer team tweaked our load balancers to shift traffic between the legacy sto

The three-step framework Shopify's Data Science & Engineering team built for evaluating new search algorithms.

Ruby on Rails is a web framework that contains many libraries you’d need to create and deploy a successful web application. We often take for granted the ability to run rails new to create a fully functional web application with tons of built-in features.
SEShopify Engineering·March 26, 2021SRE & Ops 
Shopify Core is a Ruby on Rails monolith that loads the Shopify admin along with other services such as checkout, APIs, etc. Hundreds of developers work with a continuous deployment cycle that ultimately results in a platform used by over 1.7 million merchants.
SEShopify Engineering·March 19, 2021SRE & Ops 
Shopify’s data warehouse has gone through many iterations since the company's founding in 2004. Since then, the data warehouse has evolved and grown into a data lake, compromising multiple storage mechanisms, systems, and consumers.

For three years we, Shopify’s Application Security team, have set aside time to reflect on our bug bounty program and share recent insights. This past year has been quite a ride, as our program has been busier than ever! We’re excited to share what we have learned, and share some of the great things we have planned!
SEShopify Engineering·March 11, 2021Security 
Shipping a new data-centric product is hard, and under tight time constraints it's even harder. Here are 4 tips that are proven to help you ship new products fast.

How Shopify’s Test Infrastructure team was able to reduce the p95 of Shopify’s core monolith CI from 45 minutes to 18.

At Shopify, we’ve long used database replication for redundancy and failure recovery, but only fairly recently started to explore the potential of replicas as an alternative read-only data source for applications.
SEShopify Engineering·February 22, 2021Databases 
We want to avoid having many hanging pennies. And when faced with such a situation, we want to identify such calculations and put a plan in place on who to notify and what to do with them.

For companies like Shopify that practice continuous deployment, our code is changing multiple times every day. We have to de-risk new features to ship safely and confidently without impacting the million+ merchants using our platform. Beta flags are one approach to feature development that gives us a number of notable advantages.
SEShopify Engineering·February 12, 2021SRE & Ops 
Authors: Sarah Naqvi and Steve Lounsbury Struggling with a concept and frantically trying to find the answers online? Are you thinking: I should just ping a teammate or Is this something I should already know? That’s fine, I’ll ask anyway. And then you do ask and get an answer and are feeling pretty darn proud of yourself and move forward. But wait, now I have about 25 follow-up questions. Sound f

A step-by-step guide to how you can create dashboards that are user-centred and impact-driven.

As a member of the Infrastructure Security team, Cailyn set out to create an application to silently and effectively manage SSH keys that persist inside of Shopify's Google Cloud Platform project-wide metadata.
SEShopify Engineering·January 27, 2021Security 
GraphQL is the syntax that describes data that a client asks from a server. The client, in this case, is a mobile application. GraphQL is usually compared with REST API, a common syntax that most mobile application developers use. We will share how GraphQL can solve some of the pain points of REST API in mobile application development and discuss tips and best practices that we learned at Shopify

To create relevant search, processing clickstream data is key: you frequently want to promote search results that are being clicked on and purchased, and demote those things users don’t love. Typically search systems think of processing clickstream data as a batch job run over historical data, perhaps using a system like Spark. But on Shopify’s Discovery team, we ask the question: What if we could

At Shopify, we’re keeping the flexibility of untrusted Partner code, but executing it on our own infrastructure with WebAssembly.
SEShopify Engineering·December 18, 2020Frontend 
On December 16, 2020 we held Shipit! presents: Performance Tips from the Storefront Renderer Team. A video for the event is now available for you to learn more about how the team optimized this Ruby application for the particular use case of serving storefront traffic. Click here to watch the video. By Celso Dantas and Maxime Vaillancourt In the previous post about our new storefront rendering eng

On January 27, 2021 Shipit!, our monthly event series, presented Building a Culture of Resiliency at Shopify. Learn about creating and maintaining resiliency plans for large development teams, testing and tooling, developing incident strategies, and incorporating and improving feedback loops. The video is now available. Each year, Black Friday Cyber Monday weekend represents the peak of activity f
SEShopify Engineering·December 10, 2020SRE & Ops 
In this post, we’ll outline the approach we took to reliably scale our data platform in preparation for Black Friday and Cyber Monday.

Shopify changes a lot. We merge around 400 commits to the main branch daily and deploy a new version of our core monolith 40 times a day. The Monolith is also big: 37,000 Ruby files, 622,000 methods, more than 2,000,000 calls. At this scale with a dynamic language, even with the most rigorous review process and over 150,000 automated tests, it’s a challenge to ensure everything works properly. Dev
SEShopify Engineering·December 7, 2020SRE & Ops 
With BFCM on the horizon, there is a lot of preparation that Shopify needs to do on our platform. This effort needs a key driver to set expectations for many teams and hold them accountable to complete the work.

By Mikko Haapoja and Stephan Leroux 2020 Black Friday Cyber Monday (BFCM) is over, and another BFCM Globe has shipped. We’re extremely proud of the globe, it focused on realism, performance, and the impact our merchants have on the world. The Black Friday Cyber Monday Live Map We knew we had a tall task in front of us this year, building something that could represent orders from our one million m

We cover our approaches to capacity planning, and how we rolled it out across the org and to dozens of teams. We’ll also share how we validated our capacity plans with scalability tests to make sure they work.

Insight into the tools Shopify uses to raise confidence in our ability to serve large sales events, as well as the experimentation and regression framework built to ensure that we’re getting better, week-over-week, at handling load.

If you were using computers in the ‘90s and the early 2000s, you probably had the experience of installing a piece of software you downloaded from the internet, only to discover that someone put some nasty into it, and now you’re dragging your computer to IT to beg them to save your data. To remedy this, software developers started “signing” their software in a way that proved both who they were a
SEShopify Engineering·December 1, 2020SRE & Ops 
I’ll show you how we moved to a SQL modelling workflow by leveraging dbt (data build tool) and created tooling for testing and documentation on top of it.
SEShopify Engineering·November 19, 2020Databases 
Presenting how Shopify got from no Sorbet in our monolith to almost full coverage in the span of a few months. I’ll explain the challenges we faced, the tools we built to solve them.

A look at how Shopify adopted static typing in it's core Ruby monolith by treating type checking like a product.

One line of code can 5x your Rails application performance. What if you could make a small change to your database design that would unlock massively more efficient data access? At Shopify, we dusted off some old database principles and did exactly that with the primary Rails application.
SEShopify Engineering·October 29, 2020Databases 
Developers constantly face information overload. There's always a new programming language, methodology, framework, or platform. How do you make sense of it all? You use mental models to guide you.

Shopify’s Online Store provides greater flexibility to build themes representing the brand of a merchant’s online store. However, like with any programming language, one often writes code without being aware of all the performance impact it creates. Whether it be performance impact on Shopify’s servers or observable performance impact on the browser, ultimately, it’s the customers that experience

A guide to building an email experimentation pipeline from the ground up. You'll learn how to implement a similar pipeline with a relatively simple setup from scratch.

A deep dive into a JavaScript-based module to scale HTML content like an image would inside of its container and a look at why it was developed.
SEShopify Engineering·October 7, 2020Frontend 
A/B tests are not the only tool to understand causality: quasi-experiments and counterfactuals are powerful tools for causal inference if used right.

Packwerk is an open source static analysis tool used to enforce boundaries between groups of Ruby files we call packages.

It’s been 18 months since we talked about our Rails modular monolith. We lay out the current state of the work, and things we’d do differently if we started again.

To provide a more convenient and painless experience, we built a tool enabling React Native developers to quickly load their peer’s work within seconds.
SEShopify Engineering·September 9, 2020Frontend 
In April, we announced Shop, our digital shopping assistant that brings together the best features of Arrive and Shop Pay. The Shop app started from our React Native codebase for our previous package tracking app Arrive, with every screen receiving a complete visual overhaul to fit the new branding. While our product designers worked on introducing a whole new design system that would decide the l
SEShopify Engineering·September 2, 2020Frontend 
In this post, I’ll show how you can create Type 2 dimensional models using modern ETL tooling like PySpark and dbt (data build tool).

Shuting Chang, Robert Saunders, Karen Xie, and Vrishti Dutta join us to talk about Shopify’s API Health Report, the tool built to surface breaking changes affecting Shopify Partner apps.

We rewrote critical software that powers all online storefronts on Shopify, offering the fastest online shopping experience, entirely from scratch and without downtime.

Merchants worldwide rely on Shopify's Point Of Sale (POS) app to operate their brick and mortar stores. Unlike many mobile apps, the POS app is mission-critical. Any downtime leads to long lineups, unhappy customers, and lost sales. The POS app must be exceptionally reliable, and any outages resolved quickly. Reliability engineering is a well-solved problem on the server-side. Back-end teams are a

If you are lacking understanding of what is DNS, traffic management, or why we would even use it, read Part 1: Introduction to DNS traffic management. Distributed systems are only as resilient as we build them to be. Domain Name System (DNS) traffic management (TM) is a well-used approach to do so. In this second part of the two-part series, we’re sharing Shopify’s DNS traffic management journey f

Distributed systems are only as resilient as we build them to be. Domain Name System (DNS) traffic management is a well-used approach to do so. In this first part of a two-part series, we aim to give a broad overview of DNS and how it’s used for traffic management, as well as the different reasons why we want to use DNS traffic management. If you already have context on what is DNS, what is traffi

Using TypeScript enabled our ever-growing Admin teams to leverage TypeScript’s compiler to catch potential bugs and errors well before they ship.
SEShopify Engineering·July 29, 2020Frontend 
On June 20, 2020, ShipIt!, our monthly event series, presented AR/VR at Shopify. Daniel Beauchamp, Head of AR/VR at Shopify talked about how we’re using this increasingly ubiquitous technology at Shopify.

Humans generate a lot of data. Every two days we create as much data as we did from the beginning of time until 2003! The International Data Corporation estimates the global datasphere totaled 33 zettabytes (one trillion gigabytes) in 2018. The estimate for 2025 is 175 ZBs, an increase of 430%. This growth is challenging organizations across all industries to rethink their data pipelines. The natu

On July 20, 2020 we held ShipIt! Presents: AR/VR at Shopify. Daniel talked about what we’re doing with AR/VR at Shopify. The video of the event is available. Size.link is a free tool built by Shopify’s Augmented Reality (AR) team that lets anyone view the size of a product in the space around them using their smartphone camera. The tool came out of our April Hack Days focused on helping retailers

Shopify recently launched the ability to add videos and 3d models to products as a native feature within the Admin section. Learn how we used pipelines to manage the long-term maintenance and understandability of the code.

With Reanimated 2, the team at Software Mansion asked: “How would it look if there were no constraints when writing gestures and animations in React Native?” and started from there.
SEShopify Engineering·July 2, 2020Frontend 
I want to talk about one contribution that has takeaways for any Ruby developer. Here are 4 reasons for why we adopted Sorbet for static type checking.

Shopify’s Data team uses these foundational approaches to data warehousing and analysis empowering us to deliver the best results for our ecosystem.

Developers write tests to ensure correctness and allow future changes to be made safely. However, as the number of features grows, so does the number of tests. Tests are a double-edged sword. On one hand, well-written ones catch bugs and maintain a program’s stability, but as the code base grows, a high number of tests impedes scalability because they take a long time to run and increase the likel

I want to show you a little of what this sea-of-nodes graph data structure looks like and how it works, and I think it’s worth doing this for this for a couple of reasons.

On May 25, 2020, ShipIt!, our monthly event series, presented How Shopify Uses Nix. Building upon on my What is Nix post, I show how we rebuilt our developer tooling using Nix, and show off some of the tooling we actually use at Shopify on a day-to-day basis. I wasn't able to answer all the questions during the event, so I've included answers to those ones below. Would runix interop well with lorr