305 articles from Shopify Engineering.
At Shopify, we’ve long used database replication for redundancy and failure recovery, but only fairly recently started to explore the potential of replicas as an alternative read-only data source for applications.
We want to avoid having many hanging pennies. And when faced with such a situation, we want to identify such calculations and put a plan in place on who to notify and what to do with them.
For companies like Shopify that practice continuous deployment, our code is changing multiple times every day. We have to de-risk new features to ship safely and confidently without impacting the million+ merchants using our platform. Beta flags are one approach to feature development that gives us a number of notable advantages.
Authors: Sarah Naqvi and Steve Lounsbury Struggling with a concept and frantically trying to find the answers online? Are you thinking: I should just ping a teammate or Is this something I should already know? That’s fine, I’ll ask anyway. And then you do ask and get an answer and are feeling pretty darn proud of yourself and move forward. But wait, now I have about 25 follow-up questions. Sound f
A step-by-step guide to how you can create dashboards that are user-centred and impact-driven.
As a member of the Infrastructure Security team, Cailyn set out to create an application to silently and effectively manage SSH keys that persist inside of Shopify's Google Cloud Platform project-wide metadata.
GraphQL is the syntax that describes data that a client asks from a server. The client, in this case, is a mobile application. GraphQL is usually compared with REST API, a common syntax that most mobile application developers use. We will share how GraphQL can solve some of the pain points of REST API in mobile application development and discuss tips and best practices that we learned at Shopify
To create relevant search, processing clickstream data is key: you frequently want to promote search results that are being clicked on and purchased, and demote those things users don’t love. Typically search systems think of processing clickstream data as a batch job run over historical data, perhaps using a system like Spark. But on Shopify’s Discovery team, we ask the question: What if we could
At Shopify, we’re keeping the flexibility of untrusted Partner code, but executing it on our own infrastructure with WebAssembly.
On December 16, 2020 we held Shipit! presents: Performance Tips from the Storefront Renderer Team. A video for the event is now available for you to learn more about how the team optimized this Ruby application for the particular use case of serving storefront traffic. Click here to watch the video. By Celso Dantas and Maxime Vaillancourt In the previous post about our new storefront rendering eng
On January 27, 2021 Shipit!, our monthly event series, presented Building a Culture of Resiliency at Shopify. Learn about creating and maintaining resiliency plans for large development teams, testing and tooling, developing incident strategies, and incorporating and improving feedback loops. The video is now available. Each year, Black Friday Cyber Monday weekend represents the peak of activity f
In this post, we’ll outline the approach we took to reliably scale our data platform in preparation for Black Friday and Cyber Monday.
Shopify changes a lot. We merge around 400 commits to the main branch daily and deploy a new version of our core monolith 40 times a day. The Monolith is also big: 37,000 Ruby files, 622,000 methods, more than 2,000,000 calls. At this scale with a dynamic language, even with the most rigorous review process and over 150,000 automated tests, it’s a challenge to ensure everything works properly. Dev
With BFCM on the horizon, there is a lot of preparation that Shopify needs to do on our platform. This effort needs a key driver to set expectations for many teams and hold them accountable to complete the work.
By Mikko Haapoja and Stephan Leroux 2020 Black Friday Cyber Monday (BFCM) is over, and another BFCM Globe has shipped. We’re extremely proud of the globe, it focused on realism, performance, and the impact our merchants have on the world. The Black Friday Cyber Monday Live Map We knew we had a tall task in front of us this year, building something that could represent orders from our one million m
We cover our approaches to capacity planning, and how we rolled it out across the org and to dozens of teams. We’ll also share how we validated our capacity plans with scalability tests to make sure they work.
Insight into the tools Shopify uses to raise confidence in our ability to serve large sales events, as well as the experimentation and regression framework built to ensure that we’re getting better, week-over-week, at handling load.
If you were using computers in the ‘90s and the early 2000s, you probably had the experience of installing a piece of software you downloaded from the internet, only to discover that someone put some nasty into it, and now you’re dragging your computer to IT to beg them to save your data. To remedy this, software developers started “signing” their software in a way that proved both who they were a
I’ll show you how we moved to a SQL modelling workflow by leveraging dbt (data build tool) and created tooling for testing and documentation on top of it.
Presenting how Shopify got from no Sorbet in our monolith to almost full coverage in the span of a few months. I’ll explain the challenges we faced, the tools we built to solve them.
A look at how Shopify adopted static typing in it's core Ruby monolith by treating type checking like a product.
One line of code can 5x your Rails application performance. What if you could make a small change to your database design that would unlock massively more efficient data access? At Shopify, we dusted off some old database principles and did exactly that with the primary Rails application.
Developers constantly face information overload. There's always a new programming language, methodology, framework, or platform. How do you make sense of it all? You use mental models to guide you.
Shopify’s Online Store provides greater flexibility to build themes representing the brand of a merchant’s online store. However, like with any programming language, one often writes code without being aware of all the performance impact it creates. Whether it be performance impact on Shopify’s servers or observable performance impact on the browser, ultimately, it’s the customers that experience
A guide to building an email experimentation pipeline from the ground up. You'll learn how to implement a similar pipeline with a relatively simple setup from scratch.
A deep dive into a JavaScript-based module to scale HTML content like an image would inside of its container and a look at why it was developed.
A/B tests are not the only tool to understand causality: quasi-experiments and counterfactuals are powerful tools for causal inference if used right.
Packwerk is an open source static analysis tool used to enforce boundaries between groups of Ruby files we call packages.
It’s been 18 months since we talked about our Rails modular monolith. We lay out the current state of the work, and things we’d do differently if we started again.
To provide a more convenient and painless experience, we built a tool enabling React Native developers to quickly load their peer’s work within seconds.
In April, we announced Shop, our digital shopping assistant that brings together the best features of Arrive and Shop Pay. The Shop app started from our React Native codebase for our previous package tracking app Arrive, with every screen receiving a complete visual overhaul to fit the new branding. While our product designers worked on introducing a whole new design system that would decide the l
In this post, I’ll show how you can create Type 2 dimensional models using modern ETL tooling like PySpark and dbt (data build tool).
Shuting Chang, Robert Saunders, Karen Xie, and Vrishti Dutta join us to talk about Shopify’s API Health Report, the tool built to surface breaking changes affecting Shopify Partner apps.
We rewrote critical software that powers all online storefronts on Shopify, offering the fastest online shopping experience, entirely from scratch and without downtime.
Merchants worldwide rely on Shopify's Point Of Sale (POS) app to operate their brick and mortar stores. Unlike many mobile apps, the POS app is mission-critical. Any downtime leads to long lineups, unhappy customers, and lost sales. The POS app must be exceptionally reliable, and any outages resolved quickly. Reliability engineering is a well-solved problem on the server-side. Back-end teams are a
If you are lacking understanding of what is DNS, traffic management, or why we would even use it, read Part 1: Introduction to DNS traffic management. Distributed systems are only as resilient as we build them to be. Domain Name System (DNS) traffic management (TM) is a well-used approach to do so. In this second part of the two-part series, we’re sharing Shopify’s DNS traffic management journey f
Distributed systems are only as resilient as we build them to be. Domain Name System (DNS) traffic management is a well-used approach to do so. In this first part of a two-part series, we aim to give a broad overview of DNS and how it’s used for traffic management, as well as the different reasons why we want to use DNS traffic management. If you already have context on what is DNS, what is traffi
Using TypeScript enabled our ever-growing Admin teams to leverage TypeScript’s compiler to catch potential bugs and errors well before they ship.
On June 20, 2020, ShipIt!, our monthly event series, presented AR/VR at Shopify. Daniel Beauchamp, Head of AR/VR at Shopify talked about how we’re using this increasingly ubiquitous technology at Shopify.
Humans generate a lot of data. Every two days we create as much data as we did from the beginning of time until 2003! The International Data Corporation estimates the global datasphere totaled 33 zettabytes (one trillion gigabytes) in 2018. The estimate for 2025 is 175 ZBs, an increase of 430%. This growth is challenging organizations across all industries to rethink their data pipelines. The natu
On July 20, 2020 we held ShipIt! Presents: AR/VR at Shopify. Daniel talked about what we’re doing with AR/VR at Shopify. The video of the event is available. Size.link is a free tool built by Shopify’s Augmented Reality (AR) team that lets anyone view the size of a product in the space around them using their smartphone camera. The tool came out of our April Hack Days focused on helping retailers
Shopify recently launched the ability to add videos and 3d models to products as a native feature within the Admin section. Learn how we used pipelines to manage the long-term maintenance and understandability of the code.
With Reanimated 2, the team at Software Mansion asked: “How would it look if there were no constraints when writing gestures and animations in React Native?” and started from there.
I want to talk about one contribution that has takeaways for any Ruby developer. Here are 4 reasons for why we adopted Sorbet for static type checking.
Shopify’s Data team uses these foundational approaches to data warehousing and analysis empowering us to deliver the best results for our ecosystem.
Developers write tests to ensure correctness and allow future changes to be made safely. However, as the number of features grows, so does the number of tests. Tests are a double-edged sword. On one hand, well-written ones catch bugs and maintain a program’s stability, but as the code base grows, a high number of tests impedes scalability because they take a long time to run and increase the likel
I want to show you a little of what this sea-of-nodes graph data structure looks like and how it works, and I think it’s worth doing this for this for a couple of reasons.
On May 25, 2020, ShipIt!, our monthly event series, presented How Shopify Uses Nix. Building upon on my What is Nix post, I show how we rebuilt our developer tooling using Nix, and show off some of the tooling we actually use at Shopify on a day-to-day basis. I wasn't able to answer all the questions during the event, so I've included answers to those ones below. Would runix interop well with lorr
Here are some of the approaches we use to help data workshop participants walk away equipped and excited to start using data to power their work.
This article is going to be a crash course in what Nix is, how to think about it, and why it's such a valuable and paradigm-shifting piece of technology.
In this post, we’ll cover Shopify’s journey from manually provisioning TLS certificates to the fully automated system that supports over 1M businesses.
On April 24th, 2020, we proudly celebrated the graduation of our first Dev Degree program cohort. This milestone holds a special place in Shopify history because it’s a day born out of trial and error, experimentation, iteration, and hustle. The 2016 cohort had the honor and challenge of being our first class, lived through the churn and pivots of a newly designed program, and completed their educ
How do we effectively find out why our application is slow? Even if we have a fix for the slow code, how can we prove that our new code is faster?
By: Jeet Mehta and Kathy Ge With over 1M business owners now on Shopify, there are billions of products being created and sold across the platform. Just like those business owners, the products that they sell are extremely diverse! Even when selling similar products, they tend to describe products very differently. One may describe their sock product as a “woolen long sock,” whereas another may ha
We try to find the path to those goals that our developers want to take. We want to create experiences through tooling that can make our developers feel productive, and we want to do our best to make shipping feel like a celebration and not a chore.
Shopify is investing in React Native as our primary choice of mobile technology moving forward. As a part of this we’ve rewritten our package tracking app Arrive with React Native and launched it on Android—an app that previously only had an iOS version. One of the most cherished features by the users of the Arrive iOS app is the confetti that rains down on the screen when an order is delivered. T
Shopify's involvement with TruffleRuby began half a year ago, with the goal of furthering the success of the project and Ruby community. TruffleRuby is an alternative implementation of the Ruby language (where the reference implementation is CRuby, or MRI) developed by Oracle Labs. TruffleRuby has high potential in speed, as it is nine times faster than CRuby on optcarrot, a NES emulator benchmark
There are strategies for reducing the size and responsibilities of large objects. Here’s one that worked for us at Shopify, an all-in-one commerce platform supporting over 1M merchants across the globe.
I’ll talk about the engineering decision my team made to transform Kit from a rule based system to an artificially-intelligent assistant.
This guide will show you how to make a native component that provides bottom sheet functionality to React Native and renders React views inside.