Nerd Out on 10 of Our Favorite Posts From 2021 - Shopify
A curated selection of our favorite 2021 posts, spanning technologies & engineering disciplines, with a few classics you still love, read, and share.
Shopify’s Unique Data Science Hierarchy Of Needs - Shopify
You’ve probably seen the “Data Science Hierarchy of Needs” (pictured below) before. Inspired by Maslow, it shows the tooling you would use at different levels in data science—from logging and user-generated content at the bottom, to AI and deep learning at the very top. While hierarchies like this one can serve as helpful guides, at Shopify, we don’t think it always captures the whole picture. For
Building a Real-time Buyer Signal Data Pipeline for Shopify Inbox - Shopify
Shopify Data built a real-time buyer signal data pipeline to show relevant customer information to merchants while they’re chatting with their customers.
Scaling Shopify's BFCM Live Map: An Apache Flink Redesign - Shopify
A deep dive into how Shopify Data revamped the data infrastructure powering our BFCM live map using Apache Flink.
Upgrading MySQL at Shopify - Shopify
Learn how the Database Platform team performed the most recent MySQL upgrade at Shopify and how this changed our upgrade guidelines moving forward.
Remote rendering: Shopify’s take on extensible UI - Shopify
A deep dive into the latest generation of technology that allows developers to extend Shopify’s UI.
Building an App Clip with React Native - Shopify
Being the first to build an App Clip in React Native that was going to be surfaced to millions of Shopify's Shop app users each day proved to be a challenging task.
Five Tips for Growing Your Engineering Career - Shopify
Five practical tips to help you grow and navigate your engineering career.
Using Propensity Score Matching to Uncover Shopify Capital’s Effect on Business Growth - Shopify
In this study, we prove how Shopify Capital (one of our first machine learning products) has real-world impact by making merchant funding accessible.
Building Blocks of High Performance Hydrogen-powered Storefronts - Shopify
The future of commerce is dynamic, contextual, and personalized. Hydrogen is a React-based framework for building custom and creative storefronts powered by Shopify’s platform and APIs.
The Vitality of Core Web Vitals - Shopify
A deep dive into each metric of Google Core Web Vitals.
Always on Calibration with a Quarterly Summary - Shopify
Learn how an always-on approach to career development improves your team. Leverage quarterly calibrations, weekly syncs, and more. Template available!
The AWARE Development Plan - Shopify
Learn how to set, track and reach your development goals. Access templates for Engineering Managers or Individual Contributors throughout your career.
Shopify's Path to a Faster Trino Query Execution: Custom Verification, Benchmarking, and Profiling Tooling - Shopify
Shopify’s Data Reliability team built custom verification, benchmarking, and profiling tooling for testing and analyzing Trino. Our tooling is designed to minimize the risk of various changes at scale.
Debugging Systems in the Cloud: MySQL, Kubernetes, and Cgroups - Shopify
An overview of how we investigated and solved the issue of some Kubernetes Pods running MySQL starting up and shutting down slower than other similar Pods with the same data set.
GitHub Does My Operations Homework: A Ruby Speed Story - Shopify
Learn how fast YJIT really is. This posts walks through Shopify's new benchmarking harness for YJIT, plus graphs and benchmarks updated twice a day.
Try Out YJIT for Faster Rubying - Shopify
Shopify is building a new JIT implementation on top of CRuby. This post walks through how to build CRuby and all about building YJIT.
YJIT: Building a New JIT Compiler for CRuby - Shopify
A team of skilled engineers from Shopify and GitHub on YJIT, a new Just-in-time (JIT) compiler built inside CRuby.
Winning AI4TSP: Solving the Travelling Salesperson Problem with Self-programming Machines - Shopify
Shopify optimization expert Meinolf Sellmann collaborated with his former colleagues to win the AI4TSP Competition that focuses on solutions to the Travelling Salesperson Problem.
Journey Through a Dev Degree Intern’s First Placement - Shopify
Carmela, Dev Degree student, shares how she learned to appreciate the stages of learning, dispel the myth of the “real developer”, combat imposter syndrome by asking for help, and the power of valuing representation.
Reusing Code with React Native Packages at Shopify - Shopify
At Shopify, we develop a bunch of different React Native mobile apps: Shop, Inbox, Point of Sale, Shopify Mobile, and Local Delivery. These apps represent different business domains, but they often have shared pieces of functionality like login or foundational blocks they build upon. Wouldn’t it be great to leverage development speed and focus on important product features by reusing code other te
Shard Balancing: Moving Shops Confidently with Zero-Downtime at Terabyte-scale - Shopify
This post explains how we’re able to balance our MySQL shards by migrating shops across shards—entirely online and with virtually zero consumer-facing downtime.
Making Shopify’s Flagship App 20% Faster in 6 Weeks Using a Novel Caching Solution - Shopify
We built a custom write-through cache for the Shop app's home feed that reduced database load by 15% and overall app latency by about 20%.
Using Rich Image and Text Data to Categorize Products at Scale - Shopify
We reevaluated our existing product categorization model to ensure we’re understanding what our merchants are selling, to build the best products that help power their sales.
A Kotlin Style .copy Function for Swift Structs - Shopify
Swift’s struct isn’t as convenient as Kotlin’s data class because it doesn't have a similar copy function. Learn how to adopt this pattern in Swift.
5 Steps for Building Machine Learning Models for Business - Shopify
We share five steps on approaching machine learning from a business perspective that helped to build and scale our suite of machine learning products.
Diggin’ and Fetchin’ with TruffleRuby - Shopify
It begins with a clever Ruby trick to extract a value from a deeply nested hash we want to refactor and ends with a prototype changing the language itself.
Modelling Developer Infrastructure Teams - Shopify
I’ve been managing Developer Infrastructure teams (alternatively known as Developer Acceleration, Developer Productivity, and other such names) for almost a decade now. Developer Infrastructure (which we usually shorten to “Dev Infra”) covers a lot of ground, especially at a company like Shopify that’s invested heavily in the area, from developer environments and continuous integration/continuous
Bridging the Gap Between Developers and End Users - Shopify
In every business, regardless of industry, it is important to keep an open communication line between the consumer and the producer. In the tech world, this line of communication comes in the form of customer support, chatbots, or contact forms.
Understanding GraphQL for Beginners–Part Three - Shopify
In Understanding GraphQL for Beginners–Part Three, we learn what are mutations and create mutation queries to modify data.
Connecting with Mob Programming - Shopify
Mob programming builds on pair programming but instead of two people pairing, it's an entire mob—more than two people—pairing on the same problem.
A Guide to Running an Engineering Program - Shopify
Shopify has a playbook for huge engineering programs in the past that cross over multiple areas of complexity on the platform, and we’d like to share it with you.
Perspectives on React Native from Three Shopify Developers - Shopify
We hope that by sharing our experiences learning React Native, we can help soothe your anxieties and empower you to explore this exciting technology.
Shopify-Made Patterns in Our Rails Apps - Shopify
At Shopify, we’ve developed our own patterns in order to support our global platform. Before coming here, I've developed multiple Ruby (and Rails) applications at multiple growth stages. Because of that, I quickly came to appreciate some workarounds and automation that were created to support the large codebase of Shopify. If there’s something I appreciate about Ruby on Rails, it’s the principle o
Shopify's Path to a Faster Trino Query Execution: Infrastructure - Shopify
We’ll discuss how we scaled our interactive query infrastructure to handle the rapid growth of our datasets, while enabling a query execution time of less than 5 seconds.
High Availability by Offloading Work Into the Background - Shopify
High availability and short response times are necessary, making background jobs an indispensable tool regardless of the application’s scale.
Understanding GraphQL for Beginners–Part Two - Shopify
In the second part of Understanding GraphQL for Beginners, we’ll learn how to create a GraphQL object and root fields.
Understanding GraphQL for Beginners–Part One - Shopify
As technology evolves, one thing that contrasts REST and is gaining popularity is GraphQL. But what is it exactly? Within this post, we will learn what GraphQL is all about!
Let’s Encrypt x Shopify: Securing the Web 4.5 Million Domains at a Time - Shopify
Let's Encrypt is a game changer for the industry. For a big software-as-a-service company like Shopify, it saves time because their implementation is built around an open specification.
Rate Limiting GraphQL APIs by Calculating Query Complexity - Shopify
GraphQL opens new possibilities for rate limiting. I’ll show you Shopify’s rate limiting system for the GraphQL Admin API and how it addresses some limitations of common methods commonly used in REST APIs.
10 Lessons Learned From Online Experiments - Shopify
10 hard-earned lessons from controlled experiment, considered the gold standard approach for determining the true effect of a change.
Querying Strategies for GraphQL Clients - Shopify
As more clients rely on GraphQL to query data, we witness performance and scalability issues emerging. We share some query strategies for GraphQL at scale.
Deleting the Undeletable: Managing PII at Shopify - Shopify
Building Shopify’s schematization platform for managing Personally Identifiable Information (PII) within our data warehouse.
Updating Illustrations at Scale - Shopify
@shopify/get-repo-images is an open source solution that makes it easily find illustrations and their references in code across multiple repositories.
Shipit! Presents: How We Write React Native Apps - Shopify
Join us as we talk with developers setting the React Native standards at Shopify. They’ll share more on how we write performant React Native apps.
How Shopify Built An In-Context Analytics Experience - Shopify
Whether determining which orders to fulfill first or assessing how their customer acquisition efforts are performing, our merchants rely on data to make informed decisions about their business. In 2013, we made easy-access to data a reality for Shopify merchants with the Analytics.
Other Driven Developments - Shopify
Mental models within an industry, company, or even a person, change constantly. As methodologies mature, we see the long term effects our choices have wrought and can adjust accordingly. As a team or company grows, methodologies that worked well for five people may not work as well for 40 people.
Three Ways We Share Context at Shopify Engineering - Shopify
To do your best work as a developer, you need context. A development community thrives when its members share context as a habit. That's why Shopify Engineering believes that sharing context is vital to our growth and success—as a company, as a team, and as individuals. Context is an abstract term, and we use it to refer to the why, what, and how of our development philosophies, approaches, and ch
How I Define My Boundaries to Prevent Burnout - Shopify
There’s a way to build a model for your life and the other demands on you. It doesn’t have to be 100 hours a week. It might not even be 40 hours a week. Whatever sustainable model you come up with, prioritize fiercely and communicate expectations accordingly. I’ve worked this way for three decades through different phases of my life. The hours have changed, but the basic model and principles remai
A Five-Step Guide for Conducting Exploratory Data Analysis - Shopify
Have you ever been handed a dataset and then been asked to describe it? When I was first starting out in data science, this question confused me. My first thought was “What do you mean?” followed by “Can you be more specific?” The reality is that exploratory data analysis (EDA) is a critical tool in every data scientist’s kit, and the results are invaluable for answering important business questio
Dynamic ProxySQL Query Rules - Shopify
Resiliency is important at Shopify and although we have many tools to help us during incidents, some problems need to be fixed at the database level. At Shopify, we use ProxySQL to manage connections between applications and our MySQL databases
Management of Native Code and React Native at Shopify - Shopify
We had to determine how to leverage React Native while still using iOS and Android native features, and when and how we write native code.
How Shopify Dynamically Routes Storefront Traffic - Shopify
In 2019 we set out to rewrite the Shopify storefront implementation. Our goal was to make it faster. We talked about the strategies we used to achieve that goal in a previous post about optimizing server-side rendering and implementing efficient caching. To build on that post, I’ll go into detail on how the Storefront Renderer team tweaked our load balancers to shift traffic between the legacy sto
Building Smarter Search Products: 3 Steps for Evaluating Search Algorithms - Shopify
The three-step framework Shopify's Data Science & Engineering team built for evaluating new search algorithms.
How to Build a Web App with and without Rails Libraries - Shopify
Ruby on Rails is a web framework that contains many libraries you’d need to create and deploy a successful web application. We often take for granted the ability to run rails new to create a fully functional web application with tons of built-in features.
Remove Circular Dependencies by Using Dependency Injection and the Repository Pattern in Ruby - Shopify
Shopify Core is a Ruby on Rails monolith that loads the Shopify admin along with other services such as checkout, APIs, etc. Hundreds of developers work with a continuous deployment cycle that ultimately results in a platform used by over 1.7 million merchants.
Capturing Every Change From Shopify’s Sharded Monolith - Shopify
Shopify’s data warehouse has gone through many iterations since the company's founding in 2004. Since then, the data warehouse has evolved and grown into a data lake, compromising multiple storage mechanisms, systems, and consumers.
Updates on Shopify's Bug Bounty Program 2021 - Shopify
For three years we, Shopify’s Application Security team, have set aside time to reflect on our bug bounty program and share recent insights. This past year has been quite a ride, as our program has been busier than ever! We’re excited to share what we have learned, and share some of the great things we have planned!
4 Tips for Shipping Data Products Fast - Shopify
Shipping a new data-centric product is hard, and under tight time constraints it's even harder. Here are 4 tips that are proven to help you ship new products fast.
Keeping Developers Happy with a Fast CI - Shopify
How Shopify’s Test Infrastructure team was able to reduce the p95 of Shopify’s core monolith CI from 45 minutes to 18.