
Using Apache Kafka to process 1 trillion inter-service messages
We learnt a lot about Kafka on the way to 1 trillion messages, and built some interesting internal tools to ease adoption that will be explored in this blog post
248 articles on Distributed.

We learnt a lot about Kafka on the way to 1 trillion messages, and built some interesting internal tools to ease adoption that will be explored in this blog post

At Spotify, we’ve tried to uphold the tradition of running one major holiday marketing campaign at the end of each year. The campaigns typically run on various platforms such as the Spotify Homepage and the Premium page.

We’re releasing exciting functionalities that will enable organizations to confidently manage and scale with Codespaces.

Last September, Slack released Clips, allowing users to capture video, audio, and screen recordings in messages to help distributed teams connect and share their work. We’ve continued iterating on Clips since its release, adding thumbnail selection, background blur, and most recently, background image replacement. This blog post provides a deep dive into our implementation of…

A conversation with Thierry Blancpain from Grilli Type

Shopify shares some of the lessons we learned and solutions we built in order to run Airflow at scale.

Utilizing icons in user interface elements is helpful. In addition to element labeling, icons can help reinforce a user element’s intention to users. But I

In the first two posts about the Duplo initiative, we described why we decided to revamp our mobile codebases, the initial phase to clean up tech debt, and our efforts to modularize our iOS and Android codebases (post 1, post 2). In this final post, we will discuss the last theme of the Duplo initiative,…

From automating builds and releases to taking care of large-scale regression testing, here are a few ways we use GitHub Actions to build GitHub.

At Slack, the goal of the Mobile Developer Experience Team (DevXp) is to empower developers to ship code with confidence while enjoying a pleasant and productive engineering experience. We use metrics and surveys to measure productivity and developer experience, such as developer sentiment, CI stability, time to merge (TTM), and test failure rate. The DevXp…

In the first post about the Duplo initiative, we discussed the reasons for launching a project to revamp Slack’s mobile codebases, and what we accomplished in Duplo’s initial Stabilization phase. This post will explore modularization, and then there will be a third post to describe how we modernized our codebase and the overall results of…

If you’re a GHES customer with heavy read traffic on your monorepo, check out the repository cache, especially if you have CI workloads distributed around the world.

TL;DR: Using the properties of the Poisson bootstrap algorithm and quantile estimators, we have been able to reduce the computational complexity of Poisson bootstrap difference-in-quantiles confidence intervals enough to unlock bootstrap inference for almost arbitrary large samples. At Spotify, we can now easily calculate bootstrap confidence intervals for difference-in-quantiles in A/B tests with

In this article, I will talk about how Slack uses Kafka, and how a small-but-mighty team built and operationalized a self-driving Kafka cluster over the last four years to run at scale. Kafka is used at Slack as a pub-sub system, playing an essential role in the all-important Job Queue, our asynchronous job execution framework…

Infrastructure Platform teams enable organisations to scale delivery by solving common product and non-functional requirements with resilient solutions. This allows other teams to focus on building their own things and releasing value for their users. But nobody said building scalable platforms was easy... In this article Poppy and Chris explore 7 key principles which can help you build the right

Never fear, HighScalability is here! Think your software diagram is complex? This is a single cell modeled using X-ray, nuclear magnetic resonance (NMR), and cryo-electron microscopy datasets. Gael McGill My Stuff: * Love this Stuff? I need your support on Patreon to keep this stuff going. * Know anyone who needs to fix their cloud-obliviousness? My book teaches them all they need to know about th

Joining a team where there are no established design practices may feel disheartening. Does this situation resonate with you? If yes, then read on — as this article is sharing the author’s advice based on his personal experience and it will show you a way (although not easy) of facing the problem in a manner that will bring other benefits, beside order and consistency.

When do you need to overhaul a large code base to address tech debt? What is the best way to address widespread inconsistencies and outdated patterns? How can you make significant architectural improvements to a complex application while still continuing to ship features? These were questions we grappled with at the beginning of 2020, when…

When Chris first sent me this prompt, I was thinking about writing about progressive enhancement, but that subject is so wide-reaching to be one thing and all

Architecture need not be a monologue; delivered top-down from the minds and mouths of a centralised few. This article describes another way to do architecture; as a series of conversations, driven by a decentralised and empowering decision-making technique, and supported by four learning and alignment mechanisms: Decision Records, Advisory Forum, Team-sourced Principles, and a Technology Radar

A deep dive into how Shopify Data revamped the data infrastructure powering our BFCM live map using Apache Flink.

Today I have the pleasure of announcing my new app—Max reHIT Workout—on Product Hunt. Max reHIT Workout is an exercise app that guides you through interval workouts. I won’t pitch the app here. I'll just say I’m proud of how it turned out and if you want an optimal algorithm for exercising, you might like it. I know I haven’t been writing much lately. That's because there’s been very little evolut

We’re excited to announce Services, the new way to build composable, distributed applications on Cloudflare Workers. Learn how Services can replace the traditional “microservice architecture” with an alternative, zero-cost abstraction model.

A Q&A with the Figma engineering team

One of the toughest things about being someone who cares deeply about design systems is making the case for a dedicated design system. Folks in leadership

We are very excited to announce that Cloudflare for SaaS is generally available, so that every customer, big and small, can use Cloudflare for SaaS to continue scaling and building their SaaS business.

How we empower engineers to build real-time data views

Which new framework should I learn this year? Is it time to ditch my CMS? What tools should I pick up if I want to scale my site to an audience of millions?

Hey, HighScalability is here again! The circulatory system of the internet. @tylermorganwall Love this Stuff? I need your support on Patreon to keep this stuff going. Sorry for the long gap in posting, but I’ve been building a new app. I’m looking for testers for my new iOS fitness app: Max reHIT Workout. It guides you through proven reduced-exertion high-intensity interval workouts. If that inter

Cloudflare announces the Private Beta of their Web3 gateways for Ethereum and IPFS. Unlocking the Metaverse, Web3, and Decentralized Finance for every developer.

We’re making it easier to build and scale real-time communications applications around open technologies, starting with WebRTC Components.

In 2019, to meet GitHub’s growth and availability challenges, we set a plan in motion to improve our tooling and ability to partition relational databases.

We reevaluated our existing product categorization model to ensure we’re understanding what our merchants are selling, to build the best products that help power their sales.

I'd say cloud functions are one of the most transformative technologies in the last bunch of years. They are (usually) cheap, scale well, secure in their

Reinventing how the world does work inevitably creates a lot of data. Each year, Slack’s scale has increased and the volume of data ingested and stored has kept pace. To make it possible to understand relationships within our data, we’ve invested heavily in an automated data lineage framework. This facilitates producer/consumer coordination, improves risk mitigation,…

What would a totally new search engine architecture look like? Who better than Julien Lemoine, Co-founder & CTO of Algolia, to describe what the future of search will look like. This is the first article in a series. Search engines, and more generally, information retrieval systems, play a central role in almost all of today’s technical stacks. Information retrieval started in the beginning of com

There are two strategies for incrementally building websites that are growing in popularity: Incremental Static Regeneration and Distributed Persistent Rendering. What’s the difference? In this article, Cassidy Williams will tell you everything about it.

High availability and short response times are necessary, making background jobs an indispensable tool regardless of the application’s scale.

Normally, a project will have a set of pre-determined font sizes, usually as variables named in such a way that seeks some semblance of order and consistency.

To make some terminology clear here:

Hey, it's HighScalability time! Only listen if you want a quantum earworm for the rest of the day. Not your style? This is completely different. No, it’s even more different than that. Today in things that nobody stopped me from doing: The AWS Elastic Load Balancer Yodel Rag. pic.twitter.com/ocyVLf8WlUMay 28, 2021 Love this Stuff? I need your support on Patreon to help keep this stuff going. Do em

Maximize conversion, scale globally, and introduce new revenue streams with new products from Stripe.

(External) memory fragmentation is a long-standing Linux kernel programming issue. As the system runs, it assigns various tasks to memory pages. Over time, memory gets fragmented, and eventually, a busy system that is up for a long time may have only a few contiguous physical pages. Because the Linux kernel supports virtual memory management, physical memory fragmentation is often not an issue. Wi

@shopify/get-repo-images is an open source solution that makes it easily find illustrations and their references in code across multiple repositories.

Durable Objects are an awesome addition to the Workers developer ecosystem, allowing you to address and work inside a specific Worker to provide consistency in your applications. You might be wondering "Okay, so what can I build with that?"

Like Jamstack, Netlify is coining this term.

Hey, HighScalability is back! This channel is the perfect blend of programming, hardware, engineering, and crazy. After watching you’ll feel inadequate, but in an entertained sort of way. Love this Stuff? I need your support on Patreon to keep this stuff going. Do employees at your company need to know about the cloud? My book will teach them all they need to know. Explain the Cloud Like I'm 10. O

At GitHub, we serve some of the largest Git repositories on the planet. We also serve some of the fastest-growing repositories. Each day, the largest repositories we host become even…

Today, we are announcing the beta of Cloudflare Images: a simple service to store, resize, optimize, and deliver images at scale.

Fun and Interesting Events * Since we’re all feeling the fatigue of staring at screens all year, we’re switching things up at this year’s Failover Conf! This one-day conference will be featuring LIVE fireside chats and keynotes where you’ll be able to get your questions answered in real time. You’ll also hear from a variety of industry experts in our two panel discussions. And when you need a brea

When a design system first starts out, the promise of visual consistency glows bright — the ideal product would have only one set of buttons, a unified typography scale, and elements that look the same no matter which designer made the design or which developer programmed them to be real and deployed.

In this article, we walk through building out a full-stack real-time and completely serverless application that allows you to create polls! All of the app’s

Hey, HighScalability is here again! Reverse engineering an ancient analog computer is a detective story worth reading. A Model of the Cosmos in the ancient Greek Antikythera Mechanism. Do you love this Stuff? Without your encouragement on Patreon this Stuff won't stuffin’ stuff. Know someone who needs to know the cloud? I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 262 most

Anomaly Detection uses an algorithm called Histogram-Based Outlier Scoring (HBOS) to detect anomalous traffic in a scalable way. While HBOS is less precise than algorithms like kNN when it comes to local outliers, it is able to score global outliers quickly (in linear time).

This is a guest post by Paddy Byers, Co-founder and CTO at Ably, a realtime data delivery platform. You can view the original article on Ably's blog. Users need to know that they can depend on the service that is provided to them. In practice, because from time to time individual elements will inevitably fail, this means you have to be able to continue in spite of those failures. In this article,

I’ve been a manager for many years at companies of different scale. Through these experiences, I’ve done my share of learning, and made some mistakes along

Who's Hiring? * Looking to rapidly hire Top Software Developers? Get Started with Toptal. Toptal will match you with top-quality, pre-screened freelance software developers that meet your project requirements. All in under 48 hours. Get started right away with a no risk trial. * InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break

We are proud to announce a new resolver for the Distributed Web, where IPFS content indexed by the Ethereum Name Service (ENS) can be accessed.

Who's Hiring? * Looking to rapidly hire Top Software Developers? Get Started with Toptal. Toptal will match you with top-quality, pre-screened freelance software developers that meet your project requirements. All in under 48 hours. Get started right away with a no risk trial. * InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break

Cron parsers are everywhere, each supporting their own extensions on the cron string format, which made consistency between parts of Cron Triggers difficult to achieve. Today we’re showing how we easily solved this problem with Rust without needing to write the same code twice.