229 articles on Distributed.
In this article, I will talk about how Slack uses Kafka, and how a small-but-mighty team built and operationalized a self-driving Kafka cluster over the last four years to run at scale. Kafka is used at Slack as a pub-sub system, playing an essential role in the all-important Job Queue, our asynchronous job execution framework…
Infrastructure Platform teams enable organisations to scale delivery by solving common product and non-functional requirements with resilient solutions. This allows other teams to focus on building their own things and releasing value for their users. But nobody said building scalable platforms was easy... In this article Poppy and Chris explore 7 key principles which can help you build the right
Never fear, HighScalability is here! Think your software diagram is complex? This is a single cell modeled using X-ray, nuclear magnetic resonance (NMR), and cryo-electron microscopy datasets. Gael McGill My Stuff: * Love this Stuff? I need your support on Patreon to keep this stuff going. * Know anyone who needs to fix their cloud-obliviousness? My book teaches them all they need to know about th
Joining a team where there are no established design practices may feel disheartening. Does this situation resonate with you? If yes, then read on — as this article is sharing the author’s advice based on his personal experience and it will show you a way (although not easy) of facing the problem in a manner that will bring other benefits, beside order and consistency.
When do you need to overhaul a large code base to address tech debt? What is the best way to address widespread inconsistencies and outdated patterns? How can you make significant architectural improvements to a complex application while still continuing to ship features? These were questions we grappled with at the beginning of 2020, when…
When Chris first sent me this prompt, I was thinking about writing about progressive enhancement, but that subject is so wide-reaching to be one thing and all
Architecture need not be a monologue; delivered top-down from the minds and mouths of a centralised few. This article describes another way to do architecture; as a series of conversations, driven by a decentralised and empowering decision-making technique, and supported by four learning and alignment mechanisms: Decision Records, Advisory Forum, Team-sourced Principles, and a Technology Radar
A deep dive into how Shopify Data revamped the data infrastructure powering our BFCM live map using Apache Flink.
Today I have the pleasure of announcing my new app—Max reHIT Workout—on Product Hunt. Max reHIT Workout is an exercise app that guides you through interval workouts. I won’t pitch the app here. I'll just say I’m proud of how it turned out and if you want an optimal algorithm for exercising, you might like it. I know I haven’t been writing much lately. That's because there’s been very little evolut
We’re excited to announce Services, the new way to build composable, distributed applications on Cloudflare Workers. Learn how Services can replace the traditional “microservice architecture” with an alternative, zero-cost abstraction model.
A Q&A with the Figma engineering team
We are very excited to announce that Cloudflare for SaaS is generally available, so that every customer, big and small, can use Cloudflare for SaaS to continue scaling and building their SaaS business.
How we empower engineers to build real-time data views
Hey, HighScalability is here again! The circulatory system of the internet. @tylermorganwall Love this Stuff? I need your support on Patreon to keep this stuff going. Sorry for the long gap in posting, but I’ve been building a new app. I’m looking for testers for my new iOS fitness app: Max reHIT Workout. It guides you through proven reduced-exertion high-intensity interval workouts. If that inter
Cloudflare announces the Private Beta of their Web3 gateways for Ethereum and IPFS. Unlocking the Metaverse, Web3, and Decentralized Finance for every developer.
We’re making it easier to build and scale real-time communications applications around open technologies, starting with WebRTC Components.
In 2019, to meet GitHub’s growth and availability challenges, we set a plan in motion to improve our tooling and ability to partition relational databases.
We reevaluated our existing product categorization model to ensure we’re understanding what our merchants are selling, to build the best products that help power their sales.
Reinventing how the world does work inevitably creates a lot of data. Each year, Slack’s scale has increased and the volume of data ingested and stored has kept pace. To make it possible to understand relationships within our data, we’ve invested heavily in an automated data lineage framework. This facilitates producer/consumer coordination, improves risk mitigation,…
What would a totally new search engine architecture look like? Who better than Julien Lemoine, Co-founder & CTO of Algolia, to describe what the future of search will look like. This is the first article in a series. Search engines, and more generally, information retrieval systems, play a central role in almost all of today’s technical stacks. Information retrieval started in the beginning of com
There are two strategies for incrementally building websites that are growing in popularity: Incremental Static Regeneration and Distributed Persistent Rendering. What’s the difference? In this article, Cassidy Williams will tell you everything about it.
High availability and short response times are necessary, making background jobs an indispensable tool regardless of the application’s scale.
Normally, a project will have a set of pre-determined font sizes, usually as variables named in such a way that seeks some semblance of order and consistency.
To make some terminology clear here:
Hey, it's HighScalability time! Only listen if you want a quantum earworm for the rest of the day. Not your style? This is completely different. No, it’s even more different than that. Today in things that nobody stopped me from doing: The AWS Elastic Load Balancer Yodel Rag. pic.twitter.com/ocyVLf8WlUMay 28, 2021 Love this Stuff? I need your support on Patreon to help keep this stuff going. Do em
Maximize conversion, scale globally, and introduce new revenue streams with new products from Stripe.
(External) memory fragmentation is a long-standing Linux kernel programming issue. As the system runs, it assigns various tasks to memory pages. Over time, memory gets fragmented, and eventually, a busy system that is up for a long time may have only a few contiguous physical pages. Because the Linux kernel supports virtual memory management, physical memory fragmentation is often not an issue. Wi
@shopify/get-repo-images is an open source solution that makes it easily find illustrations and their references in code across multiple repositories.
Durable Objects are an awesome addition to the Workers developer ecosystem, allowing you to address and work inside a specific Worker to provide consistency in your applications. You might be wondering "Okay, so what can I build with that?"
Hey, HighScalability is back! This channel is the perfect blend of programming, hardware, engineering, and crazy. After watching you’ll feel inadequate, but in an entertained sort of way. Love this Stuff? I need your support on Patreon to keep this stuff going. Do employees at your company need to know about the cloud? My book will teach them all they need to know. Explain the Cloud Like I'm 10. O
At GitHub, we serve some of the largest Git repositories on the planet. We also serve some of the fastest-growing repositories. Each day, the largest repositories we host become even…
Today, we are announcing the beta of Cloudflare Images: a simple service to store, resize, optimize, and deliver images at scale.
Fun and Interesting Events * Since we’re all feeling the fatigue of staring at screens all year, we’re switching things up at this year’s Failover Conf! This one-day conference will be featuring LIVE fireside chats and keynotes where you’ll be able to get your questions answered in real time. You’ll also hear from a variety of industry experts in our two panel discussions. And when you need a brea
When a design system first starts out, the promise of visual consistency glows bright — the ideal product would have only one set of buttons, a unified typography scale, and elements that look the same no matter which designer made the design or which developer programmed them to be real and deployed.
Hey, HighScalability is here again! Reverse engineering an ancient analog computer is a detective story worth reading. A Model of the Cosmos in the ancient Greek Antikythera Mechanism. Do you love this Stuff? Without your encouragement on Patreon this Stuff won't stuffin’ stuff. Know someone who needs to know the cloud? I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 262 most
Anomaly Detection uses an algorithm called Histogram-Based Outlier Scoring (HBOS) to detect anomalous traffic in a scalable way. While HBOS is less precise than algorithms like kNN when it comes to local outliers, it is able to score global outliers quickly (in linear time).
This is a guest post by Paddy Byers, Co-founder and CTO at Ably, a realtime data delivery platform. You can view the original article on Ably's blog. Users need to know that they can depend on the service that is provided to them. In practice, because from time to time individual elements will inevitably fail, this means you have to be able to continue in spite of those failures. In this article,
I’ve been a manager for many years at companies of different scale. Through these experiences, I’ve done my share of learning, and made some mistakes along
Who's Hiring? * Looking to rapidly hire Top Software Developers? Get Started with Toptal. Toptal will match you with top-quality, pre-screened freelance software developers that meet your project requirements. All in under 48 hours. Get started right away with a no risk trial. * InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break
We are proud to announce a new resolver for the Distributed Web, where IPFS content indexed by the Ethereum Name Service (ENS) can be accessed.
Who's Hiring? * Looking to rapidly hire Top Software Developers? Get Started with Toptal. Toptal will match you with top-quality, pre-screened freelance software developers that meet your project requirements. All in under 48 hours. Get started right away with a no risk trial. * InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break
Cron parsers are everywhere, each supporting their own extensions on the cron string format, which made consistency between parts of Cron Triggers difficult to achieve. Today we’re showing how we easily solved this problem with Rust without needing to write the same code twice.
It's Christmas time and you've been wracking your brain trying to find the perfect gift that will teach your loved ones about the cloud in a simple and entertaining way. What to do? Fortunately for you, Santa has a new elf this year—Forrest Brazeal—who is part AWS Serverless Hero, part skilled cartoonist, and part cloud guru. Yes, it's a cartoon book about the cloud! The Read Aloud Cloud No, I did
At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3, and each day we ingest and create additional Petabytes. At this scale, we can gain a significant amount…
Hey, it's HighScalability time once again! NASA Do you like this sort of Stuff? Without your support on Patreon this Stuff won't happen. Know someone who could benefit from becoming one with the cloud? I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 212 mostly 5 star reviews. Here's a load-balanced and fault-tolerant review: Number Stuff: * 1 billion: TikTok users in half the
Last year we deployed a CSAM image scanning tool. This is so cool! Image processing is always hard, and deploying a real image identification system at a Cloudflare scale is no small achievement! But we hit a problem - the matching algorithm was too slow for our needs.
Learn more about how we are bringing encapsulation to our views as we scale to over 4,500 templates in our Ruby on Rails monolith.
Who's Hiring? * Looking to rapidly hire Top Software Developers? Get Started with Toptal. Toptal will match you with top-quality, pre-screened freelance software developers that meet your project requirements. All in under 48 hours. Get started right away with a no risk trial. * InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break
In this post, we’ll outline the approach we took to reliably scale our data platform in preparation for Black Friday and Cyber Monday.
Our aspiration to augment and improve every aspect of business and life with data, demands a paradigm shift in how we manage data at scale. While the technology advances of the past decade have addressed the scale of volume of data and data processing compute, they have failed to address scale in other dimensions: changes in the data landscape, proliferation of sources of data, diversity of data u
We cover our approaches to capacity planning, and how we rolled it out across the org and to dozens of teams. We’ll also share how we validated our capacity plans with scalability tests to make sure they work.
Presenting how Shopify got from no Sorbet in our monolith to almost full coverage in the span of a few months. I’ll explain the challenges we faced, the tools we built to solve them.
Hey, no power outages this week, so it's finally HighScalability time! Stunning: Tycho Crater Region with Colours by Alain Paillou Do you like this sort of Stuff? Without your support on Patreon this Stuff won't happen. Know someone who could benefit from becoming one with the cloud? I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 189 mostly 5 star reviews. Here's a 100% lact
What do you do when you find a snake in your datacenter? You might say this. (NSFW) Fault tolerance is the property that enables a system to continue operating properly in the event of the failure of some of its components. You might think Facebook solved all of its fault tolerance problems long ago, but when a serpent enters the Edenic datacenter realm, even Facebook must consult the Tree of Know
It's not all that different really, especially that part where you can lose all your bitcoin. Here's an excerpt from East of Eden by John Steinbeck: Say, Carlton, how do you go about telegraphing money?” “Well, you bring me a hundred and two dollars and sixty cents and I send a wire telling the Valdosta operator to pay Adam one hundred dollars. You owe me sixty cents too.” “I’ll pay—say, how do I
Who's Hiring? * InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform. * Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n).
Hey, it's HighScalability time! I can't wait for the duel. Just don't shoot into the air. Do you like this sort of Stuff? Without your support on Patreon this kind of Stuff won't happen. Know someone who could benefit from becoming one with the cloud? Of course you do. I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 167 mostly 5 star reviews. Here's a 100% lectin-free review:
Like many of those in tech, we invest heavily in our cloud and data infrastructure. While seemingly routine, the ability to manage and scale our infrastructure to support our 299+ million listeners worldwide, 24/7, without missing a beat (or syllable) is crucial for the business and our brand.
Stripe’s approach to creating a 1:40 million-scale, interactive 3D model of the earth.
Are you leading teams in a remote world? We are wrapping up our remote working series with some final tips on leading remote and distributed teams. We hope our teams have helped while we collectively work remotely from around the world.
