80 articles from High Scalability.
This is a guest article by Stanislav Kozlovski, an Apache Kafka Committer. If you would like to connect with Stanislav, you can do so on Twitter and LinkedIn. Originally developed in LinkedIn during 2011, Apache Kafka is one of the most popular open-source Apache projects out there. So far it has had a total of 24 notable releases and most intriguingly, its code base has grown at an average rate o
This blog post was written by Dedeepya Bonthu. This is a repost from her Medium article, approved by the author. In stadiums, sports fans love to express themselves by cheering for their favorite teams, holding up placards and team logos. Emoji’s allow fans at home to rapidly express themselves, and when millions of fans do it simultaneously, that’s a technical problem that we solved! At Hotstar,
This blog post was written by Josh Clemm, Senior Director of Engineering at Uber Eats. This is a repost from his LinkedIn article, approved by the author. On a cold evening in Paris in 2008, Travis Kalanick and Garrett Camp couldn't get a cab. That's when the idea for Uber was born. How great would it be if you could "push a button and get a ride?" Fast forward to today where Uber is the largest m
This is a guest article by Stanislav Kozlovski, an Apache Kafka Committer. If you would like to connect with Stanislav, you can do so on Twitter and LinkedIn. AWS S3 is a service every engineer is familiar with. It’s the service that popularized the notion of cold-storage to the world of cloud. In essence - a scalable multi-tenant storage service which provides interfaces to store and retrieve obj
This week I’ve been reading through the recent judgment from the Swedish FSA on the Swedbank outage. If you’re unfamiliar with this story, Swedbank had a major outage in April 2022 that was caused by an unapproved change to their IT systems. It temporarily left nearly a million customers with incorrect balances, many of whom were unable to meet payments. After investigation, the regulator found th
You can subscribe to the system design newsletter to excel in system design interviews and software architecture. The original article was published on systemdesign.one website. What Is Gossip Protocol? The typical problems in a distributed system are the following [1], [11]: * maintaining the system state (liveness of nodes) * communication between nodes The potential solutions to these problems
Presto is a free, open source SQL query engine. We’ve been using it at Meta for the past ten years, and learned a lot while doing so. Running anything at scale - tools, processes, services - takes problem solving to overcome unexpected challenges. Here are four things we learned while scaling up Presto to Meta scale, and some advice if you’re interested in running your own queries at scale. Scalin
You can subscribe to the system design newsletter to excel in system design interviews and software architecture. You can view the original article Consistent hashing explained on systemdesign.one website. How does consistent hashing work? At a high level, consistent hashing performs the following operations: 1. The output of the hash function is placed on a virtual ring structure (known as the ha
DALL-E picture generated from the answer. I was curious, given all the ChatGPT love, what it would make of some of our favorite topics. I was both impressed and unimpressed. Impressed. ChatGPT instantly generated an response that might make a good answer in a Miss Universe contest. You know, the kind of answer delivered with a radiant smile calculated to offend no one yet still sounds comprehensiv
Never fear, HighScalability is here! 1958: An engineer wiring an early IBM computer 2021: An engineer wiring an early IBM quantum computer. @enclanglement My Stuff: * I'm proud to announce a completely updated and expanded version of Explain the Cloud Like I'm 10! This version adds 2x more coverage, with special coverage of AWS, Azure, GCP, and K8s. It has 482 mostly 5 star reviews on Amazon. Here
There's a move to regulate cloud providers by vertically separating the services they offer. Like railroads of yore, who were not allowed to provide freight services on top of their base services, cloud providers would not be allowed to provide services on top of their base platform services. Vertical separation would be new to the cloud industry. Is it a good idea? Would it actually solve any pro
Never fear, HighScalability is here! Every cell a universe. Most detailed image of a human cell to date. @microscopicture Other images considered: one byte of RAM in 1946; visual guide on troubleshooting Kubernetes; Cloudflare using lava lamps to generate cryptographic keys; 5MB of data looked like in 1966 My Stuff: * Love this Stuff? I need your support on Patreon to help keep this stuff going. *
Who's Hiring? * Close is building the sales communication platform of the future and we're looking for a Site Reliability Engineer to help us accomplish that goal. If you have a passion for databases (both SQL and NoSQL), significant experience building, managing, and monitoring infrastructure, databases, and backend services at scale and want to work with a rag-tag team of hardworking, but humble
Who's Hiring? * Close is building the sales communication platform of the future and we're looking for a Site Reliability Engineer to help us accomplish that goal. If you have a passion for databases (both SQL and NoSQL), significant experience building, managing, and monitoring infrastructure, databases, and backend services at scale and want to work with a rag-tag team of hardworking, but humble
Never fear, HighScalability is here! Think your software diagram is complex? This is a single cell modeled using X-ray, nuclear magnetic resonance (NMR), and cryo-electron microscopy datasets. Gael McGill My Stuff: * Love this Stuff? I need your support on Patreon to keep this stuff going. * Know anyone who needs to fix their cloud-obliviousness? My book teaches them all they need to know about th
This is a guest post by Ankit Sirmorya. Ankit is working as a Machine Learning Lead/Sr. Machine Learning Engineer at Amazon and has led several machine-learning initiatives across the Amazon ecosystem. Ankit has been working on applying machine learning to solve ambiguous business problems and improve customer experience. For instance, he created a platform for experimenting with different hypothe
This is a guest post by Ankit Sirmorya. Ankit is working as a Machine Learning Lead/Sr. Machine Learning Engineer at Amazon and has led several machine-learning initiatives across the Amazon ecosystem. Ankit has been working on applying machine learning to solve ambiguous business problems and improve customer experience. For instance, he created a platform for experimenting with different hypothe
This is a guest post by Ankit Sirmorya. Ankit is working as a Machine Learning Lead/Sr. Machine Learning Engineer at Amazon and has led several machine-learning initiatives across the Amazon ecosystem. Ankit has been working on applying machine learning to solve ambiguous business problems and improve customer experience. For instance, he created a platform for experimenting with different hypothe
This is a guest post by Ankit Sirmorya. Ankit is working as a Machine Learning Lead/Sr. Machine Learning Engineer at Amazon and has led several machine-learning initiatives across the Amazon ecosystem. Ankit has been working on applying machine learning to solve ambiguous business problems and improve customer experience. For instance, he created a platform for experimenting with different hypothe
This is a guest post by Ankit Sirmorya. Ankit is working as a Machine Learning Lead/Sr. Machine Learning Engineer at Amazon and has led several machine-learning initiatives across the Amazon ecosystem. Ankit has been working on applying machine learning to solve ambiguous business problems and improve customer experience. For instance, he created a platform for experimenting with different hypothe
Today I have the pleasure of announcing my new app—Max reHIT Workout—on Product Hunt. Max reHIT Workout is an exercise app that guides you through interval workouts. I won’t pitch the app here. I'll just say I’m proud of how it turned out and if you want an optimal algorithm for exercising, you might like it. I know I haven’t been writing much lately. That's because there’s been very little evolut
What would a totally new search engine architecture look like? Who better than Julien Lemoine, Co-founder & CTO of Algolia, to describe what the future of search will look like. This is the second article in a series. Here's Part 1. Search engines need to support fast scaling for both Read and Write operations. Rapid scaling is essential in most use cases. For example, adding a vendor in a marketp
Hey, HighScalability is here again! The circulatory system of the internet. @tylermorganwall Love this Stuff? I need your support on Patreon to keep this stuff going. Sorry for the long gap in posting, but I’ve been building a new app. I’m looking for testers for my new iOS fitness app: Max reHIT Workout. It guides you through proven reduced-exertion high-intensity interval workouts. If that inter
Who's Hiring? * Wynter is looking for system administrators, engineers, and developers to join its research panel. Participate in research surveys, get paid ($90-$180/hr) for your feedback and comments. Super low key commitment, 10-15 mins per survey. Learn more and sign up. * DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced
What would a totally new search engine architecture look like? Who better than Julien Lemoine, Co-founder & CTO of Algolia, to describe what the future of search will look like. This is the first article in a series. Search engines, and more generally, information retrieval systems, play a central role in almost all of today’s technical stacks. Information retrieval started in the beginning of com
Hey, it's HighScalability time! Only listen if you want a quantum earworm for the rest of the day. Not your style? This is completely different. No, it’s even more different than that. Today in things that nobody stopped me from doing: The AWS Elastic Load Balancer Yodel Rag. pic.twitter.com/ocyVLf8WlUMay 28, 2021 Love this Stuff? I need your support on Patreon to help keep this stuff going. Do em
Who's Hiring? * DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining
(External) memory fragmentation is a long-standing Linux kernel programming issue. As the system runs, it assigns various tasks to memory pages. Over time, memory gets fragmented, and eventually, a busy system that is up for a long time may have only a few contiguous physical pages. Because the Linux kernel supports virtual memory management, physical memory fragmentation is often not an issue. Wi
Hey, HighScalability is back! This channel is the perfect blend of programming, hardware, engineering, and crazy. After watching you’ll feel inadequate, but in an entertained sort of way. Love this Stuff? I need your support on Patreon to keep this stuff going. Do employees at your company need to know about the cloud? My book will teach them all they need to know. Explain the Cloud Like I'm 10. O
Fun and Interesting Events * Since we’re all feeling the fatigue of staring at screens all year, we’re switching things up at this year’s Failover Conf! This one-day conference will be featuring LIVE fireside chats and keynotes where you’ll be able to get your questions answered in real time. You’ll also hear from a variety of industry experts in our two panel discussions. And when you need a brea
Who's Hiring? * DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining
Hey, HighScalability is here again! Reverse engineering an ancient analog computer is a detective story worth reading. A Model of the Cosmos in the ancient Greek Antikythera Mechanism. Do you love this Stuff? Without your encouragement on Patreon this Stuff won't stuffin’ stuff. Know someone who needs to know the cloud? I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 262 most
This is a guest post by Paddy Byers, Co-founder and CTO at Ably, a realtime data delivery platform. You can view the original article on Ably's blog. Users need to know that they can depend on the service that is provided to them. In practice, because from time to time individual elements will inevitably fail, this means you have to be able to continue in spite of those failures. In this article,
This is guest post by Sachin Sinha who is passionate about data, analytics and machine learning at scale. Author & founder of BangDB. This article is to simply report the YCSB bench test results in detail for five NoSQL databases namely Redis, MongoDB, Couchbase, Yugabyte and BangDB and compare the result side by side. I have used latest versions for each NoSQL DB and have followed the recommendat
Hey, it's HighScalability time once again! Amazon converts expenses into revenue by transforming needs into products. Take a look at a fulfillment center and you can see the need for Outpost, machine learning, IoT, etc, all dogfooded. Willy Wonka would be proud. Do you like this sort of Stuff? Without your support on Patreon this Stuff won't happen. Know someone who could benefit from becoming one
Who's Hiring? * Looking to rapidly hire Top Software Developers? Get Started with Toptal. Toptal will match you with top-quality, pre-screened freelance software developers that meet your project requirements. All in under 48 hours. Get started right away with a no risk trial. * InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break
Who's Hiring? * Looking to rapidly hire Top Software Developers? Get Started with Toptal. Toptal will match you with top-quality, pre-screened freelance software developers that meet your project requirements. All in under 48 hours. Get started right away with a no risk trial. * InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break
It's Christmas time and you've been wracking your brain trying to find the perfect gift that will teach your loved ones about the cloud in a simple and entertaining way. What to do? Fortunately for you, Santa has a new elf this year—Forrest Brazeal—who is part AWS Serverless Hero, part skilled cartoonist, and part cloud guru. Yes, it's a cartoon book about the cloud! The Read Aloud Cloud No, I did
Hey, it's HighScalability time once again! NASA Do you like this sort of Stuff? Without your support on Patreon this Stuff won't happen. Know someone who could benefit from becoming one with the cloud? I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 212 mostly 5 star reviews. Here's a load-balanced and fault-tolerant review: Number Stuff: * 1 billion: TikTok users in half the
Who's Hiring? * Looking to rapidly hire Top Software Developers? Get Started with Toptal. Toptal will match you with top-quality, pre-screened freelance software developers that meet your project requirements. All in under 48 hours. Get started right away with a no risk trial. * InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break
Hey, no power outages this week, so it's finally HighScalability time! Stunning: Tycho Crater Region with Colours by Alain Paillou Do you like this sort of Stuff? Without your support on Patreon this Stuff won't happen. Know someone who could benefit from becoming one with the cloud? I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 189 mostly 5 star reviews. Here's a 100% lact
This is guest a post by Preetam Jinka, Senior Infrastructure Engineer at ShiftLeft. Originally published here. ShiftLeft NextGen Static Analysis (NG SAST) is a software-as-a-service static analysis solution that allows developers to scan every pull request for security issues. Earlier this year we released Secrets, Security Insights, and a v4 API. Secrets and Security Insights are two new types of
What do you do when you find a snake in your datacenter? You might say this. (NSFW) Fault tolerance is the property that enables a system to continue operating properly in the event of the failure of some of its components. You might think Facebook solved all of its fault tolerance problems long ago, but when a serpent enters the Edenic datacenter realm, even Facebook must consult the Tree of Know
It's not all that different really, especially that part where you can lose all your bitcoin. Here's an excerpt from East of Eden by John Steinbeck: Say, Carlton, how do you go about telegraphing money?” “Well, you bring me a hundred and two dollars and sixty cents and I send a wire telling the Valdosta operator to pay Adam one hundred dollars. You owe me sixty cents too.” “I’ll pay—say, how do I
Who's Hiring? * InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform. * Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n).
Hey, it's HighScalability time! I can't wait for the duel. Just don't shoot into the air. Do you like this sort of Stuff? Without your support on Patreon this kind of Stuff won't happen. Know someone who could benefit from becoming one with the cloud? Of course you do. I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 167 mostly 5 star reviews. Here's a 100% lectin-free review:
This is a guest post by Limor Maayan-Wainstein, a senior technical writer with 10 years of experience writing about cybersecurity, big data, cloud computing, web development, and more. High performance computing (HPC) enables you to solve complex problems which cannot be solved by regular computing. Traditionally, HPC solutions provided mainly supercomputers. Today, HPC is typically a mix of resou
Microsoft Azure is one of the most popular cloud providers in the world, and a natural fit for database hosting on applications leveraging Microsoft across their infrastructure. MySQL is the number one open source database that’s commonly hosted through Azure instances. While Microsoft offers their own Azure Database product, there are other alternatives available that may be able to help you impr
Hey, it's HighScalability time! Serverless is really complex. Or is it? @paulbiggar sparked a thoughtful Twitter thread. Do you like this sort of Stuff? Without your support on Patreon this kind of Stuff won't happen. Usually I pitch my book Explain the Cloud Like I'm 10 here, but if you're an author with books on Amazon I'd like to ask you to give Best Sellers Rank a try. I started it to send dat
Who's Hiring? * InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform. * Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n).
Hey, it's HighScalability time! Line noise? Perl? Still uncertain? It's how you program a quantum computer. Silq. Do you like this sort of Stuff? Without your support on Patreon this kind of Stuff can't happen. You are that important to the fate of the smart and thoughtful world. Know someone who could benefit from understanding the cloud? Of course you do. I wrote Explain the Cloud Like I'm 10 ju
This is a guest post by Eunice Do, Data Engineer at TripleLift, a technology company leading the next generation of programmatic advertising. What is the name of your system and where can we find out more about it? The system is the data pipeline at TripleLift. TripleLift is an adtech company, and like most companies in this industry, we deal with high volumes of data on a daily basis. Over the la
Who's Hiring? * InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform. * Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n).
Hey, it's HighScalability time! LOL. Who knew a birthday service could lead to an existential crisis? Do you like this sort of Stuff? Without your support on Patreon this kind of Stuff can't happen. You are that important to the fate of the smart and thoughtful world. Know someone who wants to understand the cloud? I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 111 mostly 5
Zoom scaled from 20 million to 300 million users virtually over night. What's incredible is from the outside they've shown little in the way of apparent growing pains, though on the inside it's a good bet a lot of craziness is going on. Sure, Zoom has made some design decisions that made sense as a small spunky startup that don't make a lot of sense as a defacto standard, but that's to be expected
Who's Hiring? * InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform. * Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems: large-scale web crawling platform, integrating Deep Learning
If you’re hosting your databases in the cloud, choosing the right cloud service provider is a significant decision to make for your long-term hosting costs. This is especially apparent in today's world where organizations are doing whatever they can to optimize and reduce their costs. Over the last few weeks, we have been inundated with requests from SMB customers looking to improve the ROI on the
This is the second part of my interview with Alex DeBrie on his new insta-classic: The DynamoDB Book. To read the first part of the interview please mosey on over to The DynamoDB Book: An Interview With Alex DeBrie On His New Book. Go ahead. Take your time. It's worth it. Quick Fire Round. What's your quick reaction to: The cost of storing data in S3 versus storing data in DynamoDB? Use S3 for lar
You know nothing about DynamoDB. At least that’s what I realized the first time I heard Rick Houlihan give his now infamous talk at AWS re:Invent 2018 on Amazon DynamoDB Deep Dive: Advanced Design Patterns for DynamoDB. In that talk Rick revealed for the first time the inner arcana of single-table design. Minds were blown. Weaknesses were revealed. Futures were changed. As a mere novice in the way
We recently migrated hundreds of ZooKeeper instances from individual server instances to Kubernetes without downtime. Our approach used powerful Kubernetes features like endpoints to ease the process, so we’re sharing the high level outline of the approach for anyone who wants to follow in our footsteps. See the end for important networking prerequisites. Click to read more
