Outsized Metrics From Across the Industry

HighScalability's weekly round-up compiles an eclectic set of numbers from cloud infrastructure, open source, and the broader technology landscape. The figures below offer a snapshot of the scale at which modern systems and business models operate.

  • $54bn: Expected AWS revenue for 2021, closing in on IBM. The company also reports 200 million Prime subscribers.
  • 77%: Increase in the number of components in the average application; 84% of codebases now contain at least one vulnerability.
  • 2 Million: HTTP requests per second forwarded by HAProxy on a single Arm-based AWS Graviton2 instance.
  • 50%+: Share of Chipotle's sales from digital orders in Q1, with online sales growing 133.9%. Much of this growth was attributed to Chipotlanes, the chain's drive-thru concept.
  • 25%: Proportion of Stack Overflow visitors who copy code. Analysis of two weeks of activity detected over 40 million copies, mostly from anonymous users.
  • 1 Trillion: DNS lookups by Comcast customers in 2020. Peak internet traffic rose 32% above pre-pandemic levels, and within four months of lockdowns the network experienced nearly two years' worth of traffic growth.
  • ~12,000: Developers who contributed via open source to the software powering NASA's Ingenuity helicopter.
  • $430B: Apple's declared investment in US manufacturing.
  • €30 million: Total GDPR fines imposed on EU countries in Q1 2021.
  • 2x: Likelihood that a 50-year-old founder will succeed compared to a 30-year-old. Among the fastest-growing new tech companies, the average founder was 45 at founding.
  • $125B: Record global venture funding in Q1, a 50% increase quarter-over-quarter and 94% year-over-year.
  • $6.01 billion: YouTube advertising revenue for the quarter, up 49% from $4 billion a year earlier, putting it on pace with Netflix.
  • 62%: Streaming's share of global recorded music revenue, at $13.4 billion — an all-time high.
  • 13%: Expected semiconductor capital expenditure growth in 2021; capex grew 9.2% in 2020 to $112.1 billion.

Hardware, Environment, and Scale

  • 98%: Reduction in lithium battery costs over three decades.
  • 78Mbps: Average Starlink download speed for one user in Walnut Creek, with 36ms latency.
  • 30,000: Malfunctions attributed to cosmic rays in Japan.
  • 4.5 billion: Words generated per day by GPT-3.
  • 62.5%: Share of indie iOS developers earning nothing; 30.3% make less than $10,000, while just 0.2% earn over $1 million.
  • 5,200: Metric tons of micrometeorites falling to Earth each year.
  • ~75%: Calories that remain in fecal matter.
  • 932,884: Number of Earths that would actually fit inside the Sun.
  • 1000: Hours the Phoebus cartel declared incandescent lights should last.
  • thirty-four billion billion: Cortical spikes over a human lifetime.

The Mechanical Turk in the Metadata Path

Werner Vogels' team at AWS has detailed a key change within the S3 metadata subsystem. The core problem is staleness: when many distributed caches hold an object's metadata, how does any one cache know if its view is still correct? The answer they landed on is a new component that acts as a witness to writes. Every time an object changes, this component is notified.

The design is a clever twist on the classic readers-writer problem. Instead of having caches check with a canonical source on every single read—which would eliminate the performance benefit of caching in the first place—they've introduced a read barrier. When a read request comes in, the barrier checks with this witness to see if the object in question has changed. If the cache's version is still fresh, it serves the value immediately. If the object has changed, the cache is invalidated and the data is fetched from the persistence tier. This splits the difference, giving strong consistency without forcing every read to hit the underlying storage.

Werner's description of this mechanism was one of the most technically substantive items in circulation this week, a rare glimpse at the operational internals of one of the world's largest storage systems.

The Architecture of a Milkshake Shakedown

A long-running investigation into McDonald's ice cream machines has revealed a business model that reads like a case study in adversarial engineering. The machines, manufactured by a company called Taylor, are notoriously complex and fragile. When they break—which is often—franchisees are locked out of repairing them themselves.

The reporting, covered this week, argues this is not just a right-to-repair issue but an intentional economic design. The "secret menu" of diagnostic codes shows a system where the manufacturer profits from the constant need for authorized repair technicians. As one source put it, "It's a huge money maker to have a customer that's purposefully, intentionally blind and unable to make very fundamental changes to their own equipment." The model gives tech Twitter a new metaphor for terrible design, with one contributor remarking that calling something "architected like a McDonald's ice cream machine" is the most devastating insult they've devised.

On the Utility of Vertical Scaling

A recurring theme in the discussions this week was the humble virtual machine. One commenter offered a broad defense: "vertical scaling is severely underrated. You can do a lot and possibly everything ever for your company with vertical scaling. It would apply to 99% of the companies or even more."

The point was echoed by another engineer who recalled a conversation with a peer at a competing company. The competitor had built a complex system of Docker, load balancing, and numerous AWS services to run their hundreds of daily data processing jobs. The engineer, whose own employer processed thousands of such jobs much faster, didn't have the heart to admit they were running everything manually on a two-year-old MacBook Air.

There is a related tendency toward over-engineering in early-stage product development. The observation became a meme in the community: a $5 digital ocean box can serve plenty of traffic and yet startups feel compelled to deploy their landing pages with kubectl.

The Real Economics of Cloud vs. On-Prem

A GigaOm Research report released this week put numbers on the Cassandra question. The analysis found that a serverless deployment could cost $740,087 over three years, versus $3,174,938 for a self-managed open-source Cassandra deployment. A dramatic difference, though it's a reminder that the devil is in the details of such TCO studies.

Yet experience is not universal. A software company called Corellian announced it was leaving the public cloud after years as a "cloud-first" company. Their complaints were typical: slow performance, blocked ports, and rising costs to get the features they needed. The public cloud wasn't fitting their speed or budget requirements anymore.

One astute commentator (@swardley) framed the shift in evolutionary terms, arguing that the choice to adopt cloud is over. "Choice ended around 2012, you're waking up to this," they wrote. For those looking for growth, their blunt advice was that the battle in the container space is over; serverless is where the action is, and even that battle is mostly won by the big players. Best to find secondary niches to attack, but only if time permits.

There’s also more anecdotal evidence of cloud's ongoing grip. A survey of HN readers mentioned running a one-person SaaS with 250K users on a fairly conventional stack—Flask, Nginx, Postgres, and S3—for about $4,000 a month in infra. A scaling story from a savvy CIO involved paying for AWS as a backup plan that saved a TV service launch when their own data centers failed.

Software, Its Promises, and Its Critics

Several comment threads turned philosophical about the nature of software development itself. The engineering discipline came under fire from one reader who noted that the first tenet of engineering is public safety. "Software is carpentered, some is designed, and little is actually engineered," they argued, noting that not holding software to a licensed standard is what allows the modern web to exist at all.

Parallel to that was a debate over the meaning of engineering, with thinker Michael I. Jordan weighing in: "I think that it's important to recall that for all of the wonderful things science has done for the human species, it really is engineering—civil, electrical, chemical, and other engineering fields—that has most directly and profoundly increased human happiness."

On the practical side, there were good reminders about the purpose of documentation. One engineer noted that publishing an API is like making a promise—a promise of maintenance and stability that needs dedicated resources to fulfill. It's the reason some companies are conservative about opening up interfaces in the first place.

Finally, there were personal stories on team dynamics and career growth. Some observed that wanting a promotion while doing your current job well is a category error. The real path is to prove you can do the next job even before you're in it.

Bits and Pieces: Discord, Streams, and Speed

  • Netflix's Re-encoding Habit: A product manager reportedly said they re-encode their entire catalog monthly to take advantage of newer encoder optimizations, reducing network transfer costs regardless of the storage overhead of storing multiple versions.
  • Bitmaps at GitHub: GitHub blogged about scaling monorepo maintenance. Reachability bitmaps dramatically sped up object traversals in Git, with the Linux kernel walk going from over 33 seconds to 1.57 seconds.
  • Gaming vs. Serverless: A report highlighted that serverless architectures hit ~100ms latencies for stateful backends—fine for a lot of applications, but borderline for fast-paced games. Long-tail spikes of a second make it a hard sell for competitive gameplay. Provisioned Concurrency is the stopgap, and represents an interesting hybrid approach.
  • The Internet Archive Grows: A look under the hood revealed the scale: almost 200PB of raw storage, a mix predicated on vast spinning disk arrays. They expect to serve about 80GB/s soon, while managing a 25% annual growth rate.
  • Nostalgia of Speed: A benchmark comparing Go and Rust on a simple URL counting job ran on the same Mac. Rust hit higher concurrency and lower system time, leading the author to think Go's memory management was taking a toll.
  • The End of AMP? A comparison of AMP versus non-AMP versions of a site found AMP measurably worse—more bytes transferred, heavier script execution, and slower loading, blowing up the promise of a speedier experience.
  • Unity’s Burst Compiler: A former employee called it the most impressive tech Unity has produced, saying it can "jobify" and multithread code with huge speedups, at times even beating hand-optimized C++.
  • A Question of Harmony: One person made an analogy asking if a flautist would join an orchestra if they had to play violin. The takeaway is that programmers are generalists, but sometimes picking the wrong instrument can still force you out of the band.

Configuration That Outperforms a Single Thread

Flipr, Uber's configuration service, manages over 350K active properties with roughly 150K changes weekly. This data serves more than 700 services across 50K+ hosts and generates around 3 million QPS for the backend. Flipr properties are key-value pairs that can hold multiple values per key (a multimap instead of a map) depending on runtime context. The API call follows the shape client.get("property_key", context) => value, where the context carries the data attributes needed to resolve the configuration value.

Use cases range from simple feature toggles to nuanced targeting: "Feature #456 is enabled," "The URL of the map service API is http://mapservice:8080/v2," or "Enable this feature #456 for all members of the 'admin' group who are currently making a request from New York." Because most use cases are read-only, a fan-out cache of gateway services keeps current copies local, sparing the backend from direct traffic. Replication to gateways is asynchronous, so changes are eventually consistent; in practice this model has proven scalable, reliable, and performant for fleet-wide config at Uber scale. Flipr also has a subscription-based distribution system in progress to achieve the same scale with more efficient resource usage. Peer reviews are enforced for most changes via standardized code review tools and custom UI, and rollouts happen incrementally across physical dimensions to reduce blast radius.

Petalith Architectures and Monolithic Performance

Adrian Cockcroft describes petalith architectures: take a monolith, break it into independently deployable microservices, and run them all on one giant machine. Shared memory between microservices eliminates serialization/deserialization costs over the wire, yielding the agility of microservices with the performance of a monolith. Machines now have a terabyte of RAM; GPUs carry 40GB of memory and 7,000 cores with an internal memory bandwidth of 1.2 terabytes per second. You can scan 40GB of data at ludicrous speed on a GPU, or connect a cluster of 8 GPUs for 320GB over a 600 gigabyte-per-second link — up to 4,000 such clusters in an EC2 environment. The question then becomes what problems become solvable when containers communicate at memory speed rather than network speed. Note, though, that shared memory brings its own pitfalls: locking, deadlocks, and memory corruption are real concerns compared to a distributed unshared model.

What Does a Computation Cost?

One discussion asks a fundamental question: what counts as computation? If water follows a windy path down a hill, is it computing its route? If a mirror reflects light, is something computing? Some would define programming as any step taken to bring about a desired result, which can be done with no processor or code at all. A soft robot navigating a human trachea by design alone — without code — solves a problem through materials and structure. Xenobots, about half a millimeter wide with no nerve cells or brains, swim through thin tubes and traverse mazes. The suggestion is that computation might be modeled in a computer without the physical process itself being a computation. Whether computation includes "anything matter and energy does" remains an open question for programmers used to working with dumb data manipulated by smart systems elsewhere. Cells are smarter: experiments measuring ATP levels directly in intact brains show that increasing neuronal activity increases energy production — the cell anticipates future energy needs rather than simply responding to present demand.

Scaling Fast at the Edge

Cloudflare Workers Unbound is generally available, removing the previous 50ms CPU limit in favor of up to 30 seconds of execution time. Egress pricing has been reduced to $0.045 per GB, and there are no charges for DDoS attacks. The service offers zero-millisecond cold-start time. WebSocket support is now included, and durable objects provide long-lasting Actor state at the edge. One angle often overlooked: much of Cloudflare's capability is underappreciated because it is minimally invasive and fixes problems without requiring changes to legacy infrastructure.

A separate case study shows the value of architecture choice in a different market. MatHem.se, a grocery delivery service, migrated from on-prem servers to a fully serverless, event-driven stack. In March 2020, concurrent users placing orders increased 800% day-over-day; the new architecture handled it without significant changes. New feature releases accelerated 5-10 times. Before EventBridge, MatHem.se used SNS/SQS, which led to tight coupling: consumers needed to know exactly which SNS topic to subscribe to, and relied on documentation for schema understanding. The team moved to a DynamoDB-to-EventBridge fanout pattern with Lambda in between.

The same fast-scaling requirement applies to EC2 Auto Scaling, which has long suffered startup latency. Warm Pools address this by pre-initializing instances in the background — they launch, run lengthy application initialization processes, then stop until needed. On scale-out, the pool serves pre-initialized instances rather than cold boots.

Storing 10PB: Buy or Build?

A discussion of storing 10PB for a startup surfaces the usual buy-versus-build tradeoffs. At that scale, S3 costs more than $210,000 per month — more than $12M over five years. Hidden costs of rolling your own include staff (at least one storage engineer at roughly $92,000-130,000 unloaded, another $70K+ for a Storage Administrator, multiplied by 1.25-1.4 for loaded cost), facilities (HVAC, electrical, floor loading), disaster recovery, and the cost of capital. One operator who ran the math for 1PB of image data found self-hosting 1.5-2 orders of magnitude cheaper over ten years, primarily driven by cheaper bandwidth than S3. Another 20-year infrastructure veteran notes they would never build their own hosting for a startup: "It would be like a grocery delivery company starting their own farm because seeds are cheap." The middle path: negotiate with the three largest cloud providers, since at that data volume storage and egress pricing are negotiable.

One real-world cost containment story comes from aviation insurance marketplace Aerospace, which cut a 30% Firebase bill by moving a single client's data to a Redis cluster, storing only the types of data actually needed.

Microservices: Disaster Patterns and Countermeasures

A practitioner's list of disasters seen in microservices:

  • Too small services
  • Poor development environments
  • End-to-end tests as a crutch
  • A huge, shared database
  • API gateways misused
  • Timeouts, retries, and resilience failure modes

The point isn't that microservices are bad for organizational problems; it's that failures stop being edge cases at scale and become the new normal. Countermeasures include one team owning each service, banning shared persistence outright, and avoiding any transaction spanning multiple services. Kong API Gateway is used deliberately to ensure consistent authentication, rate limiting, and logging. The core insight: don't multiply services unnecessarily, and treat distributed failures as inevitable rather than exceptional.

Airbnb's cloud efficiency journey is instructive. In nine months ending September 30, hosting costs fell $63.5 million year-over-year, a 26% reduction in cost of revenue. AWS Savings Plans cover most compute; a 3-year convertible plan allows migration to new instance types. A group of people with broad expertise meets weekly to review the entire cost footprint. Their findings: S3 storage was a top spend area, fixed by retention policies, storage tiers, and cleanup. Large services not using horizontal-pod-autoscaler (HPA) or using it badly (high minReplicas or low maxReplicas) wasted substantial resources. Spot instances and carefully managed Savings Plans keep utilization high.

Load Balancer Limits and Session Races

Connection limits apply to more than the servers themselves. AWS recommends supporting no more than 400K simultaneous connections on a single Network Load Balancer. If each instance handles 10K connections, that's a maximum of 40 fully-loaded server instances per NLB — a significant constraint on elasticity when absorbing unexpected spikes. Even under the 400K threshold, ably reports that roughly 20% of connections would occasionally drop on their NLBs.

A race condition found in GitHub's session handling is more subtle. If an exception fired at the right moment and concurrent request processing happened in the right sequence across multiple requests, one response's session cookie could be swapped with a session from an earlier request. The response body (HTML) was still based on the authenticated user, but the incorrect cookie went out. The clear identification in request logs points to a key lesson: session handling, like all stateful code, deserves fault injection testing.

A Monolith That Runs Itself

One-person tech startup architectures show how far a single operator can take a platform. The current setup: a Django monolith on Postgres, ClickHouse for analytics, Redis for caching, Celery for scheduled tasks, and a custom event queue for write buffering — all on managed Kubernetes (EKS). "It may sound complicated, but it's practically an old-school monolithic architecture running on Kubernetes."

For a real-time recommendation engine in Rust, that operator made more extreme choices: no external services (not even a database), everything in pure in-memory Rust data structures (HashMap, Vector, HashSet). There is no REST API — only ~30 predefined event types with JSON schemas, which enables full state reconstruction and time travel for debugging ML models. They process up to 100,000 events per second, two orders of magnitude faster than their previous Python stack. The costs: verbose code, poor compilation times (around a minute), and a choice that works only if the price of in-memory state is worth paying.

Wakatime, whose dashboards run on DigitalOcean, faces a different tradeoff. S3 storage is cheap; transfer and operation costs are not. Running a Redis cache of S3 reads barely dented traffic because RAM-limited caches can't hold terabytes. Using SSDB, a disk-based Redis clone, significantly reduced the outbound data transfer portion of the their AWS bill.

The lessons after 2 billion Lambda executions reinforce the obvious but not universal: don't expect events to arrive in order, use idempotency, control execution flow, make monitoring and alerting a first-class concern, and don't ask Lambda to serve massive web traffic.

Scaling The API Gateway: A Redis Rate-Limiter Deep Dive

GitHub has published a detailed account of how it scaled its API rate limiter using a sharded, replicated Redis setup. The write-up includes the actual code, offering a practical blueprint for teams dealing with similar API scalability challenges.

New Tools and Frameworks for Distributed Systems

The ecosystem for building distributed applications continues to expand with several notable releases:

  • Stateful Functions 3.0.0: The Apache Flink community has announced this release, bringing "Remote Functions" to the forefront. This cross-platform stack aims to simplify the development of stateful serverless applications, making distributed apps more scalable, consistent, and elastic.
  • libmtev: Circonus Labs has open-sourced this toolkit for building high-performance servers. Its subsystems, including the eventer, logging, clustering, and network listeners, are all driven by a robust configuration system.
  • Goblins: This is a transactional, distributed actor model environment for Racket. It functions as a quasi-functional distributed object system with a strong focus on object-capability security for safe distributed programming. Relatedly, CapTP is a protocol designed to facilitate "distributed object programming over mutually suspicious networks," built to work within an object capability model.
  • serverless-patterns: AWS has published a repository containing serverless patterns that demonstrate how to integrate services using infrastructure-as-code (IaC), allowing developers to bootstrap their own projects by reusing these proven blueprints.

Publications and Research on Scalability

A fresh batch of academic and industry papers covers a wide range of topics, from specialized hardware to data structure efficiency:

  • Warehouse-scale video acceleration: A paper describing Google's design and large-scale deployment of a new video coding unit (VCU) accelerator. It discusses the trade-offs in co-designing hardware with distributed software systems and reports a 20-33x efficiency improvement over a well-tuned, non-accelerated baseline in live data center jobs.
  • Guide to the Software Engineering Body of Knowledge: The Swebok 3.0 guide summarizes generally accepted knowledge in software engineering across 15 knowledge areas (KAs), providing basic concepts and reference lists for more in-depth study.
  • The Algorithm Commons: Research analyzing the origins of algorithms underpinning modern enterprise computing finds that the United States has been the largest contributor historically, with universities and large private labs like IBM leading the way, though U.S. leadership has diminished in recent decades.
  • Unikraft: This Linux Foundation project aims to make unikernels easy. Tests with off-the-shelf applications like nginx, SQLite, and Redis show a 1.7x-2.7x performance improvement over Linux guests. Unikraft images are roughly 1MB, use less than 10MB of RAM, and can boot in around 1ms (with a total boot time of 3ms-40ms including VMM time).
  • FoundationDB: The paper presents this open-source, transactional key-value store as one of the first to combine NoSQL scalability with ACID transactions. Its "unbundled" architecture decouples the in-memory transaction management system, distributed storage, and a built-in distributed configuration system, with the project now over ten years old.
  • Segcache: A memory-efficient in-memory key-value cache for small objects. Evaluations with production traces from Twitter, Facebook, and Reddit show it uses 22-60% less memory than state-of-the-art designs and provides close-to-linear scalability, achieving up to an 8x speedup over Memcached with 24 threads. This is significant because most cached objects are small; for example, Twitter's clusters often have a median object size under 300 bytes, yet Memcached has 56 bytes of metadata per key.
  • Read-Write Quorum Systems: A pragmatic re-examination of quorum systems introduces practical refinements to find systems that boost throughput while lowering latency and network load. The authors also developed a library called Quoracle to help engineers quantify trade-offs and find optimal quorum systems for their specific deployments.
  • SLIDE on Modern CPUs: Research on the SLIDE deep learning algorithm shows it exploits the AVX-512 vectorization capabilities for up to a 7x speedup on the same hardware, focusing on recommendation and NLP models with hundreds of millions of parameters.

Perspectives and Commentary

Several other pieces sparked discussion within the community:

  • SQLite's Misleading Name: One author argues that SQLite should be called "AwesomeSQL," as the "lite" moniker is undeserved and that for most use cases, it is the only database ever needed.
  • The Robustness Paradox: Commentator "wahern" highlighted the irony that "intended robustness hacks invariably end up creating or compounding reliability problems," pointing to systems like Kubernetes with its forwarding proxies and centralized retries. Rather than fixing issues, these additions often create a situation where "instead of N problems you have 2N or N^2 problems."
  • WASM's Real Value: On the topic of WebAssembly, commentator "flohofwoe" argued that its main point isn't raw performance gains but that it "opens up a fairly straightforward path to use different languages on the web," freeing JavaScript from being a compilation target so it can focus on being a language written by humans.
  • Parallel Programming Guide: The second revision of "Is Parallel Programming Hard, And, If So, What Can You Do About It?" by Paul E. McKenney is available, serving as a guide to help programmers work with shared-memory parallel systems "without risking your sanity."
  • AI and Analytics at Scale: HPE has released a book on AI and Analytics systems containing over a dozen real-world use cases with practical advice for building scale-efficient systems in various industries.