Numbers Roundup: Where The Scale Bar Is Set

It has been a busy stretch for notable numbers across the infrastructure and software landscape. A few of the more striking figures from the last several weeks:

  • Stack Overflow is back in the spotlight with a key data point on efficiency: just 9 on-premises servers run the entire operation. That set-up handles 200 sites, 6,000 requests per second, and 2 billion views per month while keeping latency in the millisecond range — all at under 10 percent utilization.
  • S3 passed an attention-grabbing milestone: 200 trillion objects stored. That breaks down to roughly 29,000 objects for every person on the planet, with the service averaging over 100 million requests per second and growing 250,000-fold in under 16 years.
  • Netflix reports its real-time data infrastructure handled 20 trillion events per day as of 2021.
  • Over at Riot Games, the infrastructure for millions of users spans 20+ shards, absorbs a peak load of 500,000 events per second, and generates 8 TB of data each day. Kafka handles on-prem buffering before data moves to AWS on the backend.

Pushing The Limits (And Making Things Smaller)

On the compute performance side, Go generics have yielded substantial gains. Applying them to Google's B-Tree implementation delivered a 40 percent increase in performance. PHP 8.1 shows a similar arrow pointing up, with benchmarks indicating a ~47 percent speed increase over PHP 8.0.

But faster isn't always bigger. A 4-bit microcontroller built on plastic shows an entirely different approach to the same engineering problem, deliberately cutting back instruction complexity to fit the constrained form factor. The processor executes each instruction in a single clock cycle and separates instruction and data memory instead of leaning on 16-bit or 32-bit architecture.

Google also kept an older benchmarking torch lit by extending its calculation of pi to 100 trillion digits, completed in 157 days of compute time — just a month longer than the 121 days needed in 2019 to reach 31.4 trillion digits.

The question of just how many connections one can handle found its answer, too: 1,000,000 concurrent connections is quite possible, provided everything is tuned correctly, according to blog post analysis.

Notes From the Network's Edges

Subsea infrastructure remains vulnerable: 2/3 of fiber optic cable faults stem from fishing vessels and ships dragging anchors. That physical fragility contrasts sharply with where the digital frontier is heading — the next human-scaled, globe-spanning platform might organize itself around machine learning pairs: a dataset from LAION contains five billion image-text pairs for open-source large-scale work.

However, energy use remains a nagging thread at all ends of the computing universe. The Bitcoin network consumed an estimated 100 terawatt-hours of electricity in 2021, exceeding Finland's typical annual energy budget. In comparison, human beings run on remarkably modest hardware: just 0.2 watts powers computation in the human cerebral cortex, though communication costs run 20-fold higher.

At the residential edge, Starlink's download speeds now edge past 100+ Mbps, while on the solar supply side, deployment of 1 TW is projected annually by 2030.

Infrastructure security continued to generate unfortunate statistics. A botnet operating for a full 18 months found ways to evade corporate defenses. IRShield, a new countermeasure, reports 95% effectiveness against passive WiFi eavesdropping attacks.

By The Numbers—Oddments & Etymologies

  • 3 hours a day of thinking might be all it takes for an abstract insight; see June Huh's path to the Fields Medal.
  • $3.4 million arrives as an estimate of the uncompensated labor provided annually by Reddit moderators.
  • At the financial layer, permanent errors add hidden costs: on-chain, $34 million now sits entirely locked into the AkuDreams contract forever as a result of a bug.
  • The GDPR enforcement machine flexed harder in the first half of 2022, dishing out fines nearing €100 million — a 92% spike over H1 2021.
  • A quick metric to calibrate geopolitics: at .73 on the Kardashev Scale, we still have a long way to go as a civilization.
  • Apps and services keep running on tight budgets: one podcast host leans on $5000 per month of spend against Linode servers for its operations, showing that even scale can be a relative proposition.

Scaling Delusions, Serverless Economics, and the Cost of Complexity

The recurring theme in this roundup of distributed-systems commentary is that the industry is over-invested in complexity before it has earned the right to it. Chris Munns echoed a sentiment that many practitioners have voiced: the constant preparation for massive "scale" is often a lie we tell ourselves. He points out that 99% of apps never break 1,000 requests per second, which can still support millions of monthly active users. Internally facing enterprise applications rarely exceed tens of RPS. With technologies like Lambda, Fargate, or even a simple EC2 setup, hitting that threshold is trivial, making a 24/7 server fleet for a two-RPS app a questionable expenditure.

Real-world examples support this view. Ben Schwarz noted that Calibreapp handled tens of millions of API requests per month on a $7 Heroku dyno for a long stretch. Joe Emison provided a detailed financial breakdown of Branch, a fully serverless workload, which grew ~15x year-over-year but holds a monthly AWS bill of around $10K. The largest cost line, DynamoDB at $4k/mo, is dominated by backups rather than usage. They run zero full-time DevOps, with senior developers handling infrastructure in code reviews. In another case, a company advised by @thdxr dropped its application hosting bill from roughly $4,000 a month on EKS to $70 after moving to Lambda because their 60 requests-per-minute workload did not justify paying for 24 hours of container uptime to use only 29 minutes of compute.

The economics of serverless, however, are not without their own failure states. A cautionary tale comes from a developer who accidentally deployed a self-recursive Lambda function. With a 30-second timeout and no termination condition, it ran for 24 hours, consuming over 70 million Gb-seconds and generating a billing alert for $1,484 before it could be stopped.

The Pendulum Swings on Architecture

Engineers continue to debate the merits of architectural patterns, often with polarizing stances. The grass is always greener, as @hkarthik observed: monolith veterans yearn to decompose services, while microservice-survivors long for the days of a single codebase. Jedberg, a proponent of microservices, offers pragmatic advice to startups: build a monolith with clean module boundaries and avoid splitting services until the organization is large enough for microservices to actually win. Steven Lemon's team cancelled their move to microservices entirely, instead dividing the monolith into separate projects for structure, which exposed coupling issues without the operational burden.

GraphQL continues to draw criticism from @jmhodges, who called it a "trap." He argues that exposing a public API as a generic graph database imposes a huge maintenance load. Without locking down query capabilities, you perform infinite performance tuning; with it, you have rebuilt standard REST APIs with more overhead.

The Cost of Convenience and the Power of Efficient Design

A growing segment of commentary argues that a sophisticated system is only better if it solves a real problem. Rick Houlihan stresses that scale is enabled by efficiency, and efficiency lowers cost. This principle is often overlooked, as @pati_gallardo notes, when mental models omit caching and other real-world factors, so developers rarely measure actual differences between an O(log n) and an O(n) algorithm despite theoretical complexity guarantees.

The obsession with avoiding vendor lock-in also takes heat. Boris Tane contends that teams building abstraction layers over managed cloud services to stay "vendor-agnostic" end up driving the lowest common denominator and losing the features that make their infrastructure a force multiplier. Instead, he suggests fully embracing relevant services. A similar critique applies to data formats. An HN comment by jiggawatts decries the industry trend of emitting 1KB of JSON to represent a single metric, citing write amplification that is "orders of magnitude of overhead, not data." The comment contrasts this with Google's default of gRPC and binary formats, arguing that compiled code and efficient serialization contribute to Kubernetes being orders of magnitude faster than some alternatives.

On the topic of technology choices, Matt Rickard contends that early-stage startups should not run Kubernetes yet, though growth and large companies eventually should. The tech stack should progress with your team. CharlieDigital extends the logic: choose the "dumbest" possible technology to solve the core business problem to minimize ramp, ops complexity, and failure points, then iterate.

The pendulum swings a different direction for many engineers at large companies. Amanda Walker discusses being a "Xoogler" where she had to push back on assumptions of massive scale, while a Google engineer jokes about a day filled entirely with algorithmic interview questions. The range of experiences—from a $7 dyno handling millions of requests to an infrastructure migration at Epic Games Store—reveals that context is king.

Discussion Highlights

  • Brain Power: François Chollet draws a comparison between the human brain's energy usage — around 15 watts with only 1B neurons active at once — and a data center, suggesting the industry's focus on enormous scale might be over-engineered.
  • An Alarm: Rick Houlihan warns that too many well-architected workloads rely on us-east-1 failover, and predicts future regional outages will cascade as failover procedures themselves brown out other AWS regions.
  • A Greener Pasture: A sentiment echoed from multiple angles: “This microservice could have been a SQL query” and the mocking trope of “we have over 170 microservices, because our principal engineer is very knowledgeable about distributed systems.”
  • Latency Numbers: Latency for Amazon EFS is reported at 600 microseconds, vs 600 milliseconds for IBM’s 1956 hard drive, a 1000x improvement. A multi-region setup to support latency routing saw a ~30-50ms increase when using an API Gateway proxy before AppSync.
  • Analytics Onboarding: Plausible says moving from PostgreSQL to ClickHouse was its best technical decision, allowing it to count more than a billion page views per month while maintaining fast dashboard loading.
  • Infrastructure Monitoring: A continuous profiling tool dramatically visualized the cost of service mesh when one user discovered 55% of CPU time was spent running Istio/Envoy.
  • Tech Nostalgia: Comparing microservices to the eras of mainframes, one commenter recalls the concept of a "channel controller" on IBM mainframes where any DASD or communication link formed its own system.
  • Cryptocurrency Rethink: One post highlights that 99% of crypto transactions occur on centralized exchanges, which write to their own SQL databases rather than the blockchain, and a growing sense of distrust about centralized or volatile-scale systems applying to new technology such as Web3 and GraphQL. The lack of clear forward progress on scale, combined with centralized databases struggling with flattened CPU clock speeds, feeds skepticism about the long-term TCO of RDBMS.

Better to Stay Dumb

Consider the field of reliability as a gauge. Slack’s analysis of its own February 2022 outage points to a critical insight: the platform's degradation was caused not by a single failing component, but by unforeseen interactions between the application, Vitess datastores, the cache, and service discovery systems. For a CEO of a decentralised startup, the industry's default route of microservices, Kubernetes, and as-a-service models increasingly feels premature. The most common counter to that is the person who retired a $7 heroku dyno after years of handling millions of requests per month, without k8s.

HBO's Silicon Valley might’ve caught lightning in a formula with “simple, stupid,” but CharlieDigital outlines a solid engineering governance philosophy: startups should run on “dumbest tech” so any new hire can quickly be productive, deployment is simple, and the number of ways the system can break is kept small.

Often, this knowledge is learned the hard way. As one engineer recounts discovering after a profound refactor: “The real cost of your infra is not in the dollars you spot in a pricing calculator. It's the maintenance, the pager alerts, and the operational knowledge spread.” In that sense, an efficient system is worth less for its elegance, and more for the team time it saves.

The Cloud, the Monolith, and the Economics of "Good Enough"

This week’s roundup is dominated by a recurring tension: the gap between architectural fashion and operational pragmatism. From Walmart’s hybrid cloud strategy to Stack Overflow’s steadfast on-prem monolith, the stories highlight that the best infrastructure decision is often the one that solves your specific problem, not the one that follows the latest trend.

Walmart’s Hybrid Cloud Reality Check

While "hybrid cloud" is often dismissed as marketing speak, Walmart is making a case for its practical application. The retail giant has developed the capability to switch seamlessly between cloud providers and its own servers, a move that has saved the company between 10% and 20% in annual cloud spending and reduced outages. This is achieved through a network of 10,000 edge nodes positioned at stores and distribution facilities, running custom software that allows backend operations to function across any cloud system. The stated philosophy is to combine the best of public cloud offerings with infrastructure that is "purpose-built" for their needs.

However, the devil is in the details. As @forrestbrazeal notes, building your own control plane for multi-region or multi-cloud is a daily cost and complexity burden. The ultimate accountability for resilience always rests with you, regardless of how many providers you use. The question remains whether the 10-20% savings justify the significant capital expenditure required to build such a complex system.

The Case for the On-Prem Monolith

Stack Overflow continues to be a data point against the microservices tide. They run a 14-year-old, monolithic, .NET-based application across just nine web servers in their own data center, handling 1.3 billion page views per month and 6,000 requests per second. The engineering team, now around 50 people, emphasizes a design focused on low latency and low memory allocation to avoid garbage collection stalls. They can deploy multiple times a day in four minutes and revert changes just as quickly, negating the primary drivers for microservices—team scaling and fast deploys.

Their pragmatic approach extends to caching. They removed all caching from their question-show page three or four years ago with no measurable performance impact. The average render time for that page is currently around 20 milliseconds. Instead of Redis, they leverage SQL servers with 1.5 terabytes of RAM, allowing a third of the database to be accessed quickly in-memory. This "single hop" architecture on 10-gigabit network cables is why a move to the cloud has never been financially worthwhile for them. They are not alone; Dan Luu points out that Wave, a $1.7B company, runs a simple Python monolith on Postgres, while Shopify also operates on a monolith.

Airbnb’s Service Migration Marathon

Airbnb’s journey is a counterpoint, showing that even with significant resources, the move to microservices yields mixed results. From 2008 to 2017, they ran a monolith and monorepo successfully until reaching $2.6B in revenue. The switch to microservices (2017-2020) was prompted by a decrease in software change velocity and difficulties working on features in parallel. This phase required a dedicated service migration team.

The results were not a silver bullet. By 2020, features had become a cross-cutting concern, requiring changes across multiple services and teams. The complexity had merely shifted from the codebase to the organizational and network boundaries. They have since moved to a "Micro + Macroservices" architecture, introducing unified APIs and central data aggregators. The lesson is that "right-sizing" services is perpetually difficult because natural service boundaries are rare, and it suggests that a well-structured monolith with clear internal service points might be a more practical starting point than a premature distributed architecture.

The Cloud Pricing Reality

According to the Cockroach Labs 2022 Cloud Report, there is no single cloud winner. All three major providers offer price-competitive options. Key findings include:

  • Thanks to AMD processors, GCP and Azure instances are outperforming AWS on price-performance.
  • The cost of persistent block storage often outweighs the instance cost, making mid-tier storage options (like pd-ssd, gp3, or premium-disk) the most cost-effective choice unless you need extreme IOPS or low latency.
  • Smaller instances of a given type provide a per-vCPU performance advantage in benchmarks.
  • A study of Cloud Pricing Comparison confirms that bandwidth costs are a primary factor that can "break the bank."

Is Bare Metal the New Cloud?

A growing sentiment, echoed by investors and engineers on Twitter, suggests that for some workloads, public cloud is being reconsidered. @martin_casado notes that growth startups are moving parts of their operations to Equinix bare metal with Kubernetes. The arguments are compelling:

  • Cost: Two months of AWS server rental can equal the outright purchase price of a server.
  • Latency: Local infrastructure offers significant latency wins that are hard to replicate in the cloud.
  • Control: Kubernetes has "leveled the playing field" for fault tolerance, making it easier to manage on-prem hardware.

One engineering lead reported saving 90% on egress traffic alone by moving to a multi-location bare metal setup. The challenge remains the lack of on-demand scaling and slower delivery speed for new hardware.

Meta’s Cloud Gaming Blueprint

For those questioning the backend feasibility of large-scale VR, Meta’s infrastructure for cloud gaming offers a preview. Their strategy is built on edge computing, deploying infrastructure in metropolitan areas to be closer to players. Key components include:

  • Partnering with NVIDIA to build hosting environments on Ampere architecture-based GPUs.
  • Using Twine (their cluster management system) and orchestration services to manage game servers on the edge.
  • Streaming user inputs and video/audio frames via WebRTC with Secure Real-Time Protocol (SRTP).
  • Employing both GPU-based encoding on the server side and hardware decoding on the client side to reduce latency.

The proximity of edge nodes is cited as reducing video and audio latency even more significantly than moving the entire streaming pipeline to GPUs. By contrast, commentary on the now-defunct Stadia suggests its failure was a management issue, not a technical one, arguing the core technology was impressive.

Rethinking Serverless and Its Costs

The definition of serverless continues to be a point of contention. @simonw offers a clean definition: "Serverless means per-request billing." This sidesteps the "scale to zero" debate, which hits a snag with services like Serverless Aurora v2 that cannot auto-pause and lack a Data API. Despite this philosophical debate, serverless adoption is growing.

On the cost side, Meta is tackling DRAM hunger with Transparent Memory Offloading (TMO), which saves 20-32% of memory per server by moving colder data to cheaper NVMe-connected SSDs. Similarly, Airbnb is using Dynamic Kubernetes Cluster Scaling to ensure its cloud footprint automatically scales with demand. They utilize Kubernetes and their service configuration interface, OneTouch, to ensure cloud spending goes up and down with daily traffic fluctuations.

The Database and Scaling Chronicles

Amidst the architectural debates, performance numbers and new tools continue to emerge.

  • Key-Value Store Kings: A benchmark of in-memory databases shows Skytable leading with 619,992 reads/second, followed by Dragonfly (408,322 reads/second) and KeyDB (288,931 reads/second). Redis trails at 112,100 reads/second.
  • New Database Tools: CloudFlare has released D1, a SQL database based on SQLite. Other new entrants include PranaDB (a distributed streaming database), FrostDB (an embeddable columnar database in Go), and tigerbeetle (a financial accounting database).
  • AlloyDB’s ML Magic: Google’s AlloyDB for PostgreSQL uses embedded machine learning to automatically organize data between row-based and columnar formats. The query planner then learns from your workload to choose the best execution plan, attempting to deliver both transactional performance and analytical speed.
  • MMO Scale: Amazon’s New World architecture uses DynamoDB to handle ~800,000 writes every 30 seconds for game state. To achieve seamless movement, they process player states 30 times a second (vs. a traditional 5 times per second) and spread compute across hub instances in multiple regions, with a single server set processing millions of state changes per second for 2,500 players.

Pinterest’s Caching Efficiency Playbook

Pinterest, an expert in distributed caching, has shared several key optimizations that resulted in significant performance gains:

  • Vertical Scaling Over Horizontal: For specific workloads, adding vertical hardware is more cost-efficient than arbitrarily scaling clusters horizontally.
  • NVMe Integration: Using extstore to push storage capacity to local NVMe flash drives reduces cluster cost footprints by up to several orders of magnitude for certain workloads.
  • SCHED_FIFO Wins: A single-line change to run memcached under a real-time scheduling policy (SCHED_FIFO) with high priority drove client-side P99 latency down by 10-40% by allowing it to monopolize the CPU and eliminate garbage collection stalls.
  • TCP Fast Open: Enabling TCP Fast Open (TFO) reduces the latency overhead of establishing a connection by saving an RTT in the three-way handshake.

Uber’s Dynamic Load Balancing

Uber has been running a Real-Time Dynamic Subsetting system in production for nearly two years across millions of containers. The core concept is that an on-host proxy can determine its contribution to a callee service's overall QPS and dynamically adjust how many backend tasks it load balances against. This self-tuning aperture has resulted in zero complaints from service owners since its rollout—a stark contrast to manually tuned subsets—and has led to a 15-30% reduction in p99 CPU utilization for the larger services that previously required manual tuning.

Defending Against the Cascading Failure

Understanding how systems fail is often more important than understanding how they succeed. A cascade begins with a single failure point, say server overload. As a server crashes from resource exhaustion, its traffic spreads to healthy nodes, increasing their load and likelihood of crashing. This "snowball effect" creates a vicious positive feedback loop of failures. To break this cycle, the advice is to:

  • Proactively Increase Resources to provide a buffer for system performance.
  • Avoid Automatic Health Check Failures which can kill overloaded servers and worsen the problem.
  • Implement Jitter and Backoff by dropping traffic significantly and slowly increasing load to allow servers to recover.
  • Switch to Degraded Mode by dropping certain types of traffic or disabling non-critical features.
  • Move to a Choreography Pattern using a publish-subscribe design to decouple services and prevent a single orchestrator from becoming the bottleneck.

    Tools Worth a Look

    Several open-source projects stood out this week for developers working across storage, query engines, and language tooling.

    • Litestream: A standalone streaming replication tool for SQLite.
    • SplinterDB: A key-value store designed for high performance on fast storage devices.
    • Zig (Zig at Uber): A general-purpose programming language and toolchain aimed at maintaining robust, optimal, and reusable software.
    • Inform: A programming language for creating interactive fiction using natural language syntax.
    • Trino: A fast distributed SQL query engine for big data analytics.
    • kic-reference-architectures: Contains the basics for a common way to deploy and manage modern applications.
    • ATL: A working space for sketching a tensor language.

    Reading List: Performance, Chemistry, and Systems

    The recent roundup features a mix of research papers on system reliability, energy harvesting, and a new book on high-performance computing.

    Microservice and Database Research

    Two papers address performance prediction and database architecture. One proposes an approach to accurately predict the deployment performance of large-scale microservice applications in various configurations from a single execution trace. This offers insights into an application's performance prior to any deployment on a real platform.

    Another paper, Succinct Data Structures and Delta Encoding for Modern Databases, details TerminusDB's departure from historical architectures. It implements a graph database with a strong schema to retain simplicity and generality, built on succinct immutable data structures for more sparing use of main memory.

    The Automated performance prediction of microservice applications using simulation study also appears in this collection.

    Systems and SRE Resources

    Several authoritative texts on site reliability are available via Google SRE Books, including Building Secure & Reliable Systems, The Site Reliability Workbook, and Site Reliability Engineering.

    For deeper dives into distributed systems, two items are worth noting. A paper on Monarch: Google’s Planet-Scale In-Memory Time Series Database looks at Google's system for storing time-series metrics used for alerting, graphing performance, and ad-hoc diagnosis of production problems. Separately, Decoupled Transactions: Low Tail Latency Online Transactions Atop Jittery Servers presents a thought experiment for avoiding cascading slowdowns when a subset of servers operate slowly but are not dead; the goal is low tail latency atop servers and networks that may intermittently run slow.

    Metaprogramming and Hardware

    Research on Zero-Overhead Metaprogramming shows that unrestricted metaobject protocols can be realized without runtime overhead when evaluated with self-optimizing interpreters. This optimization applies to just-in-time compilation via meta-tracing as well as partial evaluation.

    In the hardware space, a bio-photovoltaic energy harvester using photosynthetic microorganisms on an aluminium anode successfully powered an Arm Cortex M0+ for over six months in a domestic environment under ambient light. This microprocessor is widely used in IoT applications.

    For performance engineers, the upcoming book Algorithms for Modern Hardware by Sergey Slotin is aimed at everyone from performance engineers to undergraduate CS students seeking practical speedups beyond theoretical complexity improvements.

    Event-Driven Design

    Ben Stopford's Designing Event-Driven Systems explores how service-based architectures and stream processing tools like Apache Kafka can support business-critical systems.

    Chemistry Goes Computational

    Two papers push the boundaries of chemical computation. Convergence of multiple synthetic paradigms in a universally programmable chemical synthesis machine describes a chemical programming language that runs on cheap hardware, countering the long-held belief that digitized chemistry is impossible due to complexity, sensitivity, and cost. An example of the code is available.

    A related paper, A Probabilistic Chemical Programmable Computer, presents a hybrid digitally programmable chemical array. This system uses chemical oscillators in interconnected cells as a probabilistic computational substrate, distributing computation between chemical and digital domains with error correction for efficiency.