The Numbers Behind the Week in Scale

Forrest Brazeal has a warning for engineers: don't burn your limited innovation credits on bespoke, complex systems that deliver no direct value to the business. Instead, he advises positioning yourself at the top of what he calls the Wisdom/Cleverness Curve — choosing proven, straightforward technology over clever but costly custom architecture.

Infrastructure Milestones

  • 380: undersea cables spanning more than 745,000 miles — over three times the distance to the moon. Google owns or co-owns 15 of them. One of the fastest currently operating, the Marea cable (partially owned by Microsoft, Amazon, and Facebook), transmits at 160 terabits per second.
  • 60: additional Starlink satellites launched by SpaceX.
  • A Department of Defense bet on containers, Kubernetes, and Istio is paying off — the Air Force deployed 3 concurrent Kubernetes clusters on F-16s in 45 days. The platform is flexible enough for software teams across the military while preventing vendor lock-in.

The Cost of the Connected World

  • $6 billion: estimated cost of fake clicks against $300 billion spent on online ads in 2019, with potentially tens of billions in fraud yet unaccounted for.
  • -65%: Fortnite's year-over-year decline in net revenue on iOS. It's likely generating less than half its total mobile revenue on Android.
  • 20%: drop in tech M&A spending in 2019.

Voice Assistants and Software Growth

  • 200 million: Alexa-powered devices — double the count from a year ago.
  • 500 million: Google Assistant users.
  • 100 million: active Apple News users.

Computing Oddities and Fundamentals

  • 32: strands of DNA used by researchers to calculate square roots of perfect squares up to 900. They control hybridization to shift the fluorescent signal proportionally, so the result can be deduced from the color.
  • 1%: of CPU and 4% of RAM globally at Google is consumed by hash tables.
  • $1 billion: spent through eBay's Buy APIs.

Worth considering: our "social metabolic rate" runs at 11,000 watts — a dozen elephants, three-quarters of a blue whale. Keeping us alive requires just 90 watts, the equivalent of a lightbulb or a 2,000-calorie daily diet. On a temporal scale, $1 invested after the Civil War would be worth $1 million today even adjusting for inflation — a reminder that compounding gains apply to infrastructure investment too.

Managed Kubernetes Is a New Kind of Lock-in

Forrest Brazeal makes a pointed argument about Kubernetes in the cloud era: the platform is so operationally demanding that running it yourself is beyond most teams. That dynamic pushes users toward managed offerings like GKE and EKS—which, he argues, turns an open-source project into a back-door form of lock-in. Unlike traditional cloud lock-in, which at least buys deep integration between native services, this variant offers neither simplicity nor portability.

Automation's Human Dependency

Adrian Colyer highlights what he calls the central irony of automation: the more sophisticated systems become, the more they depend on highly skilled operators to manage them. He ties this to Kernighan's Law—if you write code as cleverly as you can, you may not be smart enough to debug it. The same logic applies to automated systems pushed to the technological limit: how do you manage something whose complexity exceeds your own?

Performance Lessons From the Field

A few practitioners weighed in with concrete scaling experiences. John Carmack recalls his formative impression of Python from the Quake Live backend, where the team hit serious performance problems at a few million users—his bias is that many "scalable" systems would fare fine with a simple, single C++ server.

Another Hacker News commenter, kevstev, counters that performance ceilings rarely matter until you reach FANG scale. He describes running a billion-dollar ecommerce business on just eight small VMs at the API layer. When a memory leak in Node caused instances to crash hourly, his ops lead's advice was pragmatic: throw hardware at the problem to maintain SLOs—monthly server costs were less than a Jersey City studio apartment rental from 15 years ago.

On metrics, @copyconstruct dismisses requests-per-second as a garbage measurement by itself. Context matters: what kind of request, what's the cache hit ratio, what does traffic distribution look like, and what are expected response times.

GraphQL performance drew a complaint from @benawad: a resolver took 19ms to fetch 20 recipes from Postgres, but GraphQL's resolving and validation of fields added 426ms. Another Reddit user reported that rewriting an AppSync GraphQL backend from Elasticsearch to Java on EKS cut response times by over 50%.

Cost and Storage Economics

Chris Swan summarizes the data gravity argument with a simple economic lens: cloud storage is cheap relative to egress costs, so the rational move is to put all data and workload in the same provider. The corollary, he notes, is that the data gravity is real—and it pulls toward the single-cloud decision.

Backblaze's Ariel Ellis observes that while SSDs were once more than ten times the cost of conventional drives and are now about three times, that price gap still makes flash unworkable for their business. A Hacker News commenter, rahvin, expects NAND prices to correct after the 2019 oversupply crash, with prices potentially doubling or tripling by the end of 2020. Spinning rust, he argues, remains the only viable answer to the exabyte-level storage needs of cloud providers for the foreseeable future.

One commenter describes a single-character fix that saved half a gigabyte of heap: most empty sets created throughout the application's life were allocated with the default initial capacity. Setting the initial capacity to zero eliminated the wasted memory—over a third of the total heap.

2020 Guidance Versus the 2010s Playbook

@mulligan contrasts startup advice across decades. The 2010s dogma—120-hour weeks, ignore profitability, grow at all costs, Bay Area or bust—gives way to a 2020s ethos: work how much feels right, focus on revenue, build a sustainable company, hire remote, and take care of yourself.

On a similar theme, softwaredoug delineates where "move fast and break things" works: when breakage risks are minimal, experiments are measurable and isolated, and upside is high—UI tweaks, search improvements, social media. But moving slow and keeping things working is the right approach where failure has huge downside, such as medicine, compliance, airplanes, and utilities.

There is also this cynical take on project incentives from flomo: with 50-80% failure rates, resume-oriented development—choosing a "cool" framework, even for a project that will likely fail—may beat a successful project on an uncool stack.

State Is Missing From Microservices

Gwen Shapira laments what gets lost in the move to microservices: state. Having state in a service is powerful, she says, because rules are often dynamic—you can't hard-code them and must look them up. Sometimes events contain only an ID, not all data needed. Sometimes you must join events, or just aggregate events per second, orders per hour, dollars per day. All of this requires state.

Sumit Khanna argues that good microservices do not start as microservices. They evolve from well-written monoliths that provide the template for splitting out smaller components. Starting from a microservice model, he says, invites a world of pain—good services emerge from an evolutionary process that is often muddled with technical debt, policy, and politics. "You don't build a city out of molecules," he writes. "You have several layers of abstraction in place so you can build with bricks, structures and buildings."

The Cloud Market Numbers

Tim Anderson's year-in-review notes that the big three cloud providers kept growing dramatically through 2019. In the quarter ending September 30, AWS reported 34% revenue growth, Microsoft Azure 59%, and Google's "other revenues"—which includes cloud—grew 39% to $6.4 billion. AWS holds the largest IaaS share, with estimates ranging (Gartner: 47.8%; Canalys: 32.6%) well ahead of Azure (15.5% or 16.9%) and Google (4.0% or 6.9%). Most analysts agree AWS's share is slipping slightly, even as it grows quickly.

Mastercard’s In-Memory Architecture for Billion-Transaction Days

Mastercard’s Decision Management Platform (DMP) is a plug-in-based transaction processing engine that supports 20 Mastercard products. Every card swipe passes through it before approval. The platform sustains 60,000 transactions per second with average response times under 100ms (targeting 50ms), using parallel processing across hundreds of commodity servers.

DMP enriches transactions with historical and real-time aggregates to run multiple risk models and hundreds of decision rules. It calculates hundreds of variables in real time and consumes even more offline-generated data to evaluate each transaction’s risk. The scaling challenge is data: DMP consumes 30+ billion aggregates in real time, with many terabytes shared across instances. Requirements include sub-millisecond reads at several million reads per second, plus atomic updates to large entries in under 3 milliseconds across hundreds of thousands of entries per second.

Because each transaction involves hundreds of reads and writes, data must live in memory. Mastercard chose Geode/GemFire, an in-memory data grid, for real-time scoring. But the distributed data system itself needed solving for: data access scalability via co-location, even data distribution, concurrent operations, and latency consistency.

Related data is stored together—same-account data on the same node—so one request returns everything. This NoSQL-like strategy supports 8 million reads per second and lets the cluster grow without multiplying network calls. Keys must distribute evenly for the cluster to scale, and bucket counts need careful selection.

Java GC pauses were a killer. GemFire supports larger heap sizes by minimizing short-lived object impact and using byte-array storage, keeping data in on-the-wire format. For large data sets, Mastercard chose large heaps with Azul Zing’s Pauseless JVM, which requires no tuning. After scaling horizontally (more nodes) and vertically (more RAM) with a pauseless JVM, they ran 40 terabyte clusters with 600 GB heaps without sacrificing latency.

Large entries caused client-side problems: long transfer and deserialization times, and slow business logic execution. Pulling large data volumes to the client was far too slow. The solution: keep logic where the data lives, using data-aware distributed functions (essentially stored procedures). This parallelizes execution, reduces client load, and cuts network utilization. For updating large chunks within a distributed function, holding locks too long was an issue, so delta propagation applies only changed fields—like a SQL update of specific columns.

Data-aware function execution plus delta propagation delivered: 95% reduction in network traffic, 50% latency reduction for function execution, 40% CPU reduction on server nodes, and 50% less network traffic when updating large values.

Hot partitions—entries with high update rates—required a different approach. Home Shopping network, for example, submits nightly batch updates, and delta propagation wasn’t enough. Mastercard relaxes strong consistency for these entries: replication, partitioning redundancy, and partitioning are suspended, the entry sits in RAM on one node, and replication happens once per second for safety—an acceptable one-second window of risk. This raised sustained updates from 1,000 to 100,000 per second on a single entry.

LinkedIn’s Experimentation Engine: 20x Faster

LinkedIn moved its experimentation engine from Clojure to Java, but that alone wasn’t the win. The engine handles up to 23 trillion experiment evaluations per day across about 500 production services. A proof-of-concept Java parser and evaluator achieved 2-3 times better performance than the previous version with little optimization. The approach: interpret DSLs by parsing into evaluation trees, then executing.

Key optimizations included replacing Java reflection with proper type resolution—15ns versus 45ns per call, about 3x faster. Switching from naive data structures to three plain arrays yielded a surprising 2.5x improvement: CPUs prefer sequential memory access and smaller structures, and plain arrays eliminate virtual call overhead of Java’s ArrayList. Auto-generated code also reduced complexity, so developers implement only processing logic for specific argument types.

Plaid’s Node Parallelism: 30x Fewer Containers

Plaid ran 4,000 Node containers for its bank integration service and cut that to roughly 30x fewer by parallelizing workers—saving about $300k annually in EC2 spend and simplifying the architecture significantly. The migration wasn’t just flipping a switch. Load balancing logic prioritizes user-present requests over transaction updates, so API spikes of 1,000% or more are absorbed at the expense of transaction freshness.

The rollout was risky: parallelism affects CPU, memory, and task latency unpredictably, and Node’s V8 event loop could become a bottleneck. Plaid put extensive observability in place first, then used LaunchDarkly feature flags to dynamically set maximum parallelism on workers in a new ECS cluster.

A chain of bottlenecks emerged, each solved in turn:

  • Memory allocation: Raising Node’s max heap from the 1.7GB default to 6GB (--max-old-space-size=6144).
  • Task throughput: Heap kept growing until they found fire-and-forget “floating promises” in the code—removing them stabilized heap usage.
  • S3 uploads: AWS’s Node documentation revealed the S3 client caps maxSockets at 50 instead of Infinity.
  • JSON serialization: The bfj package was about 5x slower than JSONStream; replacing it gave significant performance gains.
  • Garbage collection: Scavenging ran too often. Raising the semi-space limit (--max-semi-space-size=1024) dropped GC scavenge time from 30% to 2%.

After these fixes, parallel workers achieved latencies nearly matching single workers at around 20 concurrent tasks. Plaid acknowledges CPU bottlenecks were reduced but not fully covered, and that the old single-request-per-container approach wasn’t inherently wrong for their economics at the time.

DigitalOcean: From 15,000 Database Connections to Under 100

DigitalOcean’s Rails application (Cloud), aided by two Perl services (Scheduler and DOBE), started in 2011. None of the three talked directly; they communicated through a MySQL database that served both as data store and message broker—a single table acted as a queue. This worked for four years as user traffic grew over 10,000% between 2012 and 2016. Each new hypervisor added another persistent DB connection, reaching over 15,000 by early 2016.

The fix came in stages. Event Router—a regional proxy—polled the database on behalf of each DOBE instance per data center, cutting connections from thousands to under 100. Scheduler V2 stopped querying the database for server metrics; it aggregated them from hypervisors into its own database, using concurrency and replication for performance. But the centralized MySQL queue remained, handling up to 400,000 records daily with 20 updates per second, until early 2017.

Harpoon replaced the database queue with RabbitMQ. Workers pulled events from one side, and Harpoon pushed updates directly to Scheduler V2 and Event Router—no more polling.

Related reading: the full DigitalOcean post, plus strategies for message queues, Apache Pulsar, and content-based filtering.

Segment’s 100µs Reads with 100% Availability

Segment built ctlstore, a distributed multi-tenant data store for its control plane. The read path centers on a SQLite database (the LDB—Local Database) containing a full copy of all ctlstore data. The LDB lives on every EC2 container instance in the fleet, shared with running containers via a mount. SQLite in WAL mode means readers never block on writers, and the kernel page cache keeps hot data in memory. Reads are low latency, scale with fleet size, and are always available.

A daemon called the Reflector runs per instance, continuously applying a ledger of sequential SQL mutation statements (REPLACE, CREATE TABLE) from a central MySQL database called ctldb. The LDB tracks its ledger position via a special table holding the last applied sequence number, updated transactionally. This enables crash recovery and means each instance’s data is usually 1-2 seconds stale—a consistency-for-availability trade-off that suits Segment’s use case.

@KoboldUnderlord: I built a crappy version of this for handling the anonymization of all US operational data and it was able to backfill 53m rows to S3 in about 15 minutes using a similar strategy. Local light dbs are wildly powerful esp if you can take advantage of giving it in memory for stuff.

Cloud Cost and Pricing Realities

RDS pricing has more than doubled over time, per this analysis. SaaS isn’t following IaaS’s price reduction curve. Commentary from Reddit frames it well: cloud services sell aversion to switching costs as much as technology. Moving 10MB from S3 is trivial; moving 10PB is not. Others defend the premium: managed databases eliminate constant maintenance headaches, and the ease of securing, scaling, and restoring is worth the cost. Related reading: The Amazon Premium.

Google’s new E2 machine type brings overselling to the cloud properly. Customers buy cores and RAM as usual; Google promises they’ll be available almost always, but on rare occasions applications might pause while resources are shuffled. Providers are better positioned than customers to bin-pack workloads since they see all of them—a genuine efficiency improvement for global computing, and likely a trend that continues.

Jonathan Ellis (DataStax) on when to use on-prem, hybrid, or multi-cloud: three factors matter. Agility favors cloud—spinning infrastructure up and down quickly lets teams focus on software and data value. Cost is the flip side: the convenience of cloud, especially higher-level services, carries a premium over time. Control favors on-premises when you need full command over hardware, network, or security. Hybrid strategies work well when different parts of the application optimize for different factors; DataStax customers in financial services commonly take this path, pairing cloud flexibility with on-prem data center control.

DNS, Web Servers, and Language Performance

DNS response times can differ by as much as 200ms depending on the domain extension, per BunnyCDN’s gTLD benchmark. The .info and .org domains showed especially poor performance at the 85th percentile, .net and .com were slightly slower than expected in Europe and North America but stable globally, while .co, .biz, and .in ended up ahead of the pack.

A web server benchmark comparing Erlang, Go, Java, and NodeJS found Go’s FastHTTP peaking near 210k responses per second, with Java’s Netty second at almost 170k. Go’s built-in server peaked above 120k, NodeJS cluster at 90k, Erlang Cowboy 1.x at 80k, with Mochiweb, Cowboy 2.x, and Java Rapidoid in the 50-60k range. Non-clustered NodeJS scored 25k. In all tests except non-clustered NodeJS, CPU saturation was the limiting factor—all servers scaled to available CPUs with varying efficiency. Clustered NodeJS and Rapidoid crashed from RAM exhaustion under overload.

HTTP/2 testing for REST APIs suggests: if raw speed is the priority, compound documents still win; but simpler, smaller-scoped APIs with more endpoints are definitely viable when elegance matters more. Caching makes only a marginal difference, and optimizations benefit the server more than the client.

PHP 7.4 took gold in all 17 applicable benchmarks (5 not applicable) across 22 platforms and configurations, including WordPress, Drupal, Joomla!, Laravel, Symfony, and others—per Kinsta’s Definitive PHP benchmarks.

For approximate set membership, Xor filters take longer to build but use less memory and run about 25% faster than Bloom or Cuckoo filters once built—useful for avoiding slow disk or network operations.

Distributed Systems Fundamentals

Marc Brooker’s answer to why distributed systems are needed: they offer better availability, better durability, better scalability, and better efficiency.

Gergely Orosz’s distributed systems learnings from 2019: building a new distributed system is easier than migrating an old one to it; improving reliability means measuring, reporting, and repeating—start simple; idempotency changes should be treated as breaking changes even if technically they aren’t; postmortems are more impactful when identifying systemic issues affecting multiple teams; running your own data centers brings unique and unexpected challenges; deploy any day—including Fridays—but think carefully about when code chills are sensible trade-offs; financial and end-user impact of outages matters as much as systems impact; and a simple ownership heuristic: whoever owns the on-call owns the service.

Testing HTTP/1.1 vs HTTP/2 with server push: if speed is paramount, keep compound documents; if a cleaner API matters more, many smaller-scoped endpoints work fine. Caching barely matters. Optimizations help the server more than the client.

Determining service ownership has a simple answer: the team on-call for it.

The Cloud’s Trajectory

Geoffrey B. West’s talk on universal laws of growth draws parallels between cities and the cloud. The cloud is the tech form of urbanization: an attractor where ideas and wealth concentrate, built of complex, evolving, interactive systems. While individual companies die, the cloud—driven by efficient metabolisms and network structures—persists. It optimizes energy use to maximize resource allocation to users, with a superlinear pace of life: a positive feedback loop where building blocks spawn ever more systems.

West also suggests another innovation equivalent to IT within the next 20-25 years. Stratechery’s “The End of the Beginning” argues there may be no significant paradigm shift on the horizon for incumbents. The hyperscalers are best positioned to handle IoT data torrents, and new I/O devices—AR, wearables, voice—are natural phone extensions. Amazon, Microsoft, Apple, and Google may be the GM, Ford, and Chrysler of the 21st century. But that doesn’t diminish technology’s impact; it only means the impact is beginning.

On Microsoft “winning” the cloud war, commentary pushes back hard: Microsoft counts Office 365 and Azure AD as “cloud,” unlike comparable AWS offerings. Enterprise decisions are often driven by licensing bundles—Walmart refusing AWS due to Amazon competition, banks moving to Azure alongside Office 365, and public sector organizations with decades of Microsoft relationships and certified staff choosing Azure by default. Multi-cloud reality: one company runs 98% of bespoke infrastructure on AWS, uses Azure for AD and Office 365, and dreads migrating to Azure—Microsoft wins out-of-the-box services, AWS wins everything else.

Google Cloud’s E2 overselling and Cockroach Labs’ 2020 Cloud Report findings: the highest-performing machine types on CPU and network throughput tests were also top TPC-C performers (AWS c5n.4xlarge, GCP c2-standard-16, Azure Standard_DS14_v2). Storage performance winners varied in TPC-C, suggesting storage tests are less influential—OLTP workloads tend to be compute-limited given their high transaction-to-data ratios.

Serverless predictions for 2020: FaaS fades as service integrations dominate; leading serverless-first companies slash maintained lines of code by an order of magnitude, accelerating value delivery and experimentation; more tools like framework transpilers emerge for early-majority organizations.

Failure Modes and Timing

Boeing’s Starliner missed its ISS rendezvous because its internal clock went wrong, and the crew couldn’t reset it—out of satellite communication range. Another entry in the “it’s always the clocks” file.

From DigitalOcean’s “tale of tech debt”—actually an age of heroes: debugging a live Saturn V meant sending a team to rewire a fully fueled rocket based on a hunch about a relay module, with instructions to not let launch happen until the crew was halfway back from the pad.

There may never again be a canonical version of digital content everyone consumed and remembers. “Cats” got “resupplied” with improved visuals post-release—CI/CD has come to movies. Digital content has become the ultimate unreliable narrator.

Open Source and Research Roundup

Tools Worth a Look

Several projects caught attention this week, spanning machine learning orchestration, privacy hardware, and data processing.

  • Fairness Indicators (tensorflow/fairness-indicators) is built to help teams evaluate and improve models for fairness concerns, working alongside the broader TensorFlow toolkit.
  • Flyte (lyft/flyte) is an open-source, Kubernetes-native extensible orchestration engine. It manages Lyft's core machine learning pipelines, covering ETAs, pricing, incentives, mapping, and vision.
  • Betrusted (betrusted-io) takes a different approach to security: it is a system with its own screen and keyboard, designed from the ground up to be auditable by anyone but sealed only by the user. Privacy, in this view, begins and ends with the person using it.
  • Stroom (gchq/stroom) is a data processing, storage, and analysis platform from GCHQ. It aims to scale simply by adding more CPUs or servers, making it suited for high-volume data like system logs to gain insight into IT performance and usage.
  • Producer-Consumer Visualization (iximiuz/producer-consumer-vis) offers a visual take on the classic producer-consumer problem.
  • EZ-RASSOR (FlaSpaceInst/EZ-RASSOR) is an inexpensive, autonomous robot designed for regolith mining.

Papers and Ideas

On the research side, a mix of system design, AI retrospectives, and even lessons from ecology and space missions made the rounds.

  • AnyLog proposes a decentralized platform for publishing, sharing, and querying IoT data. It aims to let an unlimited number of independent participants publish and access structured data, much like the web did for unstructured data. The key difference is that AnyLog builds in incentives and financial rewards for tasks that keep the system healthy—contribution, integration, storage, and processing—while also enforcing good behavior via blockchain, secure enclaves, and state channels.
  • Amazon Aurora Papers Review makes a case for why engineers should read papers, pointing out that they often miss deep dives into algorithms, data structures, techniques, and lessons learned.
  • 2019 in Review: 10 AI Papers That Made an Impact spotlights artificial intelligence papers that gathered extraordinary attention and accolades during the year.
  • A less obvious read comes from The Community Ecology of Herbivore Regulation in an Agroecosystem. The spatial distribution of ants is oddly nonrandom—nests cluster near each other under shade trees, but large sections of farms have none. Evidence suggests this pattern is self-organized, emerging from the internal dynamics of the ant population itself rather than environmental forces, in a process similar to one Alan Turing described in 1952.
  • KRAKsat Satellite Mission - Lessons Learned reflects on how the "Space 4.0" age has shifted space exploration from political and economic grounds to academic and commercial fields, fueling growth in the CubeSat market as more organizations take on small satellite projects.