Scaling Through a Pandemic: The Numbers That Matter

COVID-19 has turned the internet into the world's primary workspace, classroom, and social venue almost overnight. The shift shows up starkly in the operational metrics tracked across the industry, from network throughput to distributed computing participation. As always, the numbers tell the story best.

One of the clearest illustrations of distributed computing's power in a crisis: Folding@Home has hit 1,500,000,000,000,000,000 operations per second (exascale) while processing COVID-19-related workloads. The project's citizen science model has proven capable of marshaling massive volunteer compute resources toward urgent biological research.

Tracking the Traffic Spike

The usage shifts are not uniform. They vary widely by platform and geography, revealing which services are absorbing the load and where infrastructure is straining.

  • Collaboration tools have seen explosive growth. Microsoft Teams now counts 44 million users, adding 12 million in a single week. Slack, which has 12 million total users, added 7,000 customers in seven weeks — beating its previous two quarters combined.
  • Entertainment platforms are also setting records. Steam hit 20 million concurrent active users, while podcast app Overcast reports weekday usage down 7% and weekend usage down 18% relative to pre-COVID baselines.
  • Ride-hailing has cratered: Uber reports a 70% reduction in rides.
  • Internet infrastructure is showing localized strain. An analysis of 200 cities found that 88 (44%) experienced some degree of network degradation in the third week of March compared to the prior 10 weeks. Only 27 cities (13.5%) saw dips of 20% or more below their normal ranges.

Security and Storage at Scale

Bad actors are capitalizing on the crisis. Barracuda reports a 667% increase in coronavirus-related phishing emails since the end of February. This surge in malicious traffic is arriving just as enterprise security teams are dealing with a sudden shift to fully remote operations.

On the storage side, Backblaze has crossed the 1 exabyte mark in customer data, a milestone that would have seemed extraordinary just a few years ago. At eBay, the engineering team reports managing 1 billion HDFS file system objects as part of its federation effort.

Cloud Spending and the Talent Map

Enterprise cloud consumption continues to concentrate heavily in specific providers. Samsung Electronics's annual AWS spend has reached $483 million — notable not only for its size but because Samsung owns Joyent, a competing cloud provider. The data point underscores how even companies with their own cloud infrastructure often end up as significant customers of the hyperscale public clouds.

The geography of AI hiring also remains lopsided. 90% of all AI-related jobs advertised on ZipRecruiter are concentrated in just four states: California, Washington, New York and Massachusetts. Remote work trends accelerated by the pandemic may shift this distribution over time, but the current data shows clear clustering.

Beyond the pandemic-specific metrics, the broader ecosystem continues to evolve. Space startups attracted $5.7 billion in investment during 2019, a 63% increase over the prior year's $3.5 billion. Reported open-source vulnerabilities are up to an average of 6,100 per year, compared with 4,100 last year — a reminder that the popularity of open-source software is a double-edged sword when it comes to security. And the temperature at the edge of space sits at -455°F, according to Discover magazine — a fittingly cold contrast to the heat generated by all this terrestrial activity.

For those looking to understand how these systems stay operational under such unusual conditions, a video from BPS.space offers an excellent practical guide: it demonstrates how to build a PID controller to fly a rocket, a classic exercise in real-time control that parallels the challenges of managing scaling systems in turbulent times.

From the Community

This week’s roundup of distributed systems chatter focuses heavily on the operational and financial disruptions of the pandemic, along with the usual mix of architectural hindsight and technical grievances.

Cloud Economics Under Pressure

The sudden demand collapse has operators questioning cloud billing models. As @bencurthoys asked AWS: is there any plan to let customers cancel reserved instances during the crisis? His point: "No one can use my services, but I'm still paying you full whack for them whether I shut the servers down or not."

Corey Quinn summed up the perennial problem with a different framing: "you aren't billed for what you use, but rather what you forget to turn off." He also noted the upper bounds of that model, pointing to an all-upfront reserved instance for a db.r5.24xlarge Enterprise Multi-AZ SQL Server in Bahrain costing $3,118,367 per API call.

On the vendor-lock-in debate, Wisen Tanasa proposed a clean formula: Lock-in cost = Migration cost - Opportunity Gain. At the other end of the spectrum, a former Facebook infrastructure engineer argued that the company's internal job scheduling abstraction (Tupperware) is "about 5 years behind—something like Borg or EC2/EMR," and that its infra teams' "impact at all costs" culture is a fundamental blocker to competing in cloud.

Capacity and Traffic Reality Check

Verizon's CEO reported web traffic spiked 20% in one week as shutdowns took hold. Meanwhile, mobile data paints a more nuanced picture: @benedictevans noted UK mobile traffic was down slightly, with people on home WiFi, while roaming traffic fell 55% in five days. His separate look at MTN (250M users, mostly Sub-Saharan Africa) showed ~50% smartphone penetration and only ~40% of the base as active data users, but average data use of 2.8 GB/month growing 46% year-on-year.

Google Cloud suffered a multi-service incident this week. The company's status page listed elevated error rates across a long roster of products, from Dataflow and BigQuery to Compute Engine and Cloud SQL.

On the hardware side, Memory Guy (Objective Analysis) predicted 2020 will be a down year for the chip market. The capEx-driven oversupply from 2018 was already forecast to cause a downturn; COVID-19 just accelerates it. "The net impact of COVID-19 will be to cause an earlier downturn in 2020 than would have otherwise occurred, but the impact is unlikely to go beyond that."

Architecture Fights and Performance Notes

Rick Houlihan shared a Lambda-specific DynamoDB tuning tip: set the @DynamoDB connect timeout to 100ms. With the default of 60 seconds, Lambda functions can hang waiting on a dropped connection request; he reported seeing this twice recently, and lowering the timeout fixed both.

On garbage collection, @cliff_click argued GC can be slower than malloc. In high-churn apps, malloc recycles memory in-cache, whereas GC typically burns through a generation before repeating addresses, leaving the working set out of cache. He claimed ~5x speedups on streaming Java using object pools versus GC.

GraphQL collided with DynamoDB's single-table design, as @rafalwilinski observed: "you can't really predict all access patterns," and their developers started requesting everything on login with 7–8 levels of nesting.

A Reddit developer (f_fat) announced a pure-Swift JSONEncoder/JSONDecoder with no third-party libraries and no Foundation, claiming 1.5–2x faster encoding/decoding on macOS and 8–10x faster on Linux.

Mikael Ronstrom touted MySQL Cluster's performance using the Yahoo Cloud Serving Benchmark (YCSB) Workload A: 1.4M operations on 2 Data Nodes and 2.8M on a 4 Data Node setup, all via a standard JDBC driver.

Rosenblum mentioned work with Balaji Prabhakar on a new clock sync algorithm that synchronizes clocks "down into the single-digit nanoseconds."

Security Roundup

tptacek weighed in on security education, arguing a 2020 web security class "needs to be teaching about SSRF, the most important current web bug class," with OAuth flows a close second.

Geoff Huston offered a bleak assessment of certificate revocation: "we are placing undue reliance on a security framework that at best offers same week service in a nanosecond world."

FireEye observed APT41, a Chinese cyber espionage actor, launch one of the broadest campaigns seen in recent years. Between January 20 and March 11, the group attempted to exploit vulnerabilities in Citrix NetScaler/ADC, Cisco routers, and Zoho ManageEngine Desktop Central against over 75 FireEye customers across 20 countries and 20 industries.

In storage research, Stathis Maneas presented findings on drive reliability indicating that flash type (eMLC vs. 3D-TLC) matters less than lithography (1xnm vs. 2xnm) or capacity.

Systems Thinking in a Crisis

Zeynep Tufekci tied the "flatten the curve" concept directly to systems thinking: it calls for isolation not because any one person is at great risk, but to avoid overwhelming hospitals in aggregate. "R0 is not a fixed number: If we isolate ourselves, infectiousness decreases. If we keep traveling and congregating, it increases."

From a frontline perspective, @ZLevyMD described a hospital reorganizing under load: "We are now a monolith. We started with a medical ICU, surgical ICU, cardiac ICU, neurosurgical ICU, and a cardiothoracic ICU... There is no more specialization—we're all treating one thing."

Charity Majors suspected many companies have 2x–4x the headcount needed to build and support their core product because "they never understood their breaky, flaky systems, so they had to plaster over the problems with people."

Language and Platform Gripes

@AmyZenunim was unimpressed with Go: "it feels like they've mashed the worst parts of C and Javascript syntax together, with the package dependency hell of early-2010s Ruby, and the elitist community of early-2000s Linux fanboys."

A former Googler (ridiculous_fish) critiqued the new book on Google's engineering practices, noting it's "a very google3 (i.e. web services) centered book" that ignores Chrome, ChromeOS, and Android, where processes differ. "During my time there, the Android team was recruiting internally, advertising 'come work on Android, we don't require Readability.' It was seen as an internal competitive advantage to reject these processes!"

History and Hardware Limits

Alan Kay, looking back at the Internet, said it was "done so well that most people think of it as a natural resource like the Pacific Ocean," adding, "The Web, in comparison, is a joke. The Web was done by amateurs."

Arthur Holland Michel detailed the computational weight of ARGUS, a surveillance system with 1,854,296,064 pixels, generating 27.8 GB of raw pixel data per second and pushing its 33,000 processing elements through 70 trillion operations per second.

Marc Andreessen reflected on the limits of lean methodology: "SpaceX and Tesla were not lean startups... The big question is what about the efforts where you have to say, 'This thing is going to take $300 million?' There's no shortcut and there's no minimum viable product."

ARM64 Graviton2 Wins Real Workloads

Honeycomb.io ran its production service on AWS's ARM64-based M6g instances and found the new platform beats the x86 C5 in every dimension they measured. Not only does M6g cost less on demand, it offers more RAM, lower median latency, significantly narrower tail latency, and runs cooler under the same proportional workload per host.

One workload saw instance count drop by 30% while each instance cost 10% less than the comparable C5 — the only change required was recompiling with GOARCH=arm64 plus some infrastructure wrangling. After a week-long bake entirely on Graviton2, peak-to-peak instance count decreased by more than 27%, and baseline load needed 37.5% fewer instances, averaging over 30% fewer instances overall. For Honeycomb, that works out to a 40% reduction on the EC2 bill for that service once the full conversion to Graviton2 completes.

The context: SemiAccurate's Charlie Demerjian argues Intel's cancellation of a mainstream server platform shows how untenable the company's roadmap has become — a situation he says has worsened since his earlier piece, "Intel has no chance in servers and they know it," which had already concluded Intel can't beat AMD until after its 2022 platform.

When Cloud Capacity Runs Out

Reports that Azure appears "full" underscore an often-forgotten fact: a cloud is a collection of servers, and when a pandemic disrupts the server supply chain, even nominally infinite capacity becomes finite. Commenters with direct experience offered context:

  • Cloud providers grow at a high rate continuously, and that's fine when they can order more racks. When supply is impacted, they fall behind — and headlines like "Azure is full" follow. People working from home increases demand, but the real problem is an unhealthy supply chain.
  • One team saw bizarre, inconsistent failures on nodes in a single Azure data center that was over capacity. After moving resources to another location, issues vanished — though Azure was not forthcoming about its capacity problems.
  • A videoconferencing provider saw traffic jump 20x in three weeks and initially struggled to get servers from AWS. AWS added them to a higher priority lane for support and capacity because they were helping maintain quarantine, and guided them toward regions with spare capacity of the instance types they needed.
  • A healthcare application provider noted Azure capacity issues predate the pandemic — there were times last fall when you couldn't provision new resources — and the supply chain situation has not improved. Healthcare's reliance on cloud infrastructure is growing, especially with telehealth now the default for non-emergency issues.
  • One Hacker News commenter raised the disaster recovery question: is it safe to assume the cloud will always have capacity? In extreme situations, capacity stress becomes more likely, and it's worth asking whether cloud vendors will learn to build slack into their systems.

Tesla's Virtual Power Plant Architecture

A presentation on Tesla's Virtual Power Plant shows how software can unite thousands of small home batteries into a grid resource. The power grid requires supply and demand to match in real time — frequency and voltage deviations can damage devices and cause blackouts, and the grid itself stores no power. Tesla's approach treats thousands of distributed batteries as a coordinated virtual power plant serving both the electrical grid and the home or business owner.

The platform's foundation is a linearly scalable WebSocket front end handling connectivity and security. Behind it, a Kafka cluster ingests high-volume telemetry from millions of IoT devices, providing messaging durability and decoupling data publishers from consumers. A publish-subscribe messaging service enables bi-directional command and control. These three shared infrastructure services support roughly 150 polyglot microservices.

APIs fall into three domains: querying or streaming telemetry, alerts and events; describing energy assets and their relationships; and commanding and controlling devices like batteries. Postgres models asset relationships, with Kafka integrating changes; at scale, devices self-reporting their configuration and state are the most reliable source of truth, and "digital twin" modeling represents physical devices in software.

The team relies heavily on Akka, a distributed computing toolkit with actor-model programming well-suited to modeling individual entities like a battery, plus Akka Streams for flow control with backpressure. Most microservices run on Kubernetes, which handles coarse-grained failures — scaling pods, liveness probes, restarting failed pods — while Akka handles fine-grained issues like circuit breaking, retrying individual requests, and tracking state like whether a battery is charging. Scala is the primary language, chosen partly because it's the first-class way to use Akka; the team values its compile-time safety, immutability, and functional programming style. The initial platform used HTTP APIs and JSON, but a year-long investment in gRPC proved to be a big win.

Tesla's vertical integration of hardware, firmware, and software makes this distributed algorithm work: devices behave sensibly during the communications failures that inevitably occur in distributed systems.

Dropbox Rebuilds Its Sync Engine in Rust

Dropbox rewrote the heart of its sync engine — called Nucleus — in Rust, a choice the team calls a "force multiplier." More than performance, Rust's ergonomics and focus on correctness helped tame sync's complexity: complex invariants are encoded in the type system and checked by the compiler.

The client-server protocol was redesigned for strong consistency, guaranteeing that server and client share the same view of the remote filesystem before any mutation is considered. Shared folders and files now have globally unique identifiers, and clients never observe them in transiently duplicated or missing states. As one engineer explained, the protocol breaks down into two subproblems: syncing a view of the remote filesystem to clients, and letting clients propose changes. They solved both with strong consistency guarantees, then built an operational-transform-flavored protocol on top. Protocol-level inconsistencies had plagued the classic sync engine — much debugging went into understanding how a client's view got into a bizarre state.

A favorite technique was deterministic simulation. Nearly all logic runs on a single thread, with futures multiplexing concurrent operations. Everything on that thread is deterministic given fixed inputs — a challenge, since code can sneak in dependencies on global random number generators or time. The team uses traits for interfaces between the control thread and other threads, mocks external time behind a trait, and wraps each real component in a mock that pauses requests into a wait queue. This approach caught bugs ranging from simple logic errors to race conditions they'd never have thought to test.

One commenter notes Dropbox had already adopted Rust for serving infrastructure more than a year before the sync rewrite started, and believes the rewrite succeeded because Rust can both interact safely with OS APIs and encode complex concepts in the type system. Internal RPC uses protobuf as the interface description language, similar to gRPC but for in-process communication.

Brief Notes

  • AWS Lambda batching: Sending ten messages individually takes 700–800ms consistently; using send_message_batch, all ten complete in about 300ms — roughly 60% faster.
  • Serverless in the wild at Azure: Over half of apps have just one function, 95% have at most ten, and only 0.04% have over 100. HTTP is the most popular trigger; only 2.2% of functions are event-triggered, but they account for 27.4% of all invocations. The fixed 10-minute keep-alive policy causes about 2.5x more cold starts at the 75th percentile than a histogram-based policy ranging up to 4 hours, using the same memory.
  • Serverless fault tolerance: AWS Lambda and Google Cloud Functions retry failed executions but offer no guarantees about cleaning up partial updates to shared state. AFT (Atomicity for Fault Tolerance), built at UC Berkeley's RISELab, is a shim layer between the compute and storage layers that treats each logical request as a transaction, atomically installing updates at the storage layer. Implemented in a few thousand lines of Go over DynamoDB, S3, and Redis, it scales to hundreds of parallel clients and thousands of transactions per second.
  • Cloudflare's edge expansion: Its network spans 200 cities in 90+ countries, but the new partnership with edge startups Vapor IO and EdgeMicro is different — small data center sites designed for edge computing. Current roundtrips of 50–75ms could drop to sub-20 or even sub-10ms. The "middle mile" between access networks and regional data center interconnection is the weak link for low-latency workloads: in Austin, all traffic is backhauled to Dallas, and major metro areas route through only seven to nine internet "drains."
  • On scale and numbers: JavaScript can't handle numbers beyond 53 bits, so when IDs grow to 64 bits, string-based IDs save pain later. Average response times hide more than they reveal; percentile metrics like perc50 and perc95 show what most users actually experience. Facebook's NTP service uses four strata: satellites with atomic clocks, atomic clock synchronizing with GNSS, a pool of NTP servers with leap-second smearing, and a large tier of servers receiving smeared time.

Open Source for Reliable Systems and Game Servers

Microsoft's coyote project offers a programming framework aimed at building reliable asynchronous software. Its core promise is keeping design and code synchronized, which simplifies adding new features. The framework includes a systematic testing engine capable of finding and deterministically reproducing difficult safety and liveness bugs. Several teams within Azure use Coyote to design, implement, and systematically test production distributed systems and services.

For developers working with game infrastructure, googleforgames/agones provides a library for hosting, running, and scaling dedicated game servers on Kubernetes. Separately, google/oss-fuzz enables continuous fuzzing of open source software to uncover vulnerabilities.

Privacy-Preserving Contact Tracing

The MIT Private Kit: Safe Paths project represents a first-generation contact-tracing tool designed for the COVID-19 crisis while addressing privacy concerns. The project argues that first-generation tools can be—and have been—used for mass surveillance and exposing private details. In contrast, these citizen-centric, privacy-first solutions are open-source, secure, and decentralized, representing the next generation of tools for disease containment during an epidemic or pandemic.

Studying Novelty and Scientific Incentives

Research from the National Bureau of Economic Research on Stagnation and Scientific Incentives demonstrates empirically that measures of novelty are correlated with, but distinct from, measures of scientific impact. This finding suggests that incorporating novelty metrics into scientist evaluation could encourage researchers to pursue more innovative, riskier projects.

The Economics of Ephemeral Caching

The InfiniCache project explores using ephemeral serverless functions to build a distributed in-memory cache in front of cloud object stores. On some workloads, InfiniCache proves to be one to two orders of magnitude cheaper than ElastiCache (a managed Redis cluster). For large objects, it outperforms S3 significantly—by at least 100x for roughly 60% of large requests. In production-like testing, the system achieved 95.4% availability, which is substantially higher than the 81.4% observed for InfiniCache without backups. This demonstrates the efficacy of placing a distributed in-memory cache before an object store.

Millions of Tiny Databases

Commentary on the Millions of tiny databases paper highlights the importance of second-order effects in large-scale systems. While these effects might seem minor, they become critical at AWS scale. If EBS comprises millions of nodes requiring configuration, relying on a single configuration box is impossible. The configuration box typically sees little traffic, but when issues occur, traffic becomes bursty. If the configuration layer fails during these peaks, the situation worsens significantly. The paper cites the "21 April 2011 cascading failure and loss of availability" as a prime example of these dynamics.

A Different Approach to Consensus

PigPaxos addresses communication bottlenecks in distributed consensus by decoupling communication from decision-making at the leader. Instead of direct leader-to-follower communication as in standard Paxos, PigPaxos introduces a relay-based flow. Relays are chosen randomly from follower clusters at each communication round, reducing contention and improving throughput scalability.

Workload Characterization at Facebook

A USENIX FAST paper on characterizing, modeling, and benchmarking RocksDB key-value workloads at Facebook reveals distinct patterns in their production use. The research uncovers several insights: the distribution of key and value sizes is highly dependent on specific use cases; access to key-value pairs shows good locality and follows special patterns; and performance metrics have a strong diurnal pattern in the UDB, but do not exhibit this periodicity in the other two analyzed databases.