The Numbers That Matter This Week

HighScalability's periodic roundup of the infrastructure and internet statistics worth knowing, from breach databases to cloud growth to the physics of time itself.

Scale Milestones

Troy Hunt's Have I Been Pwned crossed a sobering threshold: its database now holds 10 billion breached records, up from 1 billion in 2016. Google claims its latest TPUs are 2.7x faster than the prior generation. Meanwhile, Azure's growth decelerated to 47% in Q4 2020, down from 59% the previous quarter.

Infrastructure Under the Hood

Subsea cables carry 98% of international internet traffic, and Google is adding to that backbone with the Grace Hopper cable, connecting the United States, the United Kingdom and Spain. Stack Exchange's public performance numbers offer a rare look at a focused architecture: 300 req/s sustained (450 peak), 55 TB of data per month, and a surprisingly lean footprint — 4 SQL Servers, 2 Redis servers, 3 Tag Engine servers, 3 Elasticsearch servers and 2 HAProxy servers. The platform sustains 600,000 web socket connections, renders the home page in 12.2 ms and a questions page in 18.3 ms, serving 1.3 billion page views monthly.

Uber's microservice architecture has a measurable decay rate: a 1.5-year half life for its services. And Wix now hosts 5% of the world's websites, drawing 700 million uniques per month; the platform executed 1.2 million browser-based tests over five days using Lambda.

Attention and Revenue

Twitch's top five games pulled in 82 million hours watched in a single week, with 2.2 million average weekly viewers. On YouTube, $97,000 is the reported earnings for a video reaching 150 million views. Google's own search results refer to its own products 41% of the time, raising questions about the company's implicit bargain with the open web.

Security and Systems

ATM "jackpotting" attacks are getting faster: hacked machines dispense 40 bills per 23 seconds. Some forensic evidence suggests the black boxes used in these attacks contain individual components of the ATMs' own software stack.

Historical and Theoretical

In a retrospective on warehouse-scale computing, the VAXcluster is cited as the first milestone. Physicists continue to probe whether time itself has a minimum quantum increment, with some theories suggesting the smallest possible tick is on the order of a millionth of a billionth of a billionth of a billionth of a second.

US mobile internet speeds rose 15.8% in Q2 2020, with fixed broadband up 19.6%. Online grocery sales hit a record $7.2 billion in June. Apple's developer ecosystem now numbers 23 million.

Cloud Economics and the Cost of Convenience

A recurring theme in operational post-mortems and engineering commentary is the gap between cloud rhetoric and actual spend. Charles Fitzgerald frames this as “hyperclown status”: the ratio of cloud marketing to capital expenditure, where the most efficient players maximize promotion while minimizing real investment. Corey Quinn echoes the sentiment with a blunt breakdown of where cloud money goes: the bulk of spend is the direct cost of EC2 instances, followed by RDS, Elastic Block Store, S3, and data transfer—the latter two being indirect EC2 costs. “The cloud really is a bunch of other people's virtualized computers being sold to you,” he concludes. The rest is largely window dressing.

That window dressing can be expensive. One team reported a $4,000 increase on an $8,000 bill traced to a single obscure change: adding a VPC to a global serverless.yml scope to reach MongoDB routed all Lambda traffic through a NAT gateway, including an ML job shuffling models to and from S3. Paying NAT gateway rates for S3 traffic is a costly mistake. Similarly, DynamoDB On-Demand capacity can double costs versus provisioned. One user found the premium visible in just three days of switching to monitor capacity needs, and noted that converting back from On-Demand is limited to once per day. The rule of thumb: avoid On-Demand unless your load is genuinely unpredictable.

Scale dictates which optimizations are worth pursuing. A 10% efficiency gain on a ten-instance system saves maybe $2,000 a year—an irrelevant rounding error for a startup. The same 10% on a 20,000-instance fleet saves $4 million annually, justifying a month of engineering effort. One engineer's counterpoint: rather than adding Redis or another network hop, an in-process cache like Caffeine took CPU from 50% to 1% with no serialization cost or extra infrastructure. A Java back-end peaked over 4,000 requests per second at 5% CPU with no caching, two instances for HA, and DNS round-robin, while a comparable Rails team ran ten instances, added caching, and still chased sub-100ms latency.

Migration advice is similarly pragmatic. A lift-and-shift to AWS worked well for one organization, which then iterated on optimization with strong management support, clear timelines, and a countdown to data-center exit. LinkedIn, after a six-month messaging-platform data migration, advises deferring timelines until throughput is certain; real-world hardware differences forced multiple revisions to their initial estimates.

Serverless Trade-offs and Complexity

Serverless is praised for bringing clarity to system design. Discrete components with greater shared context make diagrams communicate intention more effectively, and it becomes easier to predict where scalability, reliability, or security issues will arise. Yet the operational reality can be messier. One engineer “loves” serverless but moved back to containers because infrastructure complexity plus per-resource IAM policies of least privilege became untenable. Another observed that no team has switched from containers to serverless and magically gotten faster—they probably got slower. The simplest solution remains the best until it becomes the problem: use a monolith, a LAMP stack, or printf debugging until the day the tool's edge cuts you.

Kubernetes drew sharp criticism from practitioners. One consultant replaced two REST APIs running on EC2 with handrolled redundancy and zero-downtime deployments—a handful of shell scripts—with a Kubernetes cluster; server costs jumped 2,000% and downtime became routine for reasons that took hours for experts to diagnose. Another warned never to self-host Kubernetes without a full-time person or team: too many moving parts, weird errors, and cryptic log messages. Even a proponent admits the developer experience is decent, but secrets handling pushes you toward something like HashiCorp Vault.

API choice can be a trap. Steve Yegge compared using Google APIs to a choose-your-own-adventure where almost every path runs two years before deprecation, singling out Google Cloud Endpoints and App Engine Java as examples of neglected docs and libraries.

Security, Trust, and Platform Power

Twitter's July 2020 breach—130 accounts targeted, 45 with tweets sent by attackers, 36 with DM inbox access, and 8 with full data archives downloaded—renewed scrutiny of insider threats. DSHR noted that Google hardened against phishing years ago, yet Twitter remained vulnerable to insider attacks after at least seven years of known incidents, arguing the board should fire Jack Dorsey. Bruce Schneier framed the event as a “class break”: a vulnerability that breaks an entire class of systems, not just one, likening it to a remote-control flaw in every computer running a given OS.

Platforms face accusations of overreach. A former AWS employee claimed their team proactively tracked traction of hosted products, built competing offerings, and scraped customer lists. Apple's App Store policies drew ire from a developer told that eight years without contributing to App Store revenue meant they'd added no value—despite their app being part of an ecosystem that sells $1,000 phones. The company believes it is entitled to all commerce on the iPhone, as one analyst noted. Meanwhile, Google's evolution from “get you out of Google and to the right place” to becoming the destination itself is backed by data: mobile search clicks on free links dropped from 40% to 27% between 2016 and 2019, while no-click searches rose to 62% and ad clicks tripled.

Comments thread into broader questions of lock-in. Beyond data gravity, identity and access management is the overlooked form of vendor lock-in. Rate limiting deserves attention too: an API client without it is “an exception generator with a remote timer.”

Outages, Monitoring, and Failure Modes

Cloudflare attributed a 23-minute outage affecting 50% of its network to a typo in a router configuration on its private backbone. Slack's “terrible day” involved autoscaling to its highest-ever webapp instance count—a 75% increase—as workers waited longer for database requests during pandemic-driven load. Notably, broken monitoring went unnoticed because the system had “just worked” for so long. A sobering observation: “received input in a form never imagined by the developer” may be a more common distributed-systems failure mode than race conditions.

Hardware and Infrastructure Contrasts

Not every workload needs the cloud. A single beefed-up application server can handle 10,000 concurrent connections if written in a compiled language with an in-process LRU cache and heavy computation pushed to app servers. OmniSciDB (formerly MapD) on an off-the-shelf MacBook Pro can perform near-instant analytics on billions of records without dedicated Linux clusters or GPUs.

DDR5 memory, with its spec finalized up to 6,400 GT/s and 2 TB LRDIMMs, brings dual 32-bit channels per DIMM. Compared to DDR4's single 64-bit channel, a dual-channel DDR5 system will behave more like quad-channel, likely delivering a noticeable GPU-bound FPS increase, starting from a much faster DDR5-4800 baseline versus officially supported DDR4-3200.

Culture, Process, and the Human Element

Engineering culture debates persist. One commenter has never seen a workplace built on genuine connection, describing environments as adversarial with a thin veneer of “we're a family”—a world their retired parents describe differently. Another argues software engineering doesn't demand nights, weekends, or all-consuming passion; it's acceptable to treat it as “just” a job, even one you like.

Startup success correlates with total focus on shipping, even at the cost of tech debt. Spending 30 engineer-days fixing an issue later can make sense when a few hours upfront represent 0.14% of a 40-hour week and a $200,000 seed round's runway. At a later stage with $40 million raised, the calculus inverts—but few get there without shipping first.

NASA's Perseverance rover offers a longer-view lesson in testing: with a round-trip radio delay of up to 44 minutes, autonomous navigation is the only viable control mode. Most algorithm tests ran in simulation against every conceivable scenario. SpaceX and Boeing embody two philosophies: rapid build-test-learn iteration versus up-front systems engineering before hardware exists.

High-frequency observations can hurt accuracy—sampling more often yields spatially correlated data that misrepresents the true environment, making decisions less precise. More generally, positive feedback loops and interval timing matter. The cost of the network itself carries information: “just because we can do a thing doesn't mean we should.” And on a personal note, one engineer captures the lifecycle: eventually you enjoy deprecating old services as much as building new ones, because you never have to teach anyone about them again.

Batteries and Computation

Qnovo reports on lessons from shipping its battery software in over 100 million smartphones. The core insight borrows from camera design: instead of tightly controlling manufacturing tolerances with expensive hardware, use software to compensate. Just as cheap plastic lenses paired with computational photography can match or exceed DSLR quality, battery intelligence can substitute for manufacturing accuracy.

Defects are treated as an inevitable part of production. The software’s predictive algorithms aim to spot those defects well before they become a safety hazard, then manage the battery to reduce failure risk. Reported figures from Qnovo’s fleet: an average longevity of 1,900 cycles at 25 °C and 1,300 cycles at 45 °C, versus the 500–1,000 cycles typical of an average smartphone.

Uber’s Middle Path After 2,200 Microservices

Uber describes the problems encountered after growing a microservice architecture built from 2,200 services: dependency graphs become hard to reason about, and latency spikes in deep dependency chains can cascade upstream. The company’s answer is Domain-Oriented Microservice Architecture (DOMA).

The argument is that a microservice architecture is effectively one large distributed program, and should be structured as such. DOMA’s elements:

  • Domains: collections of related microservices, rather than single services as the unit of organization.
  • Layers: collections of domains.
  • Gateways: clean, single-point-of-entry interfaces for each domain.
  • Agnosticism: each domain should avoid hard-coding logic about other domains in its own code or data models.

Uber acknowledges that microservices were necessary to reach its scale but admits that complexity grew as the engineering organization expanded from hundreds to thousands of engineers.

Systems and Load Balancing at Scale

Picnic, an online grocery operation in the Netherlands, describes how it handled traffic spikes during the COVID-19 pandemic. Systems run on AWS with Kubernetes (EKS) for compute and managed services for data. A key detail: the biggest traffic peaks arrive when new delivery slots open, so the team pre-empts the surge by combining Kubernetes’ Horizontal Pod Autoscaler with a CronJob that raises autoscaler minimums and maximums ahead of time. The Cluster Autoscaler handles node capacity. Other changes included reducing unnecessary internal service calls, adding caching at hot spots, and introducing random jitter to a polling cache to avoid thundering herds.

Twitter explains its deterministic aperture load-balancing algorithm, which uses the power-of-two-choices (P2C) strategy. For each request, the balancer samples two instances at random and selects the one with less load. Reported results: a 78% reduction in relative standard deviation of request load, and a 91% drop in connection count (from roughly 280K to 25K) on one service topology.

Wayfair writes about consolidating Kafka mirroring pipelines: from 212 distinct Mirror Maker clusters down to 14 Brooklin clusters, handling 33 local and 17 aggregate Kafka clusters. The migration used 57% fewer CPUs and processed over 15 million messages per second across thousands of machines, with no global downtime.

Scaling by the Numbers

Several posts offer tiered approaches to scaling. One guide starts with the premise that nearly every application has an API, a database, and a client. The milestones: split the database layer at ~10 users, split out clients at ~100, add a load balancer at ~1,000, introduce a CDN at ~10,000, then tackle caching and read replicas at ~100,000.

Another post makes a distinction between raw performance and scalability. Performance is about handling a single unit of work efficiently; scalability is about total volume. A site under 100,000 dynamic requests per day may have hard performance problems but generally not true scalability problems. Above 1 million dynamic requests per day, a site crosses into a different operational territory. As one bottleneck is fixed, the next inevitably appears — application servers, then database reads, then writes.

John Deere’s presentation traces a longer historical effort, from batching and Lambda-based processing through stream processing with Flink and back toward a hybrid model. What started in 2014 as ~5,000 measurements per second aggregated in 30-minute summaries scaled to 12 million measurements per second by 2018, sampled five times per second from thousands of machines in the field. The team repeatedly re-architected, moving between Lambda, Spark over EMR, Flink, and eventually back to batching with ECS. Storage requirements went from hundreds of terabytes to petabytes and then dropped to 2 PB — about 2% of what would have been needed with full pre-computation.

Cost, Pricing, and Commoditization

Analysis from RedMonk finds cloud providers are largely no longer competing on price for base compute. When controlling for instance size, standard infrastructure pricing has leveled off, and the authors conclude that base offerings are reaching commodity status. Instances are still important, but price is no longer the primary differentiator.

A separate case study claims an AWS bill around 2% of revenue. The author’s recommendations: use Lightsail rather than EC2 (~$20 versus ~$37 per instance), a Lightsail database rather than RDS (~$60 versus ~$200), self-hosted Redis (~$40) instead of ElastiCache (~$112), NGINX rather than a load balancer where feasible, and a free CDN.

Two Weeks of Launch Chaos

The Basecamp podcast describes what happened during the launch of HEY, its first cloud product. The episode covers the practical realities of a product taking off: triaging bugs without panic, hiring temporary support staff, and building an onboarding process under pressure. They also created a “bulkhead” separating abusive behavior with temporary email accounts from paying customers. Basecamp credits the cloud with letting them scale in days rather than weeks.

Bits, Books, and Building Blocks

For those looking to train reinforcement learning agents faster, the Sample Factory codebase is built for high-throughput, asynchronous training. On the data store side, GhostDB offers a distributed, in-memory, general-purpose key-value store that aims for microsecond performance at any scale by keeping data in RAM and reducing reads from external databases or APIs.

Two video items caught attention this week. The Bit Player documents Claude Shannon's life and his 1948 introduction of the "bit," tracing how his curiosity shaped computing, genetics, neuroscience, and AI through a mix of interviews, archival footage, and animation. For a more historical look at infrastructure economics, Thomas Edison's footage of the Klondike Gold Rush is a reminder that the real money went to the suppliers; of the thousands who rushed north, only 30,000 reached the fields, extracting 75 tons of gold, and very few of them profited.

Research and Reading

A cluster of new papers covers consensus, graph learning, and systems performance. Matchmaker Paxos introduces a reconfigurable consensus protocol (and its MultiPaxos counterpart) that claims to execute reconfigurations in a single round trip theoretically and in only a few milliseconds in practice, with minimal impact on command processing latency or throughput. Another paper, Learning Graph Structure With A Finite-State Automaton Layer, tackles the problem of deriving abstract relations from graph structure by treating them as paths accepted by a finite-state automaton. The authors show how to learn these relations end-to-end by relaxing the task into learning finite-state automata policies on a graph-based POMDP, resulting in a differentiable Graph Finite-State Automaton (GFSA) layer that adds a new, weighted edge type to a base graph.

Facebook's engineering team shared details on its scalable data classification system for security and privacy, which combines data signals, machine learning, and fingerprinting to map and classify data assets across dozens of data stores. The system is in production and reports an average F2 score of 0.9+ across privacy classes.

Several other publications are worth a look:

  • The full ISCA 2020 program is now available.
  • "There's plenty of room at the Top" argues that as silicon fabrication improvements at the Bottom wane, gains will have to come from software performance engineering, algorithm development, and hardware streamlining. These top-level improvements, however, will be opportunistic, uneven, sporadic, and subject to diminishing returns.
  • "Demystifying the Real-Time Linux Scheduling Latency" uses the Thread Synchronization Model of Linux to derive properties and rules about kernel scheduling behavior, leading to a sound bound on scheduling latency that accounts for all delay sources in any sequence of kernel synchronization events.

Books and Debugging Practices

Brendan Gregg's Systems Performance: Enterprise and the Cloud, 2nd Edition has arrived, adding material on BPF, BCC, bpftrace, perf, and Ftrace. The new edition drops most Solaris content in favor of extensive Linux and cloud updates, and it benefits from Gregg's six years as a senior performance engineer at Netflix and a review team of over 30 engineers.

For those dealing with distributed systems, the LogPlayer paper details a component for fault-tolerant, exactly-once delivery of transactional mutations from a write-ahead log to backend storage shards, built on gRPC asynchronous streaming. Meanwhile, a piece in ACM Queue on Debugging Incidents in Google's Distributed Systems highlights how debugging style varies by role: software engineers tend to jump to logs early to find error indicators, while site reliability engineers apply a more generic approach, scanning service health metrics for common failure patterns and digging into logs only when the mitigation path is unclear. Notably, engineers with a decade or more of experience lean on legacy tools, while newer engineers adopt recently developed ones. The article also categorizes incident complexity by blast radius and maps symptoms to six common underlying causes: capacity, code changes, configuration changes, dependency failures, infrastructure issues, and external traffic problems.