The Numbers Behind the Internet's Infrastructure

Ecosystems, both digital and biological, run on astonishing scales. The world's trees share a fungal network, while the cloud's biggest spenders continue their multi-billion-dollar arms race. Here's what the numbers show this week.

The high cost of platforms: Spending across Alphabet, Amazon, Microsoft, Facebook and Apple totaled $900 billion in combined R&D and capital expenditure over the last decade, with $184 billion of that coming in 2019 alone. On the more economical side, AWS now offers up to 17% savings on Lambda workloads if you commit to a three-year term. Google has pledged $10 billion this year to build out its datacenters and offices. Microsoft deploys 82,000 times each day.

Not everything on those platforms is legitimate. Analysis suggests that 30% of Amazon reviews are fabricated, a figure that climbs to 52% on Walmart.com. The integrity of viral content is also questionable—roughly 25% of tweets about the climate crisis are generated by bots rather than humans, and DDoS attack volume nearly doubled between Q4 2018 and Q4 2019, likely fueled by non-standard protocol amplification.

Scale of User Data and Trust

Facebook's sprawling social graph continues to present challenges. A typical user has 40,000 friends-of-friends (FoFs), while a power user with thousands of friends might reach 800,000 FoFs. When asked what it would take to share contact information, U.S. users demanded $3.50 per month; German users held out for a steeper $8.

One Microsoft employee learned the hard way about retail fraud, now facing up to 20 years after allegedly stealing $10 million from the company's own online sales platform, which they helped test.

A Growing Backbone of Certificates and Files

Encryption adoption and data backup have crossed notable milestones. Let's Encrypt has issued one billion certificates, a free service that enforces renewal every three months. Meanwhile Backblaze reports having restored 50 billion files, split almost evenly between Mac and Windows, with 1.06% of restores exceeding 500 GB.

The sheer volume of web services is reflected elsewhere, with Google reporting over 36 trillion calls to its font API. Yet the scale of moderation is also massive: YouTube received more than 109,000 appeals, reinstating 23,000 videos after removing 5 million videos and over 2 million channels between October and December.

Quirks of the Real and Virtual World

Some numbers defy neat categorization. The Smithsonian offers open access to millions of pieces, a three-dimensional representation of the 1903 Wright Flyer included. Compromised CISO positions see an average tenure of only 26 months, proving burnout isn't exclusive to developers. Even biological cryptology has scaled: two-thousand-year-old Judean date palm seeds have been sprouted, and one algorithm spotted 11 potentially Earth-destroying asteroids that NASA's software missed.

Creative compensation remains elusive in the digital age. The person behind TikTok's biggest song, which has sparked over 18 million videos, has earned exactly zilch in royalties. For those seeking validation, one Amazon book on cloud computing holds 98 mostly five-star reviews—a particularly interesting metric given the prevalence of fake reviews elsewhere.

The Week in Scalability: Quotes and Notes

Another week brings another collection of sharp observations from across the technology landscape. Here’s what stood out to us.

On AI’s Missing Ingredient

Gary Marcus adds his voice to the growing chorus questioning the single-minded focus on big data for AI advancement. His judgment is blunt: the current paradigm is “long on data, but short on knowledge, reasoning and cognitive models.” For AI to be trustworthy, particularly in high-stakes domains like autonomous vehicles or medical diagnosis, Marcus argues we need systems with a deeper causal understanding of the world, not just the ability to find subtler correlations in massive datasets.

The Evolution of Infrastructure

Cloudflare details its tenth-generation server, Gen X. The new hardware, powered by 2nd Gen AMD EPYC processors—marking the first time Intel is not inside—processes up to 36% more requests than its predecessor. This efficiency gain brings a roughly 50% decrease in L3 cache miss rate and up to a 50% decrease in NGINX p99 latency, all while running on a CPU with a 25% lower TDP per core.

A clever distributed systems pattern from @jeremy_daly pairs Lambda Destinations with EventBridge. The approach allows for event processing without writing any error-handling code, as failures can be routed by EventBridge rules and handled by SQS. His verdict after weeks of use: “extremely effective.”

Amidst all the new, a reminder from Marc Brooker about the older, harder problems. He shares that AWS built Physalia, a specialized database to improve EBS availability and scalability, using knowledge of data center topology to “avoid making the hard decision.”

Architecture and Teams

There is sound advice on Reddit for breaking down a system without the theatrics of a full microservices overhaul. The suggestion is to start with logical boundaries in a monolith, deploy it everywhere, and then assign specific instances to dedicated tasks before slowly carving out pieces. This path lets you define an API, test it instantly, and form a focused team around a single domain.

This week’s warning labels are also plentiful. @mweagle cautions that “By infusing microservices architecture gratuitously, you’re just going to turn your bad code into a bad infrastructure.” @MissAmyTobey wonders aloud about the evolution of the sidecar concept from “attachable small thing outside the hot path” to “full-on complex system in its own right.”

And @benbjohnson offers a reality check for strict uptime requirements. He points out that GitHub, a service nearly everyone relies on, operates at approximately 99.9% availability and sold for $7.5 billion. His recommendation: unless you’re making pacemakers, perhaps scale back those five-nines demands.

The Cost of Abstraction

The familiar “cloud vs. on-prem” debate continues. One startup founder highlights the insidious effect of free cloud credits, which can remove internal incentives for conservation, leading to outsized bills for traffic that is “95% CI.” Legogris takes a more pragmatic view, suggesting a startup begin on a public cloud for velocity, then consider a move when the long-term requirements become clear.

Hardware and the Cloud's

Backblaze’s 2019 drive stats show a notable overall rise in Annualized Failure Rates. The primary culprits: aging 8 TB drives experiencing a “mid-life crisis” and Seagate 12 TB models, the latter being actively addressed by a data migration project.

The regulatory and economic environment for tech appears to be shifting as well. The era of hyper-growth seems to be giving way to something more measured, with advice circulating about the importance of negotiation and boundaries in one’s career, and a reminder that venture-backed moonshots often have short lifespans. As one commenter notes, it is reasonable to kill a project when the question of profitability changes from “Likely, in a few years” to “Maybe, perhaps never.”

Scaling Data and Engineering

Data point of note comes from @alexbdebrie on DynamoDB performance: batch writes are only about 35% slower than single writes (sub-20ms), while transactions come in 3-4x slower than batch. Batch size remains a major factor in performance.

@PeterVosshall lists his proudest accomplishments in a thread touching on Amazon’s early days: the services-oriented architecture, real-time data propagation, and the work that birthed Dynamo, a paper that later received the ACM Hall of Fame award.

Finally, two contrasting wisdom nuggets on the practice of engineering. Paul Brebner offers his number one Kafka rule: “Kafka is easy to scale, when you use the smallest number of consumers possible.” And @danielbryantuk reminds us that in software architecture, there are no best practices—only tradeoffs.

Scaling Through Insight: This Week's Notes on Architecture and Operations

This week's roundup is heavy on architectural philosophy, platform pressure, and how systems behave when the real world shows up unannounced. From a prime-time television appearance to the design of high-assurance software, the recurring theme is that scale and resilience are products of deliberate, often early, choices about structure and dependencies.

Traffic That Goes 211x in Minutes

Troy Hunt recounted the experience of haveibeenpwned (HIBP) being featured on a prime-time ITV show for a second time. The first time, in 2016, the sudden influx of traffic caused roughly a third of requests to fail. For the recent appearance, the traffic increased 211 times in a very short period—peaking at 33,260 requests per minute (554 requests per second) searching 9.5 billion breach records and 133 million paste accounts—yet the failed request count was zero.

The difference wasn't luck but architecture. Over the prior 3.5 years, Hunt had moved all critical APIs to Azure Functions (serverless) and invested heavily in optimizing the Cloudflare edge, including caching and Workers. The cache hit ratio went from only about 76% in 2016 to a much more aggressive 96% in the latest peak. The roughly 163,000 uncached requests were almost entirely the dynamic traffic that absolutely must reach the origin: searches against the database, new subscriptions, and the domain search feature.

The SQL database, which handles breach subscriptions, peaked at about 40% utilization—running on a relatively modest S2 instance for about A$103 per month. The key insight, as Hunt noted, wasn't just that the system held up, but that he didn't need to change his usual approach to make it happen. The traffic patterns from such broadcasts are spiky and sudden, but having the critical path on serverless functions with a hot edge cache made the spike a non-event. The busiest 15 minutes saw 3.67 million requests, more than double the traffic of the 2016 experience.

Platform Fees and Hostile Relationships

Marketplaces have a lifecycle. Etsy sellers are facing a new, mandatory offsite ads program that has sparked a significant backlash. The system features no opt-out for sellers who have generated over $10,000 in rolling 12-month sales, forcing them into the program. One HN commenter, a former member of Etsy's ads team, framed the shift succinctly: the initial marketplace success comes from aligning buyers and sellers, but as the platform matures, the relationship turns adversarial. The original ad tools were optional and felt homegrown, but this new scheme is seen by sellers as effectively an extortionate tax on their success—a "release the platform, die from the platform" scenario where the seller bears the risk and the marketplace extracts the rent.

Architecture Lessons in Abstraction and People

A lecture by Jim Keller on microprocessors and first principles offers several relevant takeaways for scaling systems and teams. The core constraint isn't the transistors, but the human factor: people aren't getting smarter, and teams can't grow much beyond a few dozen effectively. The power of abstraction layers is very high—teams can turn transistors into logic cells, functional units, and finally, computers. However, simply using faster computers to build bigger computers doesn't work, particularly when algorithms are N^2. Keller noted that while costs trend toward zero once the production equipment is figured out, the real limitation is a "people are how constrained" mindset—incremental tweaks on known designs versus the much rarer discipline of deciding what you want to build and then figuring out how architecture can obey those constraints.

Sandboxing Libraries with WebAssembly

Mozilla is pursuing a novel approach to security in Firefox by compiling C/C++ libraries into WebAssembly, then compiling that wasm into native code at build time, ahead of Firefox's own compilation. This is a departure from browser-based wasm usage; the goal is to place each library in its own sandbox to mitigate the risk of vulnerabilities in third-party code. Called RLBox, the project represents a significant architectural investment in treating libraries as a major threat vector.

Performance is a Feature, Not an Afterthought

Nelson Elhage's reflections on software performance argue strongly that performance is persistently underrated. Tools that are genuinely fast don't merely complete tasks faster; they change how users approach their work. Users adapt their strategies based on the latency/cost tradeoff perceived; a fast tool invites more frequent, more ambitious use. This is in opposition to the "performance last" model, which fails for two main reasons: the high-level architecture and data-flow of a system have profound implications for whether speed is even possible later, and retrofitting performance onto a slow system tends to add complexity (caching, distributed systems, incremental bookkeeping) which adds new classes of bugs and makes straight-line performance even worse. Starting with a performant core can ultimately simplify the architecture relative to a given level of functionality.

Systems Thinking and Quality

Uwe Friedrichsen summarized a talk by Dr. Russell Ackoff on the nature of systems and quality. The key distinctions are worth holding onto. Quality is defined by meeting or exceeding customer expectations, but Ackoff's deeper point is that a system is a whole whose parts are interdependent. The essential properties of a system exist only in the whole; an automobile's ability to carry you places is not a property of any single part. Therefore, optimizing parts in isolation will never improve the performance of the whole; the performance depends on how the parts fit, not how they perform separately.

The most striking point is the criticism of ongoing improvement for its own sake. Continuous improvement imitates a leader, but overcoming a leader requires "discontinuous improvement"—creativity that leapfrogs the competition. The distinction between efficiency and effectiveness, between doing things right and doing the right thing, is central. Ackoff posits that improvement programs must be directed at what you want, not merely at what you don't want—getting rid of a deficiency does not mean you've achieved a virtue. Until managers account for the systemic nature of their organizations, improvement efforts are doomed.

The Practical Edge: Wi-Fi, Funding, and IoT

A handful of practical links round out the field of hard-won operational experience. Ars Technica published "Ten rules for placing your Wi-Fi access points," and many directly reference distributed systems thinking: too much transmit power is a bug, use spectrum wisely, and a Wi-Fi network is only as fast as its slowest connected device—a strategy of redundant placement isn't as good as reducing roaming latency.

A startup thread discussed using AWS Marketplace + Cloudformation to create a nearly free stack for running a small operation which ties in several key pieces: DynamoDB, Lambda, Netlify, S3, and Cloudflare can comprise a full platform for roughly $6/yr for a domain. Another post from Lightspeed Venture Partners pushes B2B startups to seek nonlinear growth—if you have a 10-year plan, asking why can't it be done in six months forces a better analysis than just accepting linear progression.

When the Physical and Digital Clash

The internet of things continues to collide with operational technology, and a pen-testing review of ships highlighted just how fractured maritime security is. The persistent belief in an "air gap" between IT and OT networks contradicts reality. The reports show that within the fifteen or so vessels reviewed, only one had a genuine air gap, and often the bypasses were deliberate, or the result of weak credential management. Insecure versions of TeamViewer had been installed without the owner's knowledge, and default credentials are still alarmingly common on public-facing systems. Combining more technologies—from booking systems and guest Wi-Fi to ballast management and bridge systems—vastly increases the attack surface, particularly on cruise ships. Ship operators consistently assume an air gap exists that, in practice, almost never does.

Tools for Incident Response and Metrics

Netflix open-sourced dispatch, a tool that automates the ad-hoc tasks involved in incident management. The project handles the routine work that teams often do manually during an outage, plus additional steps that are frequently neglected due to time pressure.

Uber released m3, a fully open source metrics platform built atop M3DB, a distributed time series database. Uber developed both after finding existing open source options unreliable, costly, and operationally heavy at its scale. The platform includes a native distributed time series database, a dynamic aggregation service, and a query engine.

Ray is a distributed execution framework designed for machine learning and reinforcement learning workloads. It achieves scalability by storing control state in a global control store while keeping other components stateless. A shared-memory distributed object store handles large data efficiently, and a hierarchical bottom-up scheduler provides low-latency, high-throughput scheduling. The API builds on dynamic task graphs and actors.

Also worth a look: linuxkit, a toolkit for building lean container operating systems, and glb-demo, a Terraform-based demonstration of a multi-cluster global load balancer on Google Cloud Platform with Cloud Armor policies.

Research and Reading

Researchers at Purdue present AppStreamer, a system that reduces mobile game storage requirements through predictive streaming. In tests with Dead Effect 2 (a 3D first-person shooter) and Fire Emblem Heroes (a 2D turn-based strategy game), 75% and 87% of users respectively reported the same quality of experience as when all game files were stored locally. The system cut storage needs by 87% for the first game and 86% for the second.

The AWS Builder's Library addresses the inherent difficulty of building distributed utility services. Independent failures and nondeterminism are the two problems with the most impact. Beyond typical computing failures, distributed systems exhibit many additional failure modes, and it is not always possible to know whether a component has actually failed.

A University of California, Santa Cruz team is using cerebral organoids derived from chimpanzee pluripotent stem cells to study human-specific brain evolution. Direct comparisons with non-human primates are difficult because chimpanzee tissue is not accessible during the neurogenesis period when key differences in brain size first emerge.

A thesis on interactive curve design proves that any spline sharing certain properties has all segments between control points cut from a single fixed generating curve. That reduces spline design to choosing the ideal generating curve; the Euler spiral performs well across the board, while log-aesthetic curves may suit some applications better.

Cloud Hypervisor is an open source Virtual Machine Monitor running on KVM. It targets modern cloud workloads: Linux distributions with paravirtualized I/O via virtio, no legacy device support, and recent CPUs.

A study of Spotify's guild structure finds that engagement declines as organizations scale. Only 20% of members regularly participate, while the majority simply follow updates. Size and geographical distribution create scheduling conflicts, prompting regional sub-guilds. The authors recommend that large distributed environments offer both regional and cross-site activities.

Amazon describes Physalia, a transactional key-value store for large-scale control planes. Instead of trying to be highly available for all keys to all clients, Physalia exploits knowledge of datacenter topology to place data where it is most likely available, focusing on the specific keys each client needs. Delivered in the context of Amazon EBS and other AWS uses, the approach is presented as applicable to control planes, configuration management, and service discovery more broadly.