Metrics from the Week in Scale

Every week, the industry publishes numbers that define where distributed systems, data infrastructure, and software engineering are heading. Here are the most interesting figures from the past seven days.

A quick framing for the data points below: roughly 84% of Cloud Native Computing Foundation survey respondents now run containers in production, up 15% from the prior year. Kubernetes adoption sits at 78%, and over 10% of those users manage more than 50 production clusters.

Streaming, Search, and Traffic at Scale

  • 14,500: containers running in Riot-operated regions alone, per the game company's online services engineering team.
  • 30–60 billion: records NASDAQ loads nightly into its systems.
  • 14 million: Uber trips per day, a number that drives its machine learning model lifecycle and version control work.
  • ~2 million: daily searches handled by QuickBooks' Elasticsearch platform, supporting ~50 million business transactions. The stack is 10 stateless application servers fronted by a load balancer and gateway.
  • 93 out of 100: most-watched shows in 2019 were live sporting events, up from roughly 20–25 two decades ago.
  • 20%: higher click-through rate for personalized search results. When evidence that a search was personal was strong and results were repositioned to the top, CTR improved by 74%.

Code, Cost, and Containers

  • 1.7 million → 360,000: lines of code in Facebook Messenger before and after its rewrite, respectively.
  • $10,000+: average annual per-employee software spend at enterprise companies.
  • $3.92 million: average cost of a cloud data breach for a company.
  • Under 1%: blockchain adoption rate, suggesting the technology remains niche despite infrastructure investment.
  • 170,000 years: time for a photon to travel from the sun's core to Earth—useful context when thinking about latency at astronomy scale.
  • 2 exaflops: target performance for the El Capitan supercomputer, which will pair AMD CPUs and GPUs.
  • 1: number of Starships Elon Musk aims to produce per week.

Interesting Correlations and Infrastructure Notes

Two data points stand out as signals rather than facts. First, only one-third of well-designed features at Microsoft actually deliver value, a reminder that feature counts say little about outcome quality. Second, Google has seen a 500% increase in requests from state and federal law enforcement, raising questions about data retention and access patterns that infrastructure teams must increasingly consider in their designs.

On the compute side, the top 10,000 most spoken English words can be represented as points in hundreds of dimensions, where distance and direction between points encode word relationships—the conceptual basis for modern natural language processing embeddings. Meanwhile, a single night of Netflix viewing (30 minutes) generates under 20 grams of CO2, roughly equivalent to driving a car 160 meters.

In 1997, the entire web had roughly 2,700 websites and Lycos indexed just 54,000 documents. That historical scale helps contextualize today's operational reality: a $252 million donation funds UC Berkeley's Data Hub, and 20% of enterprises now report higher costs for software subscriptions that are increasingly difficult to manage.

Kubernetes: Know When to Hold ’Em

The debate over Kubernetes continues to rage, with the loudest voices questioning its value for smaller teams. One Hacker News commenter wonders why anyone would choose a full cluster when a large instance with 24 cores could serve fewer than 10,000 users: “Kubernetes feels like an insane complexity that doesn't need to be taken on and managed. You're gonna spend more time managing Kubernetes than writing actual software.” The sentiment is echoed more bluntly elsewhere: “If you can’t afford 6+ full-time people to babysit k8s, you shouldn’t be using it.”

A common analogy making the rounds compares Kubernetes to a bakery’s cleaning cloths. The service is crucial, but it doesn't differentiate the business; as long as the clean cloths arrive, the bakery can keep making bread. The lesson: outsource the non-value-added work, or skip the orchestration layer entirely. “Maybe you should outsource one level higher, and not even worry about Kubernetes at all,” one commenter suggests, pointing to platforms like Heroku or Amazon Elastic Beanstalk, which “handle the scaling and a lot of other concerns for you with a much simpler model.” The trade-off is provider lock-in, but that’s often preferable to the complexity tax of self-managed infrastructure.

For those who do make the jump, the transition can be jarring. One engineer describes an on-premises Kubernetes rollout as “challenging but rewarding,” noting that his company hit a “sweet spot” between size and tooling. And the cloud providers are clearly courting this crowd: Azure and DigitalOcean don’t charge for the compute resources used by the managed control plane, making them the cheapest options for running many smaller clusters. For fewer, larger clusters, Google Kubernetes Engine is the most affordable, with spot/preemptible and long-term committed nodes providing massive savings across all platforms.

Yet some see the market moving in the opposite direction. One developer complains that Google Cloud’s new “Cluster Management Fee” puts the smallest possible Kubernetes cluster at around $300 per month: “It seems Google Cloud doesn't want individual developers on their platform. Huge mistake.”

Serverless: Small Functions, Big Claims

The serverless pitch has a compelling new data point. A developer reports that an AWS application handled more than 200,000 requests in a few minutes without any prior optimization, responding in milliseconds: “Just something to think about when anyone talks about whether serverless apps are fast, scalable, or easy to create.”

The architectural reasoning behind such results is often distilled into a few key realizations: sooner or later, you’ll have to scale horizontally; it’s easier, cheaper, and more reliable to scale small functions (“cattle”) than big applications (“pets”); and the CAP theorem still applies, meaning a low-latency, constant-time store like DynamoDB is essential. The mental model differs from traditional scaling, as one developer notes: “An asynchronous callback is the programming equivalent of when a toaster bell goes off. It means the toast is ready for your attention.”

The Economics and Physics of Scale

Scaling down to the individual level can be equally tricky. A commenter points out that Turbo Boost on an 8-core laptop lifts a single core to about 4.5 GHz, but with all cores loaded, that falls to 3.5 GHz. The theoretical 8x speedup becomes just 6.2x in real time due to the lower sustained clock rate. More fundamentally, another observer reframes the parallel computing equation: “Parallelism is specifically the stuff that actually does happen completely independently... Concurrency refers to the overhead of coordinating activity... Parallelism giveth, and concurrency taketh away.”

Thread creation overhead is similarly confounding. In C under Linux, creating a thread can cost thousands of CPU cycles. If the function you’re trying to parallelize needs only hundreds of cycles, “it is almost surely wasteful to create a thread to execute it,” notes one researcher.

For those considering the costs of large-scale systems, a former Dropbox engineer offers a dose of reality on exabyte-scale storage estimates from others: “these numbers are all extremely optimistic... Networking is much more expensive, labor is much more expensive, space is much more expensive.” He expects the final cost projections to be off by a factor of 2-5x, largely because “hard drives are a surprisingly low percentage of the cost of a storage system.” And when you do get to that scale, dense storage presents its own problems. “There is a scaling point where virtually every single insert requires at least one page fault,” notes another engineer. With a B+Tree or similar structure, the index itself may no longer fit in cache at high densities.

Storage inefficiency, in turn, spawns complexity elsewhere: “If your storage system didn’t suck so much, you might not need so many layers of caches.”

On the chip-design front, costs are climbing exponentially as process nodes shrink. An advanced 16nm design averages $106.3 million; moving to 10nm pushes that to $174.4 million; 7nm costs $297.8 million; and
5nm projections are on the order of $542.2 million. Nearly half of that is software. Chiplets could help contain the other half of the bill by combining smaller compute units in a single socket, improving manufacturing yields — a defect takes out a small chiplet rather than a complete monolithic die — and allowing mixed nodes, such as the latest compute cores alongside older memory and bus interfaces.

Telemetry and Remote Work Realities

Even with perfect infrastructure, measuring your system’s performance often requires a shift in sampling strategy. High-resolution telemetry sources can be expensive to aggregate and store. In-process summarization offers a path forward: rather than exporting 60 seconds of raw time-series data, you can export a few percentiles and cut your storage costs dramatically. The savings grow when you need to sample something every 100 milliseconds or even faster to catch the small problems that actually matter.

Meanwhile, the forced experiment of working from home may have mixed results. One engineer reports that when his company was forced to go all-remote, productivity “fell immensely” in certain teams. “The problem is that you can’t suddenly transition people to WFH and expect them to figure it out right away,” he explains. The parents of young children struggled with noise and distraction; some employees couldn’t stay productive without direct management; others tried to be “always on” from vacations. He worries the sudden shift “will hurt perceptions of remote work more than it helps.”

Around the Industry

The industry continues to spawn cautionary tales and hopeful signals in equal measure:

  • A survey from the Cloud Native Computing Foundation finds 84% of respondents are using containers in production, with 78% running Kubernetes in production; Amazon EKS leads the pack for managed platforms.
  • CloudFront reports that recent changes cut configuration deployment time to about 5 minutes consistently, with most edge locations updating in seconds even as the company targets p100 metrics.
  • Microsoft’s takedown of the Necurs botnet revealed a single infected computer sending more than 3.8 million emails over 58 days.
  • Zoom, Shopify and Stripe collectively boast a $123bn market value, undermining the old wisdom that tech giants will simply expand to crush all competition.
  • A natural gas power plant in upstate New York is using 14 megawatts to mine bitcoin — enough power for 11,000 homes.
  • Even in history, there’s a note on remote work: the University of Cambridge closed in 1665 for the plague, and Isaac Newton used the time to develop calculus and the theory of gravity.

Engineering Lessons From Rockets, Rewrites, and Runtimes

Rocket fabrication offers a surprisingly relevant lens for software design. The quality of your tools dictates the kinds of designs you can produce. For instance, a Delta booster panel uses a triangle-based isogrid structure, while the newer Vulcan rocket can use non-symmetric square orthogrids because modern finite element analysis tools are better. This new design is both stronger and takes half the time to manufacture. Another key difference is the factor of safety: rockets only have a 10% margin because they can't afford the weight, whereas buildings use 2.0, pressure vessels use 3.5 to 4, and bridges use 5 to 7. Every 7 pounds in the booster plate costs a pound in the spacecraft.

The manufacturing process combines heavy machinery with human craftsmanship, reinforcing that we are heading toward human-machine cooperation rather than full replacement. A lighter, stronger form of welding called friction stir welding provides a good analogy for software—we often tack things together with filler and melt them, resulting in a joint with different properties than the original material. For different purposes, use different technologies. On a booster rocket, you can afford 7kg of inert mass (aluminum) before it costs you 1kg on the upper stages where the ratio is 1-1, so they switch to higher-performance stainless steel. A key test strategy is to "fly it before you fly it"—try new components on old systems like Atlas to verify they work before the first Vulcan flight.

The guidance system software can be upgraded up to 60 seconds before launch to handle changing wind conditions. They measure the wind, run a simulation, recertify the trajectory, and load the new parameters—not new code—onto the rocket, with certification handled by a hardware-in-the-loop simulation lab. Once flying, the rocket can adjust its own parameters, such as extending the mission if it uses less propellant than expected.

Microservices: Reaping Benefits and Managing Complexity

Riot Games adopted microservices and were rarely blocked on anything they couldn't resolve themselves. But worrying trends emerged. The QA and load-testing environments became less stable, and a risky feedback loop developed: teams optimized logs, metrics, and processes for themselves, rarely considering that others might need to reason about their services. As the number of microservices grew, operating the holistic product became difficult, leading to an increasing number of failures. A fluid team structure left some services with unclear ownership, making triage difficult. The lesson: allowing unaligned streams of changes into a distributed system will eventually lead to preventable incidents.

Their solution was to eliminate partial state manipulation by creating an opinionated declarative specification capturing the entirety of a distributed product—an "environment." It contains all the declarative metadata required to fully specify, deploy, configure, run, and operate a set of microservices collectively representing a product, and it is holistically and immutably versioned. This led to consistent, deterministic deployments. The future involves describing expected latencies between services so tooling can optimize for underlying regions and PaaS services—some servers might need to be co-located in the same rack, while others can be distributed. This matters for performance-sensitive game servers.

Segment took the opposite path. Years of adding to a microservice architecture led to declining developer velocity as they tripped over its complexity. Their session "To Microservices and Back Again" advises learning which anti-patterns to avoid, recognizing the trade-offs, and identifying when moving back to a monolith is the solution.

Optimization Advice Worth Its Salt

Cloudflare chose the single-socket 48-core AMD 2nd Gen EPYC 7642 for its Gen X servers, after discovering the large L3 cache led to a low cache miss rate and high sustained frequency. When tuned with AMD via power determinism, these servers process up to 36% more requests while costing less, with a ~50% decrease in L3 cache miss rate and up to a 50% decrease in NGINX p99 latency.

The company warns against advanced data structures on modern CPUs. CPUs are superb at sequential memory access when fetch patterns are predictable; random access is disastrously costly. Bloom filters are great until they no longer fit in L3, at which point performance collapses. One engineer notes, "You can assume modern CPUs are infinitely fast. They run at infinite speed until they hit the memory wall." For profiling, start with perf stat -d and look at "Instructions per cycle" (IPC): below 1 usually means the program is stuck on memory, above 2 means it's CPU-bound.

The essential tuning advice from a CPU microarchitecture overview includes using cacheline-aligned data structures, compressing data in memory, avoiding random access, configuring huge pages with madvise, partitioning memory with libnuma, and analyzing perf counters with top-down architecture analysis.

Rethinking Scale and Architecture

Facebook's Project LightSpeed rewrote the Messenger codebase from more than 1.7 million lines to 360,000—an 84 percent reduction—based on four principles: use the OS, reuse the UI, leverage the SQLite database, and push to the server. This resulted in a 2x faster startup time and a quarter the size (from a peak of 130MB). Each user-interface screen is a single dynamic template; SQLite holds the instructions for how to display different building blocks, rather than having dozens of pre-built designs. SQLite acts as a universal system, extended with stored procedures and orchestrated via the MSYS platform for queued changes, deferred tasks, and data sync. Features that don't fit send everything to a new server that acts as a universal broker. They set per-feature budgets and built tooling to understand the binary weight each feature brings, holding engineers accountable as part of acceptance criteria.

Those chasing raw performance improvements should be skeptical of cross-platform migrations. A 3x improvement after moving to another system turned out to be a mirage: the lower load on the new system meant fewer context switches, enabling the application to use more cache. Netflix and its infrastructure teams remain vigilant precisely because such illusions are common.

To reach 23,000 Kafka events per second, one approach relies on 11 Kubernetes nodes and 280 pods on Azure. The architecture's cleverest component is the application of the SAGA pattern to implement transactions across microservices. For cheaper options, highly-requested static content can be pre-rendered and cached: 350k requests can cost just $3 with Lambda, but one practitioner points out that caching pre-rendered HTML on S3 and CloudFront makes your cost curve nearly vanish, serving hundreds of millions of requests for roughly $10/month. Another runs over 2 million Lambda invocations for a static-site generator for 26 cents per month.

The old build-versus-buy decision has been inverted by platforms. Software was once seen as a cost center and a point of parity; now in-house software is a way to differentiate and create value. Building custom software, leveraging a rich ecosystem, is a way to out-compete businesses that outsourced their core differentiation. Thinking of platforms, frameworks, and services as a digital supply chain makes this clear: you can manufacture something unique—software—for much less than the cost of competing on the old model.

On Performance, Openness, and the Environment

Netflix, LISA sessions, and practitioners all emphasize profiling as distinct from measuring. The delta in performance claims should invite skepticism. When Cloudflare saw a performance jump, investigation revealed the effects were real but resulted from cache behavior, not inherent superiority—reinforcing that output metrics rarely move by magic.

Amir Michael coined the term infrastructure privilege to explain that commercial companies develop faster, more efficient systems others don't have access to, achieving only local efficiencies rather than global progress. This informed his work on the Open Compute Project. An emerging company, Oxide, argues that with a from-scratch rethinking of the server (a real hardware root-of-trust, a purpose-built BMC, a fully open software stack, a rack-scale design, and co-designed network and compute), a superior alternative to hyperscaler architectures is achievable, though history warns against expecting a 20% better system to survive against commodity components.

A study correcting earlier claims about Netflix's carbon footprint found the true emissions from 30 minutes of streaming are less than 20 grams of CO2—equivalent to driving 160 meters, not 4 miles. The prior analysis was off by a factor of 40x on video download electricity. Similarly, global data center computing output jumped sixfold from 2010 to 2018 while energy consumption rose only 6 percent. Moving bits, not atoms, continues to be the environmentally preferable path.

FOX Sports sees the entire video industry reorganizing into live and on-demand marketplaces. Live content (sports, cable news) supports commercial breaks users can't avoid; on-demand platforms avoid ads altogether. This shift explains why 93 of the top 100 most watched shows in 2019 were live sports events, compared to 20-25 twenty years ago. Marginal content like college basketball is lifted by this rising tide as scripted content leaves the network schedule.

Research on DevOps and system performance shows high performers do better at everything: speed and stability go together, and companies don't need to make a tradeoff between moving fast and breaking things. One of the biggest predictors of IT performance is the ability of teams to get things done without depending on other teams. Failure is inevitable; what matters is recovering quickly with a small blast radius. Maturity models are inferior to capability models: capabilities drive outcomes and provide an ongoing path of improvement rather than a static end state. Rather than hunting for fault, constructive organizations encourage psychological safety, so risks are taken and failures are treated as learning.

Picks From the Repository

Firecracker has carved out a niche as an open-source virtualization technology built specifically for secure, multi-tenant container and function-based services. It runs workloads inside lightweight VMs — microVMs — that borrow the isolation guarantees of hardware virtualization and pair them with the startup speed and flexibility typically associated with containers.

For those running container workloads at the OS level, Bottlerocket offers a free, open-source Linux distribution designed exclusively for hosting containers. It is currently in developer preview; the quickstart guide walks through deploying it on an Amazon EKS cluster.

On the communication front, Matrix is pitched as a new foundation for open, interoperable, decentralized real-time messaging. Elsewhere in open-source land, EarthSpecies maintains a collaborative, nonprofit project dedicated to decoding animal communication.

Papers and Reading

A paper from arXiv makes the case that smart algorithms can beat hardware acceleration for large-scale deep learning. The proposed system, SLIDE (Sub-LInear Deep learning Engine), uses randomized algorithms combined with multi-core parallelism and workload optimization. Running on CPU only, SLIDE reduces computation in both training and inference, and the authors report it outperforms an optimized TensorFlow implementation on a Tesla V100 GPU. On industry-scale recommendation datasets with large fully connected architectures, training SLIDE on a 44-core CPU took roughly one hour versus 3.5 hours for TensorFlow on the V100 at the same accuracy; on the same CPU hardware, SLIDE was over 10x faster than TensorFlow.

From the NIH Director's Blog comes work on the brain's nightly "rinse cycle." Published in Science, the findings are the first to suggest that the brain's sleep-related fluctuations in blood and electrical activity may also drive cleansing waves of blood and cerebrospinal fluid. The experiments involved healthy adults, and the authors note that further investigation could clarify the link between poor sleep and the spread of toxic proteins and memory decline in Alzheimer's disease.

For those learning deep learning, the fastai project maintains draft notebooks covering an introduction to deep learning, fastai, and PyTorch, with fastai positioned as a layered API.

A classic post from the Morning Paper, "Out of the Tar Pit," argues that despite the accidental complexity that inevitably exists, it is possible to preserve much of the simplicity of the ideal world in real systems. The recommended approach to managing complexity — whether from state or control — is summed up in two words: avoid and separate.

Finally, a CMU technical report tackles the widening gap between DRAM costs and database sizes. As DBMSs operate with lower memory-to-storage ratios, they still depend on in-memory search trees such as indexes and filters for throughput and latency. These structures consume a disproportionate share of available memory. The dissertation presents succinct data structures that reduce memory usage to the theoretical limit, after identifying sources of waste in existing read-optimized trees.