Numbers From the Week in Infrastructure

Amazon’s Prime Day 2020 offers a fresh set of scale figures from AWS. Across the 66-hour shopping event, DynamoDB handled 16.4 trillion calls from Alexa, the main Amazon.com site, and all fulfillment centers, peaking at 80.1 million requests per second. EBS processed 6.2 trillion requests per day and moved 563 petabytes daily. CloudFront, meanwhile, sustained over 280 million HTTP requests per minute, totaling 450 billion requests across all Amazon properties over the event.

A few other figures stood out this week:

  • 20x: Microsoft's speedup for Apache Spark workloads, achieved by rewriting the engine in C++ to exploit modern hardware and instruction-level parallelism.
  • $1,000 per day: The cost for Nielsen to process 250 billion events per day (55 TB of data) using AWS Lambda. The serverless architecture scales automatically, handling between 1 TB and 6 TB per hour.
  • 247 zeptoseconds: How long it takes a photon to cross a hydrogen molecule.
  • 18 million: Daily requests to Uber's platform from more than 10,000 cities.
  • 1.1 million gigabytes per second: The rate at which enterprises will store data by 2024, according to projections highlighted in a study on data gravity.
  • $44.75 million: Total bug bounties paid out through HackerOne in the last year, an 86% year-over-year increase. Nine individual researchers have now earned over $1 million.
  • 11: Critical vulnerabilities found in Apple's corporate network during a white-hat engagement; researchers identified 55 total vulnerabilities and received a $288,000 reward.
  • 18: Crashes logged by Waymo in its Phoenix, Arizona autonomous vehicle operations, with 29 near-miss collisions.
  • $10.60: Per-person daily food cost for three meals aboard a US submarine.

Cloud Economics

AWS revenue rose 29% in the third quarter. By analyst estimates, Microsoft's cloud business is now about 54% the size of AWS. Amazon's own numbers show AWS contributing roughly 57% of operating income while accounting for 12% of total revenue. The parent company has grown its headcount to 1,125,300 employees, a 50% increase over the past year.

Performance and Optimization

Cloudflare's Automatic Platform Optimization produced a 300% improvement for a WordPress site. On the database side, Netflix detailed its "Bulldozer" approach for moving batch data from a warehouse to online key-value stores, a challenge now relevant to its 195+ million subscribers.

For passwords, the minimum recommended length is now 12 characters, according to Carnegie Mellon's CyLab. And despite the loneliness of remote work, 45% of professionals want to work remotely full-time.

Quotable Bits From the Week in Scalability

The internet is rarely short of hot takes. Here's what caught our eye this week on the state of distributed systems, software architecture, and the broader tech landscape.

The Architecture Wars Continue

The debate over microservices versus monoliths shows no signs of cooling down. A recurring theme is that the choice isn't about a specific technology but about the context of the problem.

  • @Carnage4Life asked a pointed question: "The longer I’m in tech, the more I’m convinced microservices were a technical solution to an organizational problem."
  • vlfig argued that the concept was right but was applied too bluntly. "Microservices" was a good idea taken too far and applied too bluntly. The fix isn’t just to dial back the granularity knob but instead to 1) focus on the split-join criteria as opposed to size; and 2) differentiate between the project model and the deployment model when applying them."
  • rufius shared a disciplined approach: "As a big monolith got too unwieldy, we’d refactor major pieces out into another service... The idea wasn’t to proliferate a million services to monitor but rather to use it as an opportunity to stand up a new Citadel service that encapsulated some related functionality. It’s worked well but it requires planning and discipline."
  • 8fingerlouie estimated that replacing their monoliths with pure microservices would have required about 40,000 of them. Instead they chose a "macroservices" model, which "focus on business level 'units of work'."
  • On the flip side, candiddevmike cautioned that microservices are often just an unhealthy vehicle for rewrites: "You can't one-at-a-time this ... Fast forward to six months or a year later, the monolith is still around, features are piling up, 20 microservices have been released, and no one has a flipping clue what does what."

There was also a more general programming philosophy debate about how to structure code itself. ajuc posited two distinct camps: making code easy to extend (SOLID) versus making it easy to rewrite (KISS). The author noted that OO developers think the first approach is the only way, but with high-level declarative languages, both can be correct.

The Real Costs and Benefits of Modern Stacks

A few data points on the financial and operational realities of running modern architectures emerged.

  • @QuinnyPig satirized AWS’s earnings talking points, which claimed customers are happy, performing well, and cutting costs by moving to the cloud. The quip: AWS lies about "moving to the cloud is a great way to do that long term" and is really helping with short-term tuning.
  • @theburningmonk touted a serverless win: "t can't stress how liberating it is to NOT have to worry about capacity planning, tuning auto-scaling configurations... everything scales itself, and only pay for what we use." Their client app built on AppSync launched with only a few weeks of work, 5,000 users, and a 99% cache hit rate.
  • Jon Holman reported a dramatic cost reduction in switching from AWS ECS Fargate to Lambda, dropping yearly cost from $1,730 to about $4.
  • @benbjohnson offered a stern warning about Kubernetes: "My rule of thumb is that you’ll probably spend at least $1M to move to Kubernetes. That’s new ops hires, training, initial productivity loss... If you don’t have a $1M problem that Kubernetes is solving then I don’t think you should even consider it."
  • Navigating identity gets pricier at scale. @jeremy_daly noted that for a consumer site with over 500,000 users, "For SaaS, I think Cognito/Auth0 are no brainers, but once you get out of the free tier, it gets pricey very quickly."

Hardware, Storage, and Performance Notes

  • Backblaze reported a Q3 2020 annualized failure rate of 0.89%, up from 0.81% last quarter but well down from the 2.07% reported a year ago. They added nearly 11,000 drives totaling over 150PB of storage in the quarter.
  • Evan Jones shared benchmark findings on filesystem durability. He found that "Overwriting is faster than appending (~2-100% faster)" because appending involves extra metadata updates. His recommendation is to "call fallocate() to pre-allocate the space you need, then explicitly zero-fill it and fsync."
  • @BrandonRich offered a philosophical shift in engineering outlook: "I used to build systems out of code. Now I'm building solutions out of systems -- and data, and relationships."
  • The shift to Arm gained a corroborating data point from Twirrim, who heard from former AWS engineers that "there's a big push on all services to evaluate and ideally migrate to ARM by default for their control plane services," citing significant internal savings.
  • In the chip world, Ed Sperling noted the industry is starting to rethink chip packaging. Moore's Law continues for digital logic at the 2/1nm nodes, but the surrounding functions will be moved off onto chiplets.

AI, Human Potential, and the Network Effect

  • Commenting on the machine-human divide, Ransbotham noted, "We’re seeing that this blending of humans and machines is where companies are performing well... The idea that either humans or machines are going to be superior, that's the same sort of fallacious thinking."
  • @NoSQLKnowHow pointed out the true scale of a successful on-demand system, noting that not only does DynamoDB handle 80.1 million requests per second for Prime Day, but it does so while "every other DynamoDB customer is doing their thing too and not having performance problems."
  • Vinton Cerf offered thoughts on the internet's role, saying, "The internet doesn’t invent or discover anything. It’s simply the medium through which people can do collaborative work and can discover new things." He also spoke about the challenges of introducing new protocols, like those for an interplanetary backbone, noting that resistance often comes because "new" implies risk, and you can’t prove it works unless you’re willing to take that risk.

The Business and Regulation Terrain

  • Ben Evans argued that the tech industry should treat TikTok as "one-off" only if the US government fails to realize it's "the new normal - there will be hundreds more of these. You can’t one-at-a-time this - you need a systematic, repeatable approach."
  • For those watching the Epic Games v. Apple case, @keachhagey flagged a wild DoJ stat: "Google's payments to Apple to be default search amount to 15-20% of Apple's global profits."
  • DSHR argued against imposing software liability, saying it would be "both ineffective and destructive" as a way to fix the supply chain. It'd be a "massive disincentive to open source contributors."
  • Broader malware-as-a-business continues to produce wild numbers. A report highlighted that Bitcoin and Ethereum now consume as much electricity as the entire country of Austria, and that a bitcoin transaction takes a half-million times more energy than a Visa payment.

Finally, amid the punditry, one timeless metaphor emerged: Gandalf warns about the perilous nature of any art we do not fully understand—an apt caution for any software team operating a complex distributed system today.

Envoy’s Open Source Lessons: Burnout, Community, and Naivety

Matt Klein, the creator of Envoy, has shared candid insights from his experience open sourcing the project while at Lyft. In retrospect, he says they were naive about the effort involved. Envoy was already feature-rich and rock solid when it was open sourced, having been built by an end user (Lyft) for an explicit use case: helping the company move to microservices. Initially an edge proxy and API gateway to replace load balancers, it offered better observability, access logs, and traffic understanding. Over time, rate limiting and MongoDB protocol parsing were added, marking the beginning of Envoy as a service mesh. Its key technical differentiators from older proxies like NGINX were its design for elastic, cloud-native loads—supporting autoscaling and containers—and its eventually consistent configuration system with a set of APIs for dynamic updates without reloads. It was also built to be extensible from the ground up, with plugin filters, metrics, stats, and tracing.

Klein came close to total burnout while working what amounted to two jobs: his responsibilities at Lyft and maintaining Envoy. He highlights a hard truth: most open source projects are a net negative in terms of effort versus community contribution, and Lyft is now far less likely to open source other projects because of the huge time commitment. Creating a successful open source company requires the same effort as starting any company—engineering, marketing, documentation, conferences, and hiring maintainers. Google's adoption helped significantly, not just through code contributions but by creating hype via conference talks. Klein also resisted VC advice that Envoy needed a company behind it to succeed, believing its independence was a strength. He recommends having honest, open conversations with employers about the effort required before open sourcing, framing it as an honest business discussion.

Robinhood’s Kubernetes Regret and the Rediscovery of Cells

Robinhood’s two-year Kubernetes migration is still incomplete, and their biggest mistake was framing it as a desire for Kubernetes in the first place. What they actually wanted were faster releases and immutable infrastructure, so patching a build is faster than making changes to a running system. Technology-framed goals are a trap, they learned; teams will interpret them differently and end up serving themselves rather than the business. Prepackaged solutions don’t work, and you need an abstraction customized for your specific business.

In the process, they are rediscovering cell architectures. Rather than scaling services independently, functionality is organized around accounts. An account includes all the services for that account, which don’t need to talk to shards elsewhere. This approach is horizontally scalable and limits the number of inter-service connections and hosts that must communicate. Under hyper-growth, it’s the second-order effects that hurt—adding machines means they all hit the service location system or all need to talk to Kafka, and it’s those cascading connections that cause problems. Cell architectures increase fault isolation, reduce scaling pain points, and, importantly, create a direction for the team that structures every conversation between teams.

Finding Bugs in Databases: The Power of Non-Optimization

TiDB’s top bug hunters, who have found over 400 bugs across popular DBMSs, have published the techniques behind their success. The first approach, Non-optimizing Reference Engine Construction (NoREC), has uncovered more than 150 bugs by rewriting a query so the DBMS cannot optimize it, revealing optimization flaws. A more powerful but elaborate method is Pivoted Query Synthesis (PQS), which also netted ~100 bugs by generating queries that ensure a randomly selected “pivot” row is fetched. The work-in-progress technique that produced the TiDB bugs is Ternary Logic Query Partitioning (TLP), which partitions a query into several others and then composes them.

Twitter’s Cache Analysis: Why Stability Beats Low Miss Ratios

An analysis of hundreds of in-memory cache clusters at Twitter, based on ~700 billion requests and 80 TB of raw log data from 306 instances of 153 Twemcache clusters, found that a cache with low miss ratios most of the time but occasional high misses is less useful than one with a slightly higher but stable ratio. Most caches seem to follow this pattern. Spikes in request rate also coincide with spikes in object access, suggesting triggers like client retries, traffic surges, or scan-like tasks rather than hot keys.

  • More than 35% of clusters are write-heavy, with over 20% having a write ratio above 50%.
  • Most caches use short TTLs, loosely bounding the effective working set size of unexpired objects, while the total working set of all active objects can be unbounded.
  • While most workloads follow a Zipfian distribution, some deviate, with the least and most popular objects differing from expectations.
  • Mean value sizes range from 10 bytes to 10 KB, with a median around 230 bytes.
  • As real-time stream processing grows, more caches will be used for computation results, which may not benefit equally from read-path optimizations like optimistic cuckoo hashing.
  • For proactive expiration, memcached uses a background crawler, which is effective only when the TTL range is narrow and is not efficient.
  • FIFO can perform as well as LRU for many in-memory workloads.

Data Points on Graviton2

AWS’s Graviton2 instances are gaining traction, with claims of significant performance and price-performance gains for a range of workloads. Community reports paint a consistent picture:

  • Amazon reports up to 35% performance and 52% price-performance improvement for RDS open source databases.
  • Early testers describe the purchase as the “lowest hanging fruit” for cost efficiency: the larger L1 cache has an incredible effect on many workloads.
  • An egress firewall company testing its product found an excellent price-to-performance ratio given low instance pricing and high network throughput.
  • A company using a database that targets ARM sees Graviton as a godsend for CI, since on-demand instances support all targeted architectures.
  • A product company has gone “all in” for its latest release.
  • One perspective notes that many software concurrency bugs have been covered up by x86’s friendlier memory model, so we can expect products to be slow to support weaker ARM models.

Netflix’s Approach to Load Shedding

Netflix’s strategy for handling too much load is prioritization. They consistently rank requests across mobile, browser, and TV, and progressively throttle traffic. All assumptions are validated with Chaos Testing, deliberately injecting faults for requests of specific priority levels.

  • Each request is characterized by throughput, functionality, and criticality.
  • Requests are bucketed as NON_CRITICAL, DEGRADED_EXPERIENCE, and CRITICAL.
  • The API gateway, Zuul, computes a priority score between 1 and 100 for each request.
  • This method allowed Netflix to shed load in a scenario that previously prevented video playback while keeping playback unaffected.

Architecture Note: Inverted Control Between Data and Control Planes

Avoiding overload in distributed systems sometimes means having the smaller service control the larger one, Amazon’s builders library explains. The directive is simple: among AWS services that are split into a large data plane and a smaller control plane, having the smaller fleet call into the larger one avoids a common scale mismatch.

When the larger data plane calls the control plane directly, the data servers initiate API calls periodically, which is simpler but requires careful tuning when the fleet outnumbers its control plane by over 100x. A tiny environmental change can cause correlated requests from clients, overwhelming the control plane. To survive, AWS recommends an inverted architecture: the up-stream control plane periodically writes configuration to Amazon S3. The data plane then polls S3 and caches locally. Similarly, the control plane polls a bucket where data plane servers post their operational status. This pattern is used by internal systems like AWS Hyperplane, the network function virtualization service backing Network Load Balancer, NAT Gateway, and AWS PrivateLink.

Moving BBC Online to the Cloud

BBC Online’s multi-year move to a cloud-native architecture is nearly complete. The effort’s overall approach and principles were instructive: don’t solve problems others have already solved, remove duplication without over-simplifying, break technology silos through culture and communication, build first and optimize later, and move fast with smaller releases to minimize risk.

The high-level system layering is: a traffic management layer capable of tens of thousands of requests per second; a website rendering layer on AWS with React that runs about 2,000 lambdas every second; a business logic layer that allows teams to create their own logic without the scaling problems; and platform and workflow layers. Their choice for serverless was based on cost: it removes operational overhead and scales more quickly than EC2 auto-scaling for breaking news traffic spikes. Teams achieved 110 releases across 6,249 code builds, with an average build time of 3.5 minutes, and an average of 1 day and 23 minutes from pull request to live production.

Rust for Telemetry: Fewer, Smaller Servers at Lower Latency

A tool that monitors 30k API calls per minute moved away from its initial Node.js implementation on AWS Fargate after hosting costs became untenable. The service was originally architected as a data pipeline with log data flowing through Kinesis into S3, and to autoscale over 4,000 requests per minute. Heavier functions started to log many garbage collection pauses.

Rust’s defining advantage is that it is not a garbage-collected language. Instead, memory is managed by a concept called ownership. Rather than performing a drop-in replacement, the new service went through Kubernetes. Its core is now an Envoy proxy with a Rust sidecar application. Notably, the new architecture handled increased data volume with fewer and smaller servers. Following the transition, latency dropped to below 90 ms even at the highest peaks, keeping the average response time under 40 ms.

Node.js to Kafka at DoorDash

After encountering repeated outages with its task-processing system, DoorDash chose to replace RabbitMQ with Apache Kafka without downtime. The decision, which on Hacker News sparked a debate about whether an engineering team is obligated to fix bugs in an open source tool rather than switch to an alternative, drew criticism. Commenters found it interesting that they decided to go all-in on a solution without investigating or trying the other proposed paths.

HTTP/3 at Facebook: 75% of Traffic and the App-Logic Ripple Effect

Facebook reports that more than 75% of its internet traffic now uses QUIC and HTTP/3 together. The shift has driven a 6% reduction in request errors, a 20% reduction in tail latency, and a 5% reduction in response header size for people using Facebook compared with HTTP/2. But tests revealed a puzzling regression: even though QUIC was enabled for dynamic requests only, the error rate increased for static content over TCP. The structural cause would repeat later: application logic adjusts the type and quantity of content requests based on the speed and reliability of other requests. Improving one type can negatively affect another.

Despite that, QUIC has had an outsized effect on the video experience. Mean time between rebuffering improved by up to 22% depending on platform; the error count on video requests fell by 8% and the stall rate by 20%. Proponents argue the optimization effort is worth it, even if the savings are a single digit percentage. The idea says that each year, users globally spend an enormous amount of human time waiting for resources. A 1% efficiency gain across that scale outweighs a sizable engineering investment in upgrading.

One counter-intuitive root cause of regressions uncovered by the transition proved to be a phenomenon familiar to anyone dealing with modern systems: improving the type and quantity of app requests based on the speed and reliability of other types of content may have detrimental side effects.

Beyond Prediction: Engineering for Adaptation

In the domain of traffic control, a complexity scientist at MIT has demonstrated a method for beating gridlock by using adaptivity instead of prediction. Self-organizing traffic lights have sensors that allow them to respond to incoming traffic by adjusting their timing. They constantly adapt to the changing flow, and because there is no direct communication between lights, the result is a self-organized system in which “green waves” of cars emerge naturally. He contrasts this to a scenario where a controller implements a solution based on predicting a fixed signal timing. Instead, he believes we should engineer systems with the goal of constantly finding a range of good, not optimal, solutions: “The moment you achieve optimality, the problem changes.” These novel approaches shift from prediction to adaptation.

Amazon’s Advice on Data Plane Failures: Put the Smaller Fleet in Control

Modern systems within Amazon are designed so that the data plane calls upstream API services only for configuration and management. Two related analyses from AWS describe how to avoid outage cascades. When the scale gap becomes very large—on the order of 100 to 1—any correlated client behavior can overwhelm the control plane. The remedy: remove this “scale mismatch” by moving configuration into S3. Control plane software writes all published configuration to an S3 bucket, while the data plane polls it. Control plane state is likewise pushed downstream to another bucket, which data plane instances periodically read their operational state from.

Sources note that cache workloads at Twitter often have more than 35% write-heavy usage, indicating that the field of in-memory caching is also shaped by storage cache patterns.

Other Noteworthy Production Notes

  • Podcast host charts show the prevalence of strong engineering opinions is spreading to many development shops. Net choice for offline requirements on a media workflow suite—for producing content over potentially unstable networks—included Kotlin Multiplatform, which now handles nearly half of the shared business logic between the iOS and Android versions of the app those developers built.
  • An ask for which performance doesn’t require an operating system-level fix was explaining how 40 ms of persistent latency in a networked service turned out to come from the Nagle algorithm. When enabled in Linux, TCP collapses tiny writes into fewer larger ones to economize on packet overhead, which involves a timeout before flushing smaller queued messages. The error is said to have gone unnoticed in the operation code and then resolved via a call to setsockopt() and TCP_NODELAY.
  • A breakdown guide on log scales: two prominent write-ups from data visualization and distributed systems teams advise engineers read a log scale differently from a linear one and point out more common situations in which users may be visually misled.
  • Cloudflare’s large-scale network traffic report since QUIC support was available shows it delivers value by “9 percent lower tail latency on popular HTTP endpoints” while also raising contention. Someone looked at popular big-bench queries and found surprising similarity in output from five top engines — a sign of data pipeline convergence after many operations decisions bring complexity and may pull the architectures into a lowest common denominator.

Tooling Notes

NGINX Service Mesh has landed as a fully integrated, lightweight option for managing container traffic in Kubernetes environments. Its data plane is powered by NGINX Plus, and the mesh is designed to be deployed without the operational overhead of larger service mesh alternatives.

TimescaleDB 2.0 is now available as a multi-node, petabyte-scale relational database for time-series data — and it remains completely free. As one Hacker News commenter pointed out, it is worth distinguishing it from ClickHouse: Timescale is row-oriented and specialized for time series, whereas ClickHouse is a columnar store. Timescale does borrow some columnar benefits, but the underlying architectures differ.

For RISC-V work, SiFive has released the HiFive Unmatched, a development platform aimed at professional engineers building RISC-V software. The board is meant to support the kind of low-level development work that has previously been hard to do without more mature hardware ecosystems.

A handful of new open-source libraries also caught attention this week:

  • cdk-patterns/serverless — The Lambda Circuit Breaker implements the simple webservice pattern, but instead of using DynamoDB for data storage, it tells a Lambda function whether the upstream webservice is currently reliable or whether a fallback function should be invoked.
  • active-logic/activelogic-cs — A Behavior Tree library built from the ground up for C# programmers, aiming for comprehensive coverage while staying easy to use.
  • americanexpress/unify-jdocs — A JSON manipulation library that removes the need for model/POJO classes entirely, operating directly on the JSON document. The pitch: once you use it, going back to Java model classes or JSON schema-based validation may feel unnecessarily heavy.
  • facebookincubator/CG-SQL — A code generation system for SQLite that lets developers write stored procedures in a variant of Transact-SQL and compile them into C code using SQLite's C API. It is aimed at engineers who need highly complex stored procedures with very large queries, without the manual checking existing approaches demand.

Reading on Distributed Systems

Several papers and resources made the rounds this week, covering everything from satellite networks to learned algorithms.

Loon SDN describes a large-scale implementation of a Temporospatial SDN used as a cloud service for coordinating aerospace networks. The system schedules both the physical wireless topology and packet routing across terrestrial, air, and space segments of participating networks. Scheduling decisions are based on the propagated motion of platforms and high-fidelity modeling of wireless link relationships, constraints, and accessibility. The SDN is designed to give network operators more flexibility and situational awareness, enable inter-network interoperability, and coordinate interference avoidance. The same paper has been evaluated for applicability to NASA's next-generation space communications architecture.

Ocean Vista tackles strict serializability in geo-replicated databases through a technique the authors summarize as "geo-replicate-ahead transactions." A transaction is first replicated to all parties across datacenters; those parties then check that the watermark rises above the transaction to confirm that all preceding transactions have also been replicated successfully. Only then does execution proceed asynchronously at each party.

The case for a learned sorting algorithm builds on SageDB work and applies machine learning to classic sorting. On a 1 billion item dataset, Learned Sort outperformed RadixSort, the next best competitor, by a factor of 1.49x — and notably, that figure includes the time spent training the model.

Software Engineering at Google is available as a paper cataloging key practices at the company, covering the engineering culture and processes that have scaled with the organization.

Ben Stopford's Designing Event-Driven Systems explains how service-based architectures and stream processing tools like Apache Kafka can support business-critical systems. The book is available for free.

The Synchronous Data Center asks whether the asynchronous model remains the right default for most distributed systems, particularly those within a single administrative domain. The authors argue that, borrowing ideas from the cyber-physical systems domain, it is technically feasible to build fully synchronous datacenter-scale systems — and that doing so would bring several advantages over current designs.

Orbital Edge Computing proposes nano satellite constellations as a new class of computer system. Coverage of a ground track and image-tile processing is divided among constellation members in what the paper calls a computational nanosatellite pipeline. On a related note, Azure Orbital has been introduced to process satellite data at cloud scale, with the goal of moving datacenters as close as possible to satellite downlinks.

For those working on IT architecture, the Architecture Playbook offers practical tools aimed at producing architecture or designs faster and better, with a focus on real-world application rather than theory.

Two free resources worth flagging: Concurrency Control and Recovery in Database Systems from Microsoft Research, and a paper from Amazon Science on distribution-key recommendation for Amazon Redshift. The Redshift work describes a method for allocating data across servers; in experiments involving multi-table queries, it reduced communications overhead by up to 97% relative to the original, unoptimized configuration.