Prime Day by the Numbers

Amazon's annual Prime Day shopping event produced some eye-popping infrastructure statistics this year. The company reports that 233 billion transactions were processed by Amazon Aurora during the event, with 1,595 terabytes of data stored and 615 terabytes transferred. Thanks to efficiency improvements, Amazon used 6,000 fewer physical servers than in 2020, at a 20% lower cost.

Other notable Prime Day metrics:

  • CloudFront handled a peak load of over 290 million HTTP requests per minute, totaling over 600 billion HTTP requests.
  • SQS processed 47.7 million messages per second at its peak.
  • EBS handled 1.1 trillion requests per day and transferred 614 petabytes daily.
  • DynamoDB sustained 89.2 million requests per second.

Network Speed Records

Researchers continue to push the boundaries of data transmission speeds. A new world record was set with a transmission of 319 Tb/s over 3,001 km using 4-core fiber. By contrast, the current fastest quantum networks operate at speeds measured in megabits per second today.

Meanwhile, real-world internet speeds continue to climb. Mean mobile download speeds increased 59.5% over the last year, reaching 55.07 Mbps. Fixed broadband saw a 31.9% increase to 107.50 Mbps.

Platform Scale and Economics

TikTok has crossed the 1 billion monthly active users mark, closing in on Facebook's roughly 2.89 billion MAU but still well ahead of many other platforms. Apple's App Store is proving remarkably lucrative: consumers spent $40 billion in the first six months of 2021, a 22.05% increase year-over-year. Notably, over 98% of Apple's in-app purchase revenue came from games, and 70% of App Store revenue is generated by less than 10% of all consumers. YouTube, meanwhile, has paid out $30 billion to creators over the past three years from ads, merchandising and other features.

Infrastructure spending continues to concentrate in a few hyperscale clouds. Pinterest has committed $3 billion to AWS through 2029 — roughly $400 million per year, or about $0.88 per user annually at the company's current scale.

Security and Threats

The threat landscape is growing more dangerous. This year has seen 66 zero-day vulnerabilities exploited in the wild — the highest number ever recorded and almost double the total for 2020. Exploits carrying price tags of north of $1 million. Bad bots now account for 25% of all internet traffic, according to Imperva's research.

Cloudflare reported thwarting a 17.2 million request-per-second DDoS attack, three times larger than the previous record. For context, the company serves 25 million HTTP requests per second on average across its entire network. Ransomware claims averaged $1.2 million in 2021, up sharply from $450,000 previously.

Storage and Compute Advances

AWS Lambda functions powered by the Graviton2 processor now offer 34% better price/performance compared to x86-based alternatives. At the storage layer, Backblaze is now tracking 181,464 drives, with HGST posting the lowest annual failure rate at 0.44%. The emerging memory market for both discrete and embedded technologies is projected to reach $44 billion in revenue by 2031, with the bulk concentrated in SoCs and 3D XPoint memory.

On the compute side, over 60 million new EC2 instances are spun up every single day. One developer benchmark demonstrated 5.1 million IOPS per core using AMD Zen 3 with Intel Optane storage.

The cost of satellite bandwidth has dropped by a factor of 7,500x, from $300,000,000 per Gbps down to $40,000 per Gbps. ARK's research suggests that figure could fall another 40-fold to roughly $1,000/Gbps once Starlink and Starship are fully operational. Starlink's current median upload speed is 13.89 Mbps with a 45 ms median latency.

Kubernetes adoption continues to mature: 95% of respondents in Cockroach Labs' trends report handle their own cloud infrastructure, 94% run Kubernetes in production, and 88% are embracing serverless architectures. As one industry observer put it: "We are inexorably marching towards a world where the trust boundary between business logic and the required tools and infrastructure to run it is simply a service API."

In other notable milestones, a Swiss research team calculated pi to a new record of 62.8 trillion digits in 108 days and nine hours — roughly twice as fast as Google's previous record. The Graesser scale of machine learning has reached 4.398 trillion parameters, an apparent nod to the answer to life, the universe and everything.

Quotable Stuff

Simon Wardley describes the "ILC" gameplay of innovate, leverage, commoditize: take a product, turn it into a utility, let others build on it, mine the metadata for patterns, and commoditize those into new component services. It's how you move up the stack, a tactic the Chinese government has used since the 1970s and Amazon applies in the commercial world.

Gergely Orosz identifies a pattern of "Promotion Driven Development." When a current solution struggles, it's seen not as an opportunity to fix, but to start anew, since new projects lead to promotions. The cycle—opportunity, proposal, funding, hiring, building, launching, promotion, iterating—repeats until growth slows, founders leave, and the project stagnates.

A developer's experience with SAFe was transformative: delivery became predictable and they got every ninth and tenth week for tinkering and refactoring. Senior leaders engaged with developers, antagonism dropped, and the team went from untrusted to fully trusted for deployments during peak periods, even winning additional maintenance contracts.

The cost of remote work, per Microsoft data, seems to be a loss of purpose, driven by strong relationships and seeing the impact of one's work.

Matt Rickard notes that "cloud prem" (cloud plus on-premise) deployment is increasingly common: vendors deploy software to a client's cloud account under an isolated VPC or account.

Uwe Friedrichsen calculates the odds of a distributed ACID transaction: for 10 systems, 99.5%^10 means roughly a 95% success probability, so one in 20 such transactions will fail.

Alex Hudson remains an Azure user but finds the proposition weakening, wondering if Microsoft is adding too many products to compete with AWS or shipping before ready. Azure often feels held together with more "sticky tape" behind the scenes than he'd like.

Cory Doctorow notes Spotify extracts heavy fees from labels, creaming 30% of a typical track's revenue. Big Three monopolists can absorb this; independent labels cannot.

Microservices, per Zababa, are OO programming in the Alan Kay sense: late binding, message passing, local retention and hiding of state.

Omegafulw outlines the main downsides of modularized monoliths versus services: independent scaling and load balancing, plus independent releases at the risk of version compatibility issues.

The most important thing in a Michelin review is what is on the plate.

Shreyas notes a storied response whenever a CEO asks at Product Review, "what do you need to 10X users?"—the PM always says "We need [N] more engineers," and the Eng Manager nods approvingly.

Sourcegraph's memory optimizations cut RAM per repo from 1400KB to 310KB, a 5x reduction enabling five times more repositories on the same servers with no measurable latency change.

Vijay Gill asserts that 100% of self-caused outages could be predicted by five metrics: thread-pool usage; memory, heap, and GC times; CPU load; network failures and connection counts; and slow database queries.

One operator runs 100% on-demand workloads: the effort to reach perfection in auto-scaling is not worth the operational complexity, especially for steady or near-steady state workloads.

Art is about oneself; design is about others.

Using cross-AZ data transfer pricing to illustrate, Corey Quinn asks: if it were free instead of $20,000 per petabyte? Customers would build more multi-AZ architectures, which provisions more storage and compute, increasing traffic over dedicated fiber between AZs, a one-time investment. It's a charge significant enough to alter quarterly earnings.

An accidental infinite event loop between two Lambda functions created a several-hundred-thousand-dollar bill in hours. The lesson: verify no loops between resources and configure scaling limits where available, as infinite scaling is fatal when unintended.

Performance is not linear, strengths have corresponding challenges, and today's organization is the sum of countless days; paradoxes, puzzles, and messes abound.

For Lorin Hochstein, "root cause of failure" loses meaning in complex systems because many control processes keep things running. A failure is the failure of that whole set; singling out one process post-incident is not meaningful since that process alone couldn't have caused the outage.

Jesse Duffield reframes coupling and cohesion: pieces that change for completely different reasons should be separated with minimal inter-dependencies, while those that change for the same reason should be co-located with interdependence coded explicitly.

Alexander Amini notes the bias in autonomous vehicle training data: about 90% comes from sunny, clear conditions, yet deployments must handle the underrepresented 10% (night, bad weather, tight turns).

Ben Kehoe disputes that cost is the sole serverless driver; it's usually combined with the reduction in operational burden, and increased cost is sometimes acceptable.

With serverless, someone else sets up networking, servers, provisioning, and scaling—no Spanning Tree knowledge required. You can focus on delivering value instead of operations.

A FAANG staff engineer shares one lesson: be inspired by colleagues, but understand the thin line between inspiration and envy and work to stay on the right side of it.

In randomized controlled trials, dashboards drove less engagement and worse patient outcomes, while non-dashboard approaches had 20x better engagement and 20-100% better outcomes depending on the condition.

Dr. Ian Cutress on IBM's cache design: the system tracks data on a cache miss using broadcasts and memory state bits to track those broadcasts to external chips, confirming all copies are invalidated before processing the data.

A new attack turns a PC's memory module into a modified Wi-Fi radio, transmitting sensitive data from air-gapped computers at 100 bits-per-second over nearly six feet.

Simon Wardley again: the future is about managing capital flow through applications, monitoring where money is spent, and tying it to value creation. iRobot's fleet of tens of millions of Roombas is operated by only six people.

On private cloud and Kubernetes: they have a purpose as a place to "strangle" legacy systems, but they are not the future and you shouldn't build new there.

In Amazon's DynamoDB ecosystem, one engineer chooses Streams-to-Lambda for guaranteed update processing and transaction Items for multi-phase commits, noting the isolation of the TransactWrite API is too low for some needs.

Notable Numbers and Notes

A Go 1.17 release delivered a 5% performance improvement and 2% smaller amd64 binaries.

Moving web apps from ECS to CloudFront quietly reduced infrastructure costs by 28%, and APIs moving from containers to Lambda/API Gateway should cut them another 44%.

A move from AWS to GCP cut one company's bill by two-thirds, though it included re-platforming. The biggest save was moving from AWS ECS to GCP Kubernetes, letting them safely use preemptible VMs. Cockroach Labs' 2021 Cloud Report found AWS's c5a.4xlarge machine 12% cheaper than GCP and 35% cheaper than Azure's most cost-efficient OLTP options, with AWS also besting network latency for the third year running. AWS's Graviton2 won the multi-core CPU test but performed lowest on seven of 12 benchmarks, including all storage I/O and single-core CPU. A reported migration of a 40,000-core monolithic Python app to Kubernetes took eight months and "a third of my sanity."

Migrating from physical servers to OpenShift was far more effort than expected. Developers benefit by having it on their resume, which helped retain talent, but users saw no benefit, CI is more complicated and slower, and now license fee increases have removed the cost advantage. A migrating team condensed five tables into a single table design inspired by Rick Houlihan and Alex DeBrie, reducing game state rebuild and push time by 60%.

Telcos once dominated communications as unassailable domestic monopolies. Deregulation brought like-for-like competition, but substitution from Internet services shifted value from carriage to content. As digital capacity improved, cloud and content providers acquired dedicated transmission systems, commoditizing shared carriage into a niche.

Ably runs public cloud on AWS, in EC2 instances—thousands of machines across ten regions. They run Docker and deploy in containers but do not use any well-known orchestration layer; each instance knows its containers based on its autoscaling group.

Netflix's ProRes video processing speed improved from 50GB/Hour to 300GB/Hour; the processing-time-to-movie-time ratio dropped from 6:1 to about 1:1.

Architecture and Process Insights

For persistent state, thinks @gunnarmorling: scale up before scaling out. Put a RDBMS on a decent machine and ensure queries aren't bad, and you will avoid the headaches of distributed state.

Marc Brooker has a dual lesson: high-percentile latency is a bad efficiency measure but a good leading indicator of overload. If latency must be used for efficiency, use the mean. On caches he warns that applied incorrectly they become unstable or even metastable: good ones have feedback loops like back pressure and limited concurrency; bad ones are open-loop.

Geoff Huston on CDNs: pick one, and you are fate sharing. Choosing multiple for a self-healing mutual backup is far more complex.

An exiled AWS user accidentally created an infinite loop between two Lambda functions that racked up hundreds of thousands of dollars in a few hours—"infinite scaling is great until you do it when you didn't mean to."

The Internet Explorer team, per @hadip, worked harder than any startup, sprinting for years; they ate every meal at the office and held foosball tournaments at 2am to keep energy up.

A project moving from physical servers to OpenShift saw developer retention benefits (resume value) but zero user benefits, a more complicated CI pipeline, and roughly net-zero cost savings.

An engineer says Kubernetes is a platform for scalable apps, but the path from app development to true scalability is painful.

Also from the tweet streams: "Quietly reduced infrastructure costs by 28% just by moving web apps from ECS to Cloudfront. Moving APIs from containers to Lambda/APIGateway will reduce it by a further 44%."

Spotify's 30% cream from a typical track's total revenue is a heavy extraction that only the Big Three labels can absorb; indies cannot.

James S. A. Corey, on bureaucratic security breakdowns: put parts of a problem into separate boxes with people working in isolation, and within a short time no one is talking across boxes.

Lorin Hochstein argues against single "root cause" thinking for complex system failures, since many controls normally keep it up

For code changes, the advice is to model domain interdependence in code: if no shared reason to change, minimize dependencies; if the same reason for change, share a common interface or call each other.

An engineer working with a behavioral economist found that dashboards produced less engagement, fewer effective interventions, and worse patient outcomes compared to non-dashboard approaches that saw 20x more engagement and 20-100% better results. There is no fundamental law forbidding subjective feelings in human-designed artifacts: consciousness may be possible in machines.

TiloDB planning should revolve around offering a managed service. The capturable value is running it well—upgrades, operations—beyond open-core access. Licensing remains fraught as database vendors seek to prevent clouds from taking code and running it directly. A customer with persistent packet drops resisted remote troubleshooting until a site visit revealed the WAN router plugged into a power strip shared with the office copier. An IT leader asserts it's easier to set up a server than ever. Small startups can outsource to Heroku, but hyperscale organizations want more control and also the developer-friendly niceties of platform abstractions. As parts of the stack turn into simple APIs, "serverless databases" may be an easier paradigm than serverless compute. A Stack Overflow critique: many popular top-voted answers are out of date to the point of being flat out wrong by modern developer standards.

The Z processor's design tracks cache misses via broadcasts, and memory state bits ensure data is valid before use, invalidating other copies system-wide.

A new attack can co-opt a memory module as a Wi-Fi radio to leak data from air-gapped systems at 100 bits-per-second up to six feet away. Prediction is central: "good enough predictions give you everything you can dream about."

Corey Doctorow's broader point about Uber: losing 38 cents on every dollar taken in, with $3.7b in assets of dubious value from overseas failed companies.

Gil Yehuda argues scaling up before scaling out applies to databases; start with a well-tuned single RDBMS machine and you'll travel far before needing distributed state complexity. A startup with $2m raised can burn through it quickly paying Zendesk, Carta, AWS, Slack, Figma, Zoom, LaunchDarkly, JIRA, Amplitude, QuickBooks, Twilio, Sendgrid, Snowflake, G Suite, and 1Password, leaving only a few months of runway.

One developer's discovery on IoT: the spouse asked about the Roomba running all day, and they had to admit they were starting it remotely from the app at work because they liked the double vacuuming and couldn't figure out why it kept pausing. Crypto is "about tearing down banking rules and finding out one failure at a time why each exists."

Kubernetes is a system for organizing, but complex. An engineer says Kubernetes is always at least many thousands of machines, across ten AWS regions? One view: "Kubernetes is a novel techno-human organizational productivity virus developed by Google to destroy startups." Erlang remains a standout choice for server systems because so much scaffolding ships with the runtime—solving hard problems directly rather than managing infrastructure. In the world of robotics, one expert deadpanned, "Robots are blind one armed idiots." Another says autonomous vehicle training data is biased. One specialist notes the majority of autonomous driving data is sunny and clean; deployments fail when they meet the underrepresented 10% of conditions. An interview excerpt notes: "This is why 'root cause of failure' doesn't make sense in complex systems. Many control processes [produce] failure only when the entire set fails." A behavioral economist's tests countered dashboard-driven data sharing, finding less engagement, more ineffective interventions, and worse outcomes. Organizations eyeing serverless should instead focus on capital flow monitoring, user value, and the software supply chain instead of low-level container orchestration. Wardley again rounds it out: serverless is the future—iRobot and Liberty Mutual exploit it profitably, while proprietary clouds and Kubernetes are legacy strangler fig patterns.

The Real Cost of Moving Bytes

Cloudflare’s announcement of R2 object storage has reignited a long-simmering debate about cloud egress pricing. The core pitch is simple: ingress and egress are always free, regardless of volume. Storage is charged, and transactions are free at low volume (roughly less than one per second) but incur fees at higher rates. For both transactions and storage, the stated goal is to be at least 10% less expensive than S3.

The reaction from the engineering community highlights just how entrenched—and how oddly priced—AWS data transfer has become. As one commenter noted, moving data between availability zones in the same region can cost twice as much as moving it between Virginia and Ohio. Another pointed out that egress pricing "still feels like 1998 pricing," starting at nine cents per gigabyte externally to the internet. The implication is that AWS is taking advantage of a generation of developers and founders who have never purchased bandwidth or run an autonomous system, and who assume "that's just what bandwidth costs."

The math gets stark when you consider the internet's 1-to-many nature. If one million people download a single gigabyte in a month, one calculation puts the cost at roughly 13 cents with Cloudflare R2 versus $59,247.52 with S3. That gap is why serious companies are looking at R2 as a front-end for their S3 buckets—not to replace the storage, but to eliminate the largest line item on the bill.

There are caveats. Cloudflare’s own Durable Objects Storage is priced at $1 per million writes at 4KB and $0.20 per million reads, but it is only accessible via a Durable Object, which limits its utility as a general-purpose store. Big-ticket AWS services like RDS, search, EventBridge, and ML are still missing from the Cloudflare stack. And some large customers report that Cloudflare’s enterprise plans cost 10 to 20 times more than its business plans while offering virtually no additional benefit.

Nevertheless, the architectural question stands: what can you build when bandwidth becomes nearly free? Data gravity is a well-known force; data anti-gravity could be just as real.

Netflix's FreeBSD Fast Path

EuroBSDcon videos from 2021 include a look at serving Netflix video at 400Gbps on FreeBSD. The presentation covers the flash servers that serve the most popular content, which are distinct from the "storage" servers with large numbers of spinning drives that handle the longer tail and are constrained by spinning rust speeds.

As one commenter explained, the architecture is about pinning at a low level: network adapters are handled by a specific set of cores, the filesystem handles specific sets of files on those same cores, and the rack router distributes HTTP requests so they always arrive on the network adapter with the appropriate core. It is not unlike partitioning a cluster of servers; it just happens at a lower level of abstraction.

Another observer noted that large multi-CPU servers behave more like a very fast local area network than a single machine when speed really matters. Netflix's approach uses each processor and its attached resources as a single unit, avoiding the bottleneck of moving large amounts of data between CPUs inside the same server.

The engineering investment is substantial. Work on FreeBSD has included async sendfile, the RACK and BBR TCP stacks for good quality of experience, kTLS to save roughly 60% CPU over encrypting in userspace, NUMA awareness, and kTLS offload to move crypto to the NIC. The message: pour years into the workload, and FreeBSD scales.

Serverless at Taco Bell

Taco Bell has gone serverless-first, a choice that it says produced a 99% reduction in costs compared to running EC2 servers 24 hours a day. The scale is real: 7,000 restaurants, millions of pricing points, menus, tax rules, and store hours.

The new order middleware platform deliberately avoided "last gen" tech like EC2 and containers. Step Functions are central, used to process orders and apply rules, with EventBridge driving async flows like delivery. The team praises Step Functions for making it easy to add retries, error handlers, and circuit breaker patterns, and for simplifying load testing by mocking upstream and downstream APIs. The downside is cost.

The broader lesson is cultural. Serverless removes work that doesn't produce business value and lets developers think in terms of composing services. Taco Bell developed a community of practice with evangelists, example repos, a Slack channel for knowledge exchange, and biweekly demo meetings. The paradigm shift itself is hard to sell across an organization, but the team notes it is easier to pick a cloud and use it completely than to stay cloud-agnostic.

What a Computer Is, and Isn't

A USENIX ATC '21/OSDI '21 joint keynote argued that it's time for operating systems to rediscover hardware. The case: Linux does not run on all the processors in a modern SoC—only on a small subset. The rest run their own operating systems, which creates a security nightmare because Linux "thinks it's the only thing running on the machine." Those other processors are capable of compromising the system, a class of attack called a Cross SoC attack.

The broader claim is that Linux was never designed; it "congealed." Modern SoCs need an OS that runs the entirety of the hardware, handling boot loading, power management, and security across asymmetric, heterogeneous, non-uniform multiprocessors. More functions are moving into hardware and being hidden from the OS, and people "have Linux blinders."

On a more prosaic note, another talk reminds us the universe is hostile to computers: a single alpha particle is enough to flip a bit in DRAM, so don't build with radioactive materials.

Monoliths, Convergence, and Costly Clouds

A common objection to monoliths—"high coupling between components" —is countered by noting that components inside a monolith invoke each other via interfaces. Properly done, those components can be compiled together in one executable or distributed across a cluster. A monolith is not just a big library; it has as much structure as you put into it.

On consistency, the terms convergence and confluent are not interchangeable. Convergence is a property of an object: a merge algorithm that smashes divergent replicas together to get the identical result. A dataflow component is confluent if it produces the same set of outputs for all orderings of its inputs. "Consistent" is overused and means little unless preceded by strong, sequential, or causal.

Shopify's capacity planning for Black Friday Cyber Monday shows the discipline required at scale. Forecasting traffic levels with data scientists sets a bar for the platform, and a master resourcing plan estimates how CPUs and storage scale. Owners for the top ten or so resource areas provide detailed breakdowns of machine types, geographic locations, and quantities. Buffers are added to allow flexibility, and partitioning each component into a separate GCP project makes quotas easier to reason about. Regular scale-up testing, backed by internal load testing tooling, is fundamental to running platform-wide load tests.

All of this is set against a backdrop of the "cost of cloud, a trillion dollar paradox," a tale that could be told equally well about health-care, where a family-paying-over-$20,000-a-year for insurance suddenly finds itself considering the repatriation of its medical needs, MRI machines and all, in the name of margin and market cap.

Notable Projects, Talks, and Papers

Open-Source Projects

Badger is an embeddable, persistent key-value (KV) store written in pure Go. It serves as the underlying storage engine for Dgraph, the distributed graph database, and is positioned as a high-performance alternative to non-Go-based KV stores such as RocksDB.

Free Cloud Learning Resources

For developers looking to deepen their cloud expertise at no cost, the Azure Friday video series offers over 700 episodes covering a broad range of cloud topics.

Neuromorphic Computing at the Edge

At the EMEA 2021 event, IMEC presented its work on spike-based neuromorphic computing for the extreme edge. Their compute architectures target event-based sensors and leverage neurons with co-located memory and processing to enable a new class of low-power, edge-native intelligence.

Published Research

Algorithmic Progress vs. Hardware Gains
How Fast Do Algorithms Improve? examines the pace of algorithmic advancement across a large corpus of algorithm families. The study uncovers stark heterogeneity: nearly half of the families see virtually no progress, while 14% experience improvements that outpace hardware scaling, including Moore's law. On aggregate, algorithmic progress for the median family is substantial but lags Moore's law for moderate problem sizes; for big-data problems, however, algorithmic gains surpass hardware gains.

DNA-Based Logic Processing
A paper on a Programmable DNA-Based Boolean Logic Microfluidic Processing Unit (MPU) describes a molecular processor controllable via a PC or smartphone app. The system performs basic AND and OR operations by reacting two input DNAs with a logic template DNA, and supports cascading combinational operations. The authors demonstrate a 2-to-1 multiplexer and XOR functionality via a three-step cascade, enabling AND, OR, and NOT Boolean logic in a simple molecular framework.

Spire: Phase-Symmetric Consensus
The paper Spire: A Cooperative, Phase-Symmetric Solution to Distributed Consensus introduces a novel consensus algorithm that abandons leadership contention. Instead, processes cooperate to iteratively converge on a dominant value and may do so concurrently without conflicts. Each iteration uses an identical message structure and behavior, lending the algorithm a symmetry that simplifies reasoning and improves robustness in distributed systems.

Deep Learning Textbook
Applications of Deep Neural Networks with Keras is a free, 500+ page textbook designed for an applications-oriented deep learning course, available for download.

FoundationDB's Transactional Model
A detailed analysis of FoundationDB highlights how the distributed key-value store is used in production at massive scale by companies such as Apple, Snowflake, and VMware. Notably, Apple's CloudKit runs on FoundationDB. In contrast to many large-scale storage systems that drop transactional guarantees to ease scaling, FoundationDB was designed from the ground up to provide strictly serializable transactions, meaning all operations can be assigned a definite global order—a significant engineering challenge for distributed systems.