Numbers That Matter

  • 60%: of people dropped an app idea or withheld features over concerns that Apple wouldn't approve it; 69% say Apple's App Store policies stifle innovation.
  • 130 million: connections in a human brain map covering 50,000 cells, built from a 1.4-petabyte dataset. A mouse brain would be about 1,000 times larger, on the order of an exabyte.
  • 60x: more power than expected consumed by the Atlas humanoid robot.
  • 1.8 EFLOPS: throughput of Tesla's supercomputer, which the company plans to use for vision-only autonomous driving.
  • 15–20%: increase in internet traffic during fall 2020 caused by COVID lockdowns.
  • 80%: of organizations that paid a ransom were hit again by the same attackers.
  • 40%: productivity gain at Microsoft Japan after switching to a four-day work week, capping meetings at 30 minutes, and limiting attendance to five people.
  • 2.8 billion: API requests processed per day on GitHub, peaking at 55k requests per second. The platform also averages 2 billion git operations per day and runs ~800M background jobs daily.
  • 1 petabyte: per week of data transferred between national laboratories.
  • 1 billion: sparrows worldwide.
  • 90 seconds: time to build LLVM on top of Lambda; the same build took 2 minutes and 37 seconds on a 160-core ARM machine.
  • 1 exabyte: total data stored across all of LinkedIn's Hadoop clusters.
  • #1: Oracle has been the most popular database since 2006; MySQL holds second place.
  • $5K: cost of a Chip Scale Atomic Clock roughly the size of a grain of sand.
  • 19 mph: speed at which seeds are dispersed from a plant.
  • 80 million: IOPS on a standard 2U Intel Xeon system using NVM Express SSDs.
  • $55 million: price for a seat on a SpaceX flight.
  • 28 billion: photos uploaded to Google Photos each week.
  • 80,000: iterations required to generate the NFL's annual schedule.
  • 268: books that sold more than 100,000 copies, out of 2.6 million books sold online in 2020.
  • 400 million: API calls per second handled by @AWSIdentity.
  • 26: new writing systems invented in West Africa.

Cloud Economics and Bare-Metal Truths

Cloud spending jokes write themselves. One engineer quipped that an SRE can burn through a mythical billion-dollar windfall faster than any genie’s rules allow, provided AWS is on the table. Beneath the humor lies a recurring theme in this week’s discourse: the real cost drivers aren't always where you expect them.

As one commenter noted, the expense with S3 lives in the PUT and GET operations, not the storage itself. For high-velocity access patterns, the advice is to bundle smaller objects into larger ones to keep request costs in check. Similarly, a developer reported a greater than 80% cost reduction serving image assets after switching away from CloudFront, conceding that at their scale, milliseconds of latency don't matter. @uhoelzle touted GCP’s new Tau VMs as 56% faster than Graviton2 with 42% better price/performance, calling it a “leapfrog” rather than an incremental improvement. Yet, cloud performance isn't uniform: benchmarks show AWS leading in latency at low to modest loads, while GCP wins under high load, and Azure sits in between. One report noted that identical instances on GCP can vary by as much as 20% in performance, complicating capacity planning.

The economics also push toward architectural simplification. @kellabyte mused whether it’s cheaper to store non-indexed data on S3 and brute-force it with parallel compute than to maintain optimized local structures. Theo Schlossnagle explained that the decision to build a custom time-series database came down to simple economics: restricting the problem space yields a cheaper, faster, more maintainable solution that can outmaneuver any technical workaround. On the flip side, a common gripe targets organizations that refuse to pay $20K/year for SaaS but burn unlimited engineering effort building internal tooling, then complain about capacity.

Reliability, Recovery, and the Fallibility of Hardware

The conversation around reliability is shifting from a focus on redundancy to a focus on recovery. As one commenter put it, perfect availability is impossible, and a system that recovers in minutes is often better than one that struggles to stay online and fails spectacularly. This sentiment echoes a deeper concern: hardware errors are inevitable at scale. Fail-silent corruption execution errors in CPUs are technically and economically infeasible to fully test against, meaning software must be skeptical of the data returned by cores, storage devices, and network interfaces alike. This could push datacenters toward simpler chipsets like ARM, which AWS has already embraced with Graviton for energy efficiency.

Tim Bray, reflecting on his years at Google and AWS, noted that outages are rarely caused by simple software bugs. More often, they stem from botched deployments, throttling misconfigurations, certificate problems, DNS hiccups, or malfunctioning canaries. Still, he argued, once you're infected with the testing mindset, you're never comfortable with untested code. A Mars Helicopter incident illustrated the subtlety of failure: on its sixth flight, a single missing image caused the onboard system to misjudge its position, leading to constant oscillation as each subsequent frame bore a wrong timestamp. The system compensated by adjusting speed and tilt, but the root cause was a data processing anomaly, not a hardware fault.

One engineer's rule of thumb for debugging: always verify that the code running in production actually matches the source code you're reviewing. Many “impossible” bugs turn out to be entirely possible in a deployed version that has diverged from the repo. Another recalled a load balancer bug at Stack Overflow that slammed web servers with 300–500 requests per second because they were assumed to be up for rotation rather than ready for health checks, causing a Redis connection to take two minutes to establish.

Data, Storage, and the Evolution of Databases

The next generation of databases may bridge the gap between transactional storage and stream processing. As one architect predicted, databases and stream processing will increasingly become two sides of the same coin, with the definition of a database expanding to natively include streaming. Others are already building on this premise: a developer described querying entire object graphs in a single round trip, which has fundamentally changed how their company builds applications.

For DynamoDB users, the suggestion is to prefer KSUIDs over UUIDv4. KSUIDs embed the current time, making them sortable and more unique, which provides a chronological access pattern for free. Spanner, meanwhile, has historically scaled indefinitely with strong semantics, but its price of entry was prohibitive for small use cases—a barrier that no longer exists, according to @tlipcon.

On the storage side, NVMe 2.0 lays groundwork for computational storage, even if the ecosystem isn't ready to standardize it. Micron, however, pulled the plug on 3D XPoint, citing insufficient market validation to justify the investment required for commercializing it at scale.

Languages, Frameworks, and the Hype Cycle

The debate over technology choices continues. One team using Rust for backend development emphasized that safety—particularly thread-safety—was the critical reason, not performance. The request rate is low; correctness is what matters. Khan Academy’s migration to Go has been positive, even if Go is more verbose than Python. The team values its speed, solid tooling, and production reliability.

Shopify remains the poster child for Rails, running a 2.8-million-line Ruby monolith that processes over $100 million in sales per hour at peak. Meanwhile, Coinbase rewrote both its Android and iOS apps in React Native and is happy with the outcome, partly because hiring enough native mobile engineers is difficult. In contrast, Stack Overflow’s architecture is still a simple server farm with aggressively maintained code and a lightweight SQL ORM, explicitly avoiding the latest trends to keep things maintainable and performant without spending tens of millions on infrastructure.

Security, Insurance, and Organizational Dynamics

The cyber insurance market is tightening as risk—defined as amount of loss multiplied by probability—has soared. A mere five payouts can erase the annual premiums from 250 companies with at least $200 million in coverage, putting pressure on insurer profitability. On the security front, one self-described hacker noted that 95% of vulnerabilities are easily found by scanners or inexperienced testers; skilled hackers cost 10–100x more but only yield an additional 2–3%. The advice: don't waste money on higher-quality services, since nobody can guarantee 100% coverage.

Organizational politics also play a role. A savvy engineer knows how to gain the CEO’s confidence long before proposing a plan, spending months ingratiating themselves with the political circle. Another perspective warns that the hyperactive hive mind of modern collaboration makes deep work nearly impossible, as constant communication overwhelms any non-trivial focus.

The Human Element

Underlying the technical discourse is a recognition of human fallibility and emotion. One developer admitted to taking down production before, a rite of passage for many. Jill Bolte Taylor reminded readers that humans are “feeling creatures who think,” not the reverse, and that bypassing emotions can derail mental health. Aleron Kong’s character lamented that the internet, meant to make everyone smarter, instead amplified the noise of idiocy, drowning out logic.

Even in engineering, there are lessons about leadership. A chief’s stern rebuke—that no job is more important than another, just different—resonates with the idea that every part of a system must be done just so, or things fall apart. Geoff Huston paints a broader picture: the borderless, open internet is increasingly viewed as a threat vector by nation-states, leading to disillusionment as the promise of a frictionless common digital infrastructure collides with national interests and unease.

Microservices: Old Disasters, New Lessons

A roundup of microservices failures reads as familiar territory, but it still managed to spark a thoughtful discussion. The recurring disasters — services sliced too thin, tangled development environments, flaky end-to-end tests, a giant shared database, API gateways as choke points, and brittle timeout/retry logic — are symptoms of a deeper truth about software creation. As one commenter put it, give a problem to five groups and they’ll return 12 different solutions. Each individual and each decision injects noise that compounds into the usual complexity and errors.

There is no final answer here. The thoughtful threads all conclude that as long as humans write software, complexity simply moves to a higher level of abstraction. One strategy for damping the noise is industrializing components, as Google does: a dedicated organization of 500-1,000 engineers maintains the microservices substrate — tracing, monitoring, auth, log parsing, anomaly detection, deployment, release management — so the other 90% of engineers don't need to think about it. Most companies have outsourced that same role to cloud providers. Education alone won't bridge the gap; one developer noted that new graduates know algorithms but have no idea how to assemble a modern large-scale system, a skill that currently requires either reading everything ever published or spending five years making every possible mistake.

Numbers, Titles, and Career Moves

A piece of infrastructure advice worth repeating: a number used for counting should always be 64 bits. Even on constrained embedded systems, it's usually fine. When an int rolls over, you didn't handle it everywhere it needed to be handled.

One senior engineer's "drunk post" of lessons learned contains much that's hard to disagree with, but the author of the roundup does push back on one item: titles do matter. If you're on the management track, go to a small startup and get the biggest title you can; it defines your base level as you progress.

A Serverless Postmortem

ASOS offered a concrete example of stateful serverless workflows built on Azure Durable Functions, an extension of Azure Functions for writing stateful functions in a serverless environment. They use Durable Entities to aggregate event data from multiple sources over indefinite periods. The system handles over 100,000 email notifications a day — though a separate review of Azure left some readers less confident.

Edge Computing Enters Another Player

Akamai entered the serverless edge space with EdgeWorkers. The platform is built on a network of more than 340,000 servers across ~4,100 locations in ~130 countries, which delivered over 300 trillion API requests — a 53% year-over-year increase. Two tiers are offered: Basic Compute for lower CPU/memory applications and Dynamic Compute for higher demands, with a combined free allowance of 60 million events per month, up to 30 million per tier.

Early impressions look similar to CloudFront's initial offering: a small function runtime and a KV store, built on V8 so most Node.js features work, but no WebSockets and various other platform limitations. Pricing details were not published. Like other edge platforms, it is not a generic place to run arbitrary code — the examples skew CDN-related: cookies, request headers, load balancing, and image optimization.

When Executives Answer Tech Questions with PR

Two high-profile interviews showed what happens when serious technical questions are met with talking points. In one, Apple's Craig Federighi and Greg Joswiak were asked about developer concerns regarding App Store policies. They responded by dismissing the concerns as not founded on reality, professing love for developers without addressing any specific issues, and eventually calling the concerns "crazy."

In the other, John Deere's CTO fielded a question about right to repair. The technical discussion leading up to it was compelling, but on this point he doubled down on safety and security arguments, suggesting a bad software change could turn a combine into a weapon. The criticism: John Deere has become a platform without realizing it. Just as Apple built iOS with APIs and secure hardware in a sandbox, John Deere needs an app store equivalent for its equipment. Repeating safety arguments to defend the status quo ignores what customers actually want.

The Cloud Repatriation Paradox

Cloud repatriation — moving workloads back on-premises — triggered a wide-ranging debate. The strongest analogy: horse owners telling car owners that horses have a lower TCO. That misses the point. But the counterargument is that cloud providers behave like Apple, assuming price doesn't matter. If they don't keep pushing costs down, they invite disruption from below, though no one expects a wholesale return to the painful days of on-premise.

The historical perspective is important. Building a site like Yahoo from scratch in a datacenter required an almost incomprehensible breadth of skills. The cloud democratized that, making it possible for anyone to build comparable systems. Those arguing for repatriation generally have rare skills or the budget to hire them — which doesn't apply to most of the industry.

Two field reports illustrated the trade-offs. One engineer who led datacenter and networking at Square from 2011 to 2017 ran roughly 100 server racks across four US datacenters, plus Japan and network pops, with a team of about 6-7 people for production. His cost analysis consistently showed self-hosting at half the price of AWS three-year reserved instances. The rebuttal: the same analysis skips the elastic aspect. When the pandemic hit, one company's metrics spiked 15-20x overnight; the entire infrastructure response was clicking buttons in the AWS console. No one can buy hard drives that fast.

The crowd's advice for anyone considering a move: exhaust cost optimizations first. Design for lower costs, turn things off, negotiate with your provider, and pit providers against each other. Also remember that VCs don't care about your career; think about your own trajectory.

It's Always DNS — And a Serialization Library

Bunny lost its DNS network in spectacular fashion. The team had switched from JSON to BinaryPack, a binary serialization library, to reduce memory, traffic, and garbage collection overhead. Everything improved for a few weeks — memory down, GC pause time down, CPU down — until it all collapsed. A corrupted file triggered a stack overflow exception in BinaryPack that bypassed exception handling and exited the process. Within minutes, the fleet of nearly 100 DNS servers was dead.

The deeper flaw: Bunny's infrastructure was designed so critical internal systems relied on each other. In this case, a component believed to be non-critical and not made redundant turned out to be load-bearing. The system was supposed to handle it gracefully if the whole thing went offline. It didn't.

One commenter pointed out the real lesson isn't about infrastructure reliance but about release practice: the triggering event — a new software version — isn't novel. Systematic improvements would include replay tests, canaries, gradual deployments, and isolated regional rollouts. Also worth asking: if a single engineer can take down the whole system with a quick fix — as happened in the Salesforce DNS outage around the same time — the system is broken by design.

The library author's response was blunt: BinaryPack hasn't been updated in three years, was never claimed to be production-ready, and would have been flagged as such if asked.

Scaling, Refactoring, and Simplicity

A case study on handling one million concurrent players on AWS is self-serving and hand-wavy, but the headline number is a 50% cost cut while doing a lift-and-shift and transforming the system into a scalable, low-latency architecture.

Backblaze ended an era by deciding to stop building its own storage pods. The company will now buy commodity hardware from Dell — a clear signal that it's a software company now. The open question is whether it can maintain its edge without innovating over the whole stack.

For capacity planning, Facebook shared its "network hose" approach. Instead of forecasting traffic between every source-destination pair — O(N²) data points per service — it forecasts only total ingress and egress per datacenter, O(N) per service. This bakes statistical multiplexing into the forecast, cut complexity by an order of magnitude, and eliminated a significant number of traffic-surge alarms, since surges are now tracked in aggregate per datacenter rather than between every pair of datacenters.

A Magento 2 MySQL detective story turned out to be a classic case of a missing index — but the value was in the method for diagnosing horrendously complex queries. The result: CPU usage dropped under 25%, QPS increased by 5x, and query time fell from 20 seconds to 66 milliseconds. That was alongside another piece on a gRPC stream performance gotcha.

A load-testing post presented numbers from the cloud: 1.5 million hits per minute generated from 500 load agents. The target was a LAMP stack on Debian, load-balanced by a customized NGINX, with heavy use of memcache for reads and a "firehose" approach to DB writes. The testing even uncovered a memcache bug that fixed high-concurrency behavior.

One banking platform's serverless experience was mostly positive: event-driven serverless microservices fit eventually-consistent systems well, tokenization protects private data, and scale was never a concern. Mistakes included a spider web of API gateways talking to each other and "fat" Lambdas. The fixes: one Lambda does exactly one thing, EventBridge sits between domain boundaries, SQS carries commands instead of API Gateway, and Step Functions handle orchestration because visualizing state machines is valuable. Fan-out first is the serverless way — SNS to SQS, Dynamo streams to SNS to SQS — with shared code packages to make subscriptions easy. Only use ordering when absolutely necessary.

Engineering Culture, from Twitch to SpaceX

On the tenth anniversary of Twitch's launch, its founder shared growth lessons that apply broadly: make something ten people love rather than something most people think is pretty good; consumer products must be a daily habit or a consistent response to an external trigger; and there are only five growth strategies — high-touch sales, paid advertising, intrinsic virality, intrinsic influencer incentives, and platform hacks. Press and word of mouth are not strategies.

SpaceX's software AMA revealed a disciplined engineering culture. The overarching principle is knowing how software behaves in every scenario, which pushes design toward small components with clear boundaries that can be fully tested before integration. Testing is a shared responsibility — every engineer writes and tests code. The CI/CD team exists to support high-fidelity physics simulations and both hardware-in-the-loop and hardware-out-of-the-loop testbeds, workloads that off-the-shelf CI/CD tools don't handle well.

Satellites serve as canaries for new features: regressions are run to protect critical functions, then a feature rolls out to a single satellite for monitored testing with minimal constellation risk. Software updates target the entire fleet — satellites, ground stations, user terminals, and WiFi routers — roughly once per week, with each device checking in and downloading during ideal low-impact windows. Dragon is fully autonomous, able to complete ISS missions without crew interaction. Interfaces follow a "quiet-dark" philosophy: minimal when nominal but showing overall system status, visually prioritizing anything off-nominal.

The Starlink network is engineered for dynamism. Satellites move over 7 km/s, so a user is never connected to one for more than a few minutes. User terminals use electrically steered beams to switch between satellites and buffer traffic in anticipation of handover. Data is aggregated over time and aged out when no longer useful rather than stored indefinitely. On vehicle software, the lesson is situational: sometimes the best place for a control algorithm is embedded close to the thing being controlled; sometimes a process needs centralization. Critical operations have defined error paths and recovery strategies. The company culture takeaway from a commenter: SpaceX understands viscerally that a company's culture is shaped by its processes and hierarchies of responsibility, whereas most companies box employees into "I was hired as an expert in hammers, so every task looks like a nail."

Performance, Algorithms, and Error

A QuestDB benchmark claims write speeds of 1.4 million rows per second, reaching maximum ingestion with just four threads. The comparison shows QuestDB at 959k rows/sec with four threads, versus InfluxDB at 334k rows/sec requiring 14 threads and TimescaleDB at 145k rows/sec with four. ClickHouse reaches 914k rows/sec but needs twice the threads. On equal thread count, that's 1.7x faster than ClickHouse, 6.4x faster than InfluxDB, and 6.5x faster than TimescaleDB.

For database availability, "AlwaysOn" is a strong claim requiring zero-downtime solutions for eight distinct events: hardware failure, software failure, software upgrade, hardware upgrade, schema reorganization, online scaling, major upgrades, and major disasters. The practical availability limit becomes failure discovery time — detection is immediate for most software failures but can take a few seconds for hardware failures relying on heartbeats.

On extreme HTTP tuning — 1.2 million API requests per second on a 4 vCPU EC2 instance — the author is refreshingly honest: don't expect 5x gains by cargo-culting the configuration. The optimizations won't matter unless you're already serving more than 50k req/s. The real value is the profiling methodology, which should give you a better understanding of your system's behavior and reveal bottlenecks you didn't know you had.

Daniel Kahneman's work on noise as a flaw in human judgment maps naturally to software. Error has two sources: bias and noise. Algorithms reduce noise because they make consistent decisions given the same data — humans are noisy, and a bad lunch can change a decision. But the theory misses that algorithms can encode bias deep within a system, so deeply we don't notice. Decision hygiene can reduce errors: averaging independent judgments, letting people form opinions before group meetings where the loudest person wins, and delaying intuition until after research loads up the facts.

Instagram pulled back the curtain on its ranking algorithm. For Feed, it makes about a dozen predictions on interaction likelihood, weighing five most closely: spending a few seconds on a post, commenting, liking, saving, and tapping the profile photo. The more likely you are to take an action and the heavier the weight on that action, the higher the post appears.

New Tools for Decentralized and Serverless Work

Yatima is a pure functional programming language built in Rust for the decentralized web. It relies on content addressing, supports first-class types, linear/affine/erased types, and type-safe dependent metaprogramming — a combination aimed at verifiable, self-contained programs.

For a simpler approach to offloading work, llama provides a CLI that outsources computation to AWS Lambda. OpenTelemetry continues to gain traction as the observability framework for cloud-native software; GitHub detailed its own adoption of the standard in a recent engineering post.

ClickHouse remains a notable open-source, column-oriented DBMS designed for real-time analytical reporting. Microsoft’s CyberBattleSim offers an experimentation platform where automated agents operate in a simulated enterprise network to study attack and defense interactions at a high level of abstraction.

Dissecting Failure: Podcast and Deep Dives

The Downtime Project podcast covers outages and their root causes. In an episode titled 7 Lessons From 10 Outages, the most common culprit is circular dependencies — the very tools you need to diagnose an outage often depend on the component that failed. Complex automation also fails spectacularly when it acts on outdated assumptions about its environment, and running high-variance queries against production databases is another path to trouble.

For a visual deep dive, RavenDB founder Oren Eini walks through database engine internals on stage, disassembling the components required for an industrial-grade database.

From NVMe Arrays to Metastable Failures

A CIDR paper examines how databases can exploit Directly-Attached NVMe Arrays (DANA). PCIe-attached SSDs now provide throughput and capacity that previously required DRAM, and modern servers can host four to eight of these drives for impressive aggregate bandwidth.

On the formal methods side, a paper from the IEEE Symposium on Security and Privacy presents an I/O separation model for kernel verification. The model defines an authorization-based separation policy that is hardware agnostic; specified and verified in Dafny, it enables generation of verified-correct assembly code enforcing the policy in the Wimpy kernel.

One key insight from analysis of Google’s TCMalloc paper: increasing hugepage coverage improves application performance, not necessarily allocator performance. Spending a little extra time in the allocator to preserve hugepage coverage reduces total time spent in the application.

Outside the strict data center world, Dressing for Altitude, a NASA publication, covers aviation pressure suits — distinct from the well-documented history of spacesuits.

In artificial intelligence, the “Reward is enough” hypothesis proposes that intelligence and its associated abilities are all subservient to reward maximization, driving behaviors like learning, perception, social intelligence and generalization. A separate paper argues dreams evolved to combat brain overfitting by creating corrupted sensory inputs from stochastic neural activity for improved generalization.

High-Performance Infrastructure at Scale

The Petascale DTN Project details how four HPC facilities achieved routine transfer rates above 1 PB/week. The paper covers the Design of Data Transfer Node clusters, software tools, and performance tuning used to enable that capability.

HotOS 2021 research describes metastable failures — a distributed systems failure pattern that often appears as a black swan event. These outages are nearly impossible to predict but easy to explain in hindsight, though a common framework unifies seemingly different incidents.

FoundationDB, the distributed unbundled transactional key-value store underpinning cloud infrastructure at Apple and Snowflake, has drawn positive industry reports. One user running a live cluster of 35+ commodity machines with 5 TB of capacity reports smooth server additions and removals, plus a 100 TB cluster in testing—using the system as a distributed file store.

Finally, the Internet Computer Consensus (ICC) family of protocols provides atomic broadcast under partial synchrony. These leader-based protocols are fully integrated with a blockchain, supporting the Byzantine fault-tolerant replicated state machines of the Internet Computer.