The Numbers That Matter This Week

It is time again for a look at the metrics, milestones, and measurements that define the current state of infrastructure and technology. From underwater data centers to database trends, this week's data points offer a snapshot of where the industry stands and where it is heading.

Microsoft's experimental underwater data center has proven remarkably reliable. After two years, the failure rate for its servers was just one-eighth that of a conventional facility, with only 8 out of 855 servers failing. The leading theories point to the absence of human interference, a nitrogen-rich atmosphere, and cooler ambient temperatures as contributing factors to the improved durability.

In other performance news, Shopify is now handling over 100,000 requests per second. Slack's tracing infrastructure is processing an immense volume of data, with approximately 330 million traces and 8.5 billion spans generated daily. SQL Server users can see significant gains by writing backups to multiple files, with speed improvements ranging from 43 to 67 percent.

Backblaze's hard drive statistics for Q2 2020 show an annualized failure rate (AFR) of just 0.81 percent. This is a notable improvement from 1.07 percent in Q1 2020 and 1.8 percent in Q2 of the previous year. Remarkably, three drive models—the Toshiba 4TB, Seagate 6TB, and HGST 8TB—reported zero failures during the quarter.

Adoption and Market Shifts

The database landscape continues to evolve, with roughly 60 percent of organizations now running a mix of SQL and NoSQL systems. Only 14 percent of companies operate exclusively on NoSQL databases.

The e-commerce sector has seen a dramatic shift. US ecommerce penetration has climbed to 22 percent, up from 17 percent just a few months prior, representing five years of growth compressed into three months. Meanwhile, Wix now hosts 5 percent of the world's websites and attracts 700 million unique visitors each month.

Renewable energy sources now supply 22.2 percent of US electricity, according to data from the first half of 2020. The cost of running a modern tech company is also a topic of interest, with one founder publicizing an itemized monthly bill of $734.38 for a full-stack insurance company operating entirely on serverless infrastructure.

Performance, Security, and Legacy Costs

SpaceX's Starlink beta tests are showing promise, with download speeds ranging from 11 to 60 Mbps and latency between 31ms and 94ms. However, the tech world also faces significant challenges. British companies paid out £200 million in ransomware demands last year, and reports indicate a $1 million plot to hack Tesla failed because the reward was not tempting enough for the attackers.

Internet service providers are experiencing ten times more outages than cloud providers, highlighting a disparity in reliability. The cost of maintaining outdated systems is also staggering, with $720 billion wasted on failed IT replacement efforts. In a more surprising historical note, the oldest known technical system built by humans—a series of fish traps in Australia—was last used in 1915, giving it an operational lifespan of approximately 40,000 years.

On the cutting edge of hardware, a new cooling approach has demonstrated a 50x improvement over typical microchannel methods. By using an optimized 3D structure to extract heat before it propagates, engineers can manage far greater heat flux, potentially allowing many more devices to be integrated onto a single chip and enabling new types of integrated power chips.

Resistance is Fertile

When the Culture’s ships get hacked, their defenses benefit from the same kind of heterogeneity that keeps human populations resilient. As Ian Banks noted, Culture ships write their own individual operating systems as they "grow up," so they are effectively individual species despite appearances — meaning bugs can’t spread. In complex systems, uniformity is vulnerability.

That theme carries over to a number of observations this week on the perils of monoliths — whether in software or in vendor relationships. As Mathias Verraes put it on Twitter, “‘Monolith’ is the word you use when you want to blame the brokenness of your system on its size, instead of on 15 years of bad development practices.” The same could be said of cloud infrastructure: @tmclaughbos was shocked at how often people manage their entire AWS footprint with a single CloudFormation stack with many nested stacks, treating infrastructure as one giant deployable unit.

Avoiding that trap can mean choosing simpler primitives from the start. “I really love this offering and I don’t think it gets enough attention,” wrote HN user redm about direct cloud interconnect. The latency and operational cost of backhauling IP over fiber is significant; with Cloudflare, all traffic already traverses its POPs, so direct connection makes more sense. The abstraction layer often is the problem.

Rethinking the Database

A recurring debate this week was how well the reigning database technologies serve their users. DynamoDB and Cosmos DB were contrasted for their design goals, but a deeper critique emerged from multiple HN comments: relational databases themselves have failed developers by omitting features needed for modern, frequent deployments. jrockway points to the impossible task of zero-downtime schema changes — the correct implementation would support multiple schema versions in one database, with applications declaring which version they speak. No relational database offers that, so armies of migration tooling have emerged to paper over a 1970s design assumption that code doesn’t get deployed multiple times a day.

Transaction isolation levels were another sore spot. Most everyone “knows” transactions are for atomicity, but few run them with the isolation level that guarantees what they assume. And jrockway's critique of transaction semantics got a wry echo from @mountain_ghosts: bank transfers are the canonical database transaction example, yet banks take days to execute them and allow dirty reads the entire time.

The alternative to modeling with SQL may be a fundamentally different data model. In a piece on DynamoDB vs. Cosmos DB, pythonpoole argued that Cosmos is designed to be a drop-in replacement for a traditional relational DB, auto-indexing all properties and supporting SQL queries that work naturally with relational shapes and flexible schemas. DynamoDB is not — its proprietary API is designed around primary keys and query patterns optimized around the partitions, not general access paths. Dynamic and ad hoc queries come with severe restrictions even when secondary indexes exist. Given the same problem, a team must spend substantial time shaping writes to support the reads, which is not the same as being able to ask arbitrary questions. The tradeoff is real, but one engineer offered a strong defense of the simpler model: @nathankpeck reported running 1 TB of data and >10 billion rows in DynamoDB with 1 billion on-demand read units per month, 200 million write units, consistent 2.5ms query latency, and a ~$6k/month bill. “Having such a worry free DB is priceless.” Others went further — @jvthing runs their entire platform on one DynamoDB table and a lot of Go Lambda functions.

The Long Road to Autonomy

In robotics, a change of mind about where autonomy actually leads. The iRobot CEO admitted, “Thinking that autonomy was the destination was where I was just completely wrong.” The new approach brings humans back into the loop: instead of a fully autonomous robot that cleans on its own schedule, the software learns when, where, and how you clean, so the robot adapts to your life. Technology that observes and adjusts to the user’s rhythm, rather than requiring the user to adapt to a fixed automation target, offers the greater practical value. Separately, a lesson for those who design any system around remote code — @ben11kehoe argued that the local-dev and cloud problem is framed backwards: “how can my local dev environment be better and more quickly manifested in the cloud?” is the real question for testing at scale.

Scaling Lessons from Practice

At Amazon, the networks were built in a way that the industry once claimed was impossible. Former AWS engineer Justin Pietsch comments that Amazon built large-scale data center networks with OSPF nearly a decade ago, yet after leaving he keeps hearing it can’t be done. He echoed a critical operational insight: “Running out of capacity on a shared resource is about the worst sin you can perform in a network. And we ran out of capacity a lot.” Justin also learned to think about the balance between magic abstraction and understandability, since infrastructure that is magic is often too good to be true when you are growing fast — it requires deep introspection, understanding of failure modes, and great monitoring.

The operational reality of moving large datasets was on display again in an AWS thread. One team moved 25 TB from one S3 bucket to another and made it a two-day slog with 7 engineers and 4 parallel sessions. But a respondent who had migrated 122 TB in ~5 hours pointed to the proper tool: S3 Batch Operations with the PUT object copy function. It is also worth remembering that the problems that are hardest to detect can hide in plain sight, as Bruce Dawson urged — invest in logs, profiler runs, and attacking compiler warnings — because “if you have compiler warnings you need to rethink your life choices.”

Even with good tooling, Kubernetes was not universally recommended — one engineer reported that across three clients and three flavors of Kubernetes, workload management worked but the upfront and ongoing investment was astonishing enough that they wouldn’t recommend it to anyone else.

Observable System Design

Slack wrote up how it handles distributed tracing by modeling traces as Causal Graphs — a Directed Acyclic Graph of a data structure called a SpanEvent. Instead of relying on flat trace lists and proprietary formats, modeling the trace as a DAG makes it possible to visualize and query in a way that maps to actual system dependencies. The approach clears a path for querying raw trace data directly, which Slack engineers felt was otherwise poorly supported by existing tooling.

Those who had to go deeper into root-cause analysis to address real incidents also shared opinions on what “latency” means. A piece on networks drew out the point that “latency” has been regurgitated until it loses meaning; productive analysis decomposes it into service time, waiting time, and the like. The lesson generalizes across stacks. But tracing only helps if you actually capture the data. On that front, one engineer posted a very Lambda-flavored success: needed to re-process ~600GB of files, so they wrote some code, pointed it at a bucket and hit go. Within two minutes, 4,000 files were downloaded, processed, and reuploaded across 4,000 files.

Observability systems that operate on raw traces can also help stitch together the other two pillars, since metrics, logs, and traces together represent the state of services and their interactions at different granularities and degrees of noise reduction. The piece at Netflix describes these three points of access and the challenge of moving from noise to insight to predict failure modes. One of the key lessons of the work was to combine trace data with higher-level semantics of user impact so engineers can avoid guessing what is relevant while it is still possible to act.

Security Realities

Confidential computing continues to attract attention as a way to relieve the cloud’s trust problem. A concise framing of the technology from the thread: in normal cloud use you effectively trust the provider not to look at or modify your code and data. Confidential computing uses special CPU features to prevent anyone, including the provider, from seeing what’s happening inside a few cores and, in EPYC’s case, encrypt all RAM accesses. The provider’s promise is backed by a CPU-level remote attestation signature keyed to shipped silicon. The computation only emits the signature in a special secure mode, so if you validate it you know the exact state of the code that is running. You can send a key over the network after verifying the image, increasing your threat model to P(cloud provider AND CPU manufacturer are both colluding) ∩ P(CPU manufacturer exploits a latent vulnerability).

Security concerns extend beyond central clouds to supply chains. The Atlantic Council released a report on software supply chain security, calling it an “under-appreciated domain of national security policymaking.” Profiling 115 attacks against the software supply chain over the past decade, the report recommends policy changes that raise the cost of those attacks and reduce the harm from successful ones.

And from the field, a note from an engineer whose production experience had an early onset: an 8-year-old created a Minecraft block that spawned dragons infinitely, ran in crying that all the megabytes would be used up and the computer was about to catch fire.

Costs, Choices, and Market Signals

Capital spending by cloud providers has consequences. The State of the Edge 2020 report from the Linux Foundation projects annual capital expenditures in edge data centers to reach $146 billion by 2028, with 102,000 MW of deployed global power footprint. A consultant projects the edge economy will reach $4.1 trillion by 2030.

Hardware announcements from Alibaba signal new momentum and directions for RISC-V. The company showed its first RISC-V product at Hot Chips: the XT910, a 16-core, 12-nanometer device running between 2.0 GHz and 2.5 GHz, featuring 16-bit instructions. A competitor comparison to Arm’s Cortex-A73 positions the part as one of if not the most powerful RISC-V processor to date.

Vendor behavior attracts granular analysis. A venture capitalist’s claim that AWS reduced prices over 40 times while HP and Dell hadn’t, and was publicly unchallenged, drew a lengthy rebuttal on how AWS’s price-for-performance comparison doesn’t hold if you account for IOPs and memory improvements. The notion that the cloud discount automatically wins on every dimension is a simplification. Similarly, a commenter noted that Snowflake is worth about $70 billion, while Teradata’s traditional data warehousing hardware and software is worth $2.5 billion — a visible shift in how value is distributed to newer compute-centric business models.

Often simpler architectures show up as cheaper in ways that undermine simple narratives. Honeycomb was experimenting with AWS Graviton instances to cut cost while keeping performance, and an engineer pitched the whole stack as follows: “we’re saving them an ARM and a leg.”

People, Process, and the Practice of Engineering

It’s a strong week for career commentary. More than 30 years of system administration experience points to the root of technical debt being management, with leaders not wanting to spend now and unwilling to make a progressive plan for future technology needs. “Few want to invest in the future when it costs now and is working now (unless it’s the stock market),” summarized one commenter.

The tendency to optimize for the wrong features in interviews was also scrutinized. FreeAgent wrote about giving up on algorithm whiteboarding challenges (like Boggle) because solving them didn’t correlate with hiring great engineers. “Of all the things the algorithm-type questions can tell you about a candidate, we cared about slightly different things.” The team no longer does interview puzzles to verify success metrics they don’t promote.

Another HN comment, by a person with three decades of data modeling, worries by day for every “mongo or other non-relational DB where the company jewels (the data) are stored with no documentation, no data model.” The instinct says start with relational for run-of-the-mill apps; only then denormalize or specialize for performance-critical details.

One thing the entire industry can relate to — management and open source maintainership are similar exercises in resource allocation. As the NTP example made clear, a few solitary unpaid or underfunded maintainers continue to carry what proves to be the backbone of the internet’s time synchronization. And then there was the HR version of a classic paradox: why firms screen and hire for what you know elsewhere while imposing non-competes and training bans that restrict what you can share across the same kinds of fences.

Of Bugs and Physics

The development trenches bring small reminders about debugging practice and tooling. “My least favorite part of programming,” one developer deadpanned, is the triumphant end-of-the-hunt moment that gives way to the soul-crushing realization that “this is not THE bug, but just a(nother) bug.” A comp-sci maxim from the network world: BGP is path-vector, and failure cascades can require downtime even in well-designed networks. When a route table is in flux, operators may watch routers oscillate and stall, meaning that even intentional traceroute or latency queries may not see the real path while the network is still converging.

That all connected to another juicy piece of lore from CERN: an error that engineers couldn’t reproduce — but only during certain weeks. The cause was traced to deformation of the lab’s 27km circumference ring by the Moon’s gravitational pull, tiny changes in geometry shifting beam physics just over tolerance. Not every bug is in the binary. Sometimes your system toggles with the tides.

Attention to bandwidth and CPUs also surfaced arguments for pushing back against the cloud-first default. The old question “How do you handle big data?” can now be countered with a simpler deployment story: use cheap compute and storage, manage your own edge. But before rewriting the data stack, it’s wise to remember the networks. As another analyst noted, modern providers spend for the same reasons the largest companies do: backhaul capacity and path convergence — and everyone pays one way or another.

But in the end, every few months there’s a new crisis in the stack. Here’s the lens someone expects us to keep: “I love how bank transfers are the canonical DB transaction example when banks take days to execute them and allow dirty reads the entire time.”

Fault Domains, Failover, and the Cost of Poor Placement

Facebook’s Systems @Scale session on optimal workload placement offers a stark lesson: the physical layout of hardware matters just as much as code. The story begins with an actual snake that shorted a main switchboard in a data center. All servers powered by that board went down—as did the snake. The resulting cascading failures forced all user traffic in that data center to shift to other facilities.

The lost fault domain represented less than 3% of the data center’s server capacity. Yet the disruption was severe because of unfortunate workload placement: that single fault domain contained a large proportion of capacity for some major services, with certain services losing over 50% of their capacity. Fire, lightning, water leaks, and routine maintenance can all cause fault-domain-level failures, and the number of such incidents will grow ninefold as regions expand.

The goal, then, is to lose a fault domain without losing the data center. That requires spreading hardware, services, and data so that any single fault domain holds only a small fraction of each service’s capacity. Historically, hardware was placed wherever space and power were available, and services were placed on whatever hardware existed at the time—leading to the current problem. Optimal placement aims to even out capacity loss, reducing the need for large buffer capacity elsewhere in the region to absorb failed-over traffic.

Basecamp’s Week of Outages

Three Basecamp outages in one week reveal that redundancy is not automatic. A surprise interdependency between network providers took down the redundant link, causing a brief data center disconnect. This triggered a failure in cross-data-center Redis replication: exceeding the maximum replication buffer size caused a catastrophic resync loop that overloaded the primary Redis server, producing very slow responses.

During recovery, a load balancer crashed. A standby picked up operations, but network routers failed to automatically synchronize with the new device. The next day, the primary load balancer crashed again, failover to the secondary proceeded, and then the secondary crashed and failed back to the former primary—repeating the desynchronization issue. Notably, Basecamp 3 and supporting services like login and billing are hosted exclusively in on-premises data centers, not the cloud.

Riot Games’ 128-Tick Server Optimization

To make a free game economically viable, Riot Games needed to radically improve server performance: a server frame took 50ms and needed to drop to under 2ms. The results of their optimization efforts were dramatic, with code optimization, hardware tweaks, and OS tunings all contributing.

  • Measuring in a production-like configuration revealed where time was spent, enabling micro-optimizations.
  • Switching from replicated variables to RPCs offered a 100x to 10,000x performance improvement in many cases.
  • Players in the idle pose during the buy phase helped reduce animation system costs by another 33% per round.
  • Modern Xeon Scalable processors yielded roughly a 30% performance increase even at similar clock speeds.
  • Memory access improved from about 50% NUMA local to 97-99% NUMA local, adding 5% with more consistent performance.
  • Setting the Linux CFS migration cost to 0 allowed immediate scheduler migration to any available core, granting another 4%.
  • Limiting to higher C-States (C0, C1, C1E) stabilized performance and hosted another 1-3% of games.
  • Re-enabling hyperthreading increased performance by 25%.
  • Switching to the tsc clocksource (rdtsc) gave a 1-3% boost.
  • Configuring the Erlang scheduler to spawn 4 threads instead of one per core (72 total) eliminated a mysterious problem overnight.

Storage-Class Memory: No Magic

MongoDB’s experiment replacing an SSD with storage-class memory (SCM) produced disappointing results for realistic workloads. The modest latency advantage of SCM over an equivalent SSD did not translate into performance gains because the storage engine masks those latency differences through batching and caching. SCM shows promise only for latency-sensitive operations that cannot be hidden—such as logging. As one commenter noted, running a disk-optimized storage engine on memory still pays all the overhead of locks and pages designed to work around disk latency. The real solution requires building a new, simpler storage engine that skips that bookkeeping entirely.

Shard Manager at Facebook

Facebook’s generic sharding platform, Shard Manager, manages tens of millions of shards on hundreds of thousands of servers across hundreds of applications. Its adoption stems from key architectural decisions: integrating requires only a small interface with add_shard and drop_shard primitives; applications declare reliability and efficiency requirements through intent-based specification; a generic constrained optimization solver provides versatile load balancing; and full integration with capacity and container management yields an end-to-end solution.

Uber’s Edge Gateway Evolution

Uber’s API gateway needed to handle 800,000 peak requests per second, multiple languages and message formats, hundreds of teams developing in parallel, 50,000 tests, and 1,500 engineers. Their third-generation gateway uses a tiered approach—edge, presentation, product, and domain layers—and an in-house Golang system called Control Flow Framework for complex, stateless business logic orchestration. Moving to Golang improved resource utilization and request-per-core metrics while reducing latency on most APIs.

Key lessons included: stick with a single protocol for mobile apps and internal services; design for horizontal scaling (a single binary would be too large for 1,600 complex APIs); maintain continuous investment as engineers rotate; and make conscious choices about dropping support for long-term sustainability.

Napkin Math and First Principles

Simon Eskildsen of Shopify advocates estimating systems with napkin math—understanding how systems should perform from first principles before building. During tech reviews, a quick calculation can save two weeks of prototype work by showing whether a solution is even in the right ballpark.

The First Principles Gap emerges when a bottom-up model doesn’t match reality. A simple model suggests MySQL should handle around 1,000 transactions per second (an fsync takes ~1ms), yet MySQL really does about 7,000 TPS. The missing knowledge: MySQL batches transactions so multiple commits share a single fsync. If you can’t do the napkin math, the system may be too early to build—a failure mode called “programming through the wall.”

Middleware, Filesystems, and a Weekend Rover

Three recent GitHub projects are worth a look for developers working across different parts of the stack. dbartholomae/lambda-middleware is a ready-made collection of middleware designed for AWS Lambda functions, useful for standard request/response processing tasks. For those dealing with distributed storage, scality/elmerfs takes a different approach: it implements a filesystem that uses Conflict Free Replicated Data Types (CRDTs) on top of AntidoteDB to achieve eventual consistency in an active-active, geo-distributed deployment. On a lighter note, jakkra/Mars-Rover provides the plans and software to build your own rover, a good project for anyone wanting hands-on experience beyond the data center.

Learning from Lectures to Books

There is a solid mix of video and reading material available for those looking to go deeper. The CMU Databases course from Spring 2020 has released its lecture on in-memory databases, a core topic for performance-minded engineers. For a broader curriculum, the Machine Learning University YouTube channel offers free machine learning classes.

On the written side, a reading list for Murat's Distributed Systems Seminar for Fall 2020 is available for anyone wanting a structured syllabus of important papers. A few key papers stand out for their depth and practical relevance:

  • DRAM Error Analysis: Cosmic Rays Don't Strike Twice presents a field study showing that a significant portion of DRAM errors are persistent "hard errors," providing a detailed analysis of their characteristics and implications for system design.
  • Cost of Scalability: The Scalability! But at what COST? paper surveys systems presented at SOSP and OSDI, finding that many either have a very high "COST" (in the hundreds of cores) or perform worse than a single thread in their reported configurations.
  • Remote Memory Prefetching: Effectively Prefetching Remote Memory with Leap describes a technique where fixing data paths yielded single-microsecond latency in 95% of tasks, with the prefetcher active, latency dropped to sub-microsecond for 85% of tasks.

Community Resources on Ethics and Protocols

The Scruples dataset is available for those researching AI and ethics. It contains 625,000 ethical judgments on 32,000 real-life anecdotes, a resource the authors say challenges state-of-the-art language models, though models perform better on simplified moral scenarios.

Two other resources are aimed at practitioners. WebRTC for the Curious is an open-source book written by implementers to explain the protocol without abstracting away the technical details. For those designing distributed systems, Martin Fowler's Patterns of Distributed Systems offers a first set of patterns observed in mainstream open-source implementations, intended to be useful for developers building or maintaining such systems.