Scaling Numbers: A Readout of the Systems World

As always, the most interesting news in systems and infrastructure is often best told through numbers. Here's a look at the latest metrics and milestones from across the industry.

Gravity, the Cosmos, and the Cloud

The Antikythera Mechanism, an ancient Greek analog computer, is the subject of a compelling detective story about its use as a model of the cosmos. The detective work involves reverse engineering the device to understand its astronomical functions.

Even at the extremes of physics, the law of gravity holds true at a scale of 90 milligrams. This verification comes from a precision test of the gravitational force between small masses.

The Highs and Lows of Infrastructure Economics

  • $7,000 in extra costs was incurred by one business due to AWS's lifecycle rule approach to changing storage classes. This is a further lesson in the importance of UI design, following Citibank's $500 million mistake.
  • Cloudflare's CEO notes a 10,000% markup on bandwidth costs at AWS.
  • A Reddit post highlights the stark reality of ad-based revenue, with $100 being the typical ad revenue earned for 1 million page views.

Milestones in Storage, Compute, and Scale

E-commerce, Security, and Market Data

Under the Hood: Hardware and Performance

  • Backblaze's reported Annualized Failure Rate for its hard drives in 2020 was 0.93%, down 50% from previous years.
  • Leading chip foundries are seeing chip demand running 30% above supply, with 20% year-over-year growth, in what is being dubbed "Chipageddon."
  • An analysis shows that about 10% of top websites track users through CNAME cloaking and subdomain collusion.

Finally, in a sign of things to come, data scientists using technology to grow strawberries produced 196% more by weight than traditional farmers. In a related field, NASA's Perseverance rover landed on Mars within 5 meters of its target, thanks to its Terrain Relative Navigation (TRN) system.

Scaling Tall Tales and Hard-Earned Lessons

This week’s collection of insights from across the industry spans everything from cosmic-scale ambition to the day-to-day realities of managing complex infrastructure. A recurring theme is the tension between raw capability and the economic or architectural constraints that shape what we actually build.

The Cloud Cost Reality Check

A frequent topic was the true cost of cloud computing, especially for high-bandwidth workloads. One CTO of a large gaming community noted that running roughly 700 rented dedicated machines to service up to 100,000 concurrent players pushes about 4PB per month in egress bandwidth. The big cloud providers quoted them an order of magnitude more just in bandwidth costs than their entire fleet’s current rental price. Even with their own ISPs and cross-connects, egress fees to carriers remained prohibitive. The conclusion was that for predictable, steady-state workloads, buying or renting hardware is often cheaper over any timeline longer than a couple of years. Cloud shines for variable workloads or small teams, but at a certain size, a dedicated sysadmin can pay for their own salary in cloud savings.

Another operator echoed this sentiment, praising providers like OVH for making powerful hardware accessible at a fraction of the cost of hyperscalers. They pointed out that a machine that might cost $1,000 a month on one platform could be had for around $100 on another, which was the difference between getting a project off the ground and not doing it at all. This was in stark contrast to some of the reactions to the OVH outage, with one observer noting the difference in perspective between the US and European markets, where OVH and Hetzner are mainstream options. A different story highlighted the flip side: a team that moved to Kubernetes on Azure ended up with significantly more DevOps staff and higher VM costs than when they were on a simpler platform, a warning that the operational complexity of the new tooling can outweigh its benefits.

The discussion on cost also touched on clever tactics and inherent inefficiencies. One suggestion was to spin up a separate account to purchase Reserved Instances or Savings Plans, as they apply to all accounts but the support cost percentage is per-account. Another pointed out the "dirty secret" of cloud—that much of it is overprovisioned and idle. The rising monthly bills for services nobody knows are being used often sour the initial enthusiasm for subscription pricing.

Managing Scale and Latency at the Edge

The challenge of moving data was a central concern. One observation used data transfer figures to deduce an architecture: with roughly 500GB out, 620GB in, and a whopping 10TB transferred between Availability Zones, the conclusion was that the workload was likely running on Kubernetes (specifically EKS), which was generating those costly cross-AZ charges. The question of whether newer versions respect topology was raised, with the implication that this one didn't.

For real-time applications, the choice of transport protocol is critical. One developer described how using TCP for a live map visualization caused the display to lag many seconds behind real-time in areas with spotty cellular reception, as the network stack retransmitted lost packets. Once behind, it never caught up and sometimes got worse. It was far better to have the map pointer jump forward than to permanently lag behind what was happening in the real world, preferring UDP-based approaches.

This idea of latency being the primary design constraint was reinforced by a broader observation on hardware trends. While memory, storage, and CPU continue to get faster and cheaper, the fundamental bottleneck has become the time it takes to fetch data. Waiting is the new problem, making latency the central design point for modern systems rather than raw capacity. This extends to the software we build; one engineer lamented that cloud egress pricing is functionally a "digital moat" designed to keep data on a platform.

Kubernetes: Ecosystem over Core?

The debate around Kubernetes continued, with one viewpoint suggesting its main advantage is no longer Kubernetes itself, but the vast ecosystem of tools and vendors surrounding it. One team shared a positive migration story, calling Kubernetes "the biggest quality-of-life improvement I've experienced in my career," citing reliable rolling deployments, seamless horizontal scale-out, and GitOps as the key wins. However, another detailed a disastrous migration that required far more staff and money, concluding with a blunt "FAIL."

This operational divide was also explored in the context of developers and serverless. An engineer from an ops background reflected on a conversation where a developer longed for the simplicity of just adding routes in a Spring Boot app. The ops person's reaction was relief at no longer orchestrating load balancers and proxies. The insight was that many developers are being exposed to this infrastructure layer for the first time, and they don't have the muscle memory of the pain that serverless eliminates.

Architectural Lessons from Antiquity and Mars

On the topic of architecture, Werner Vogels of AWS shared a core tenet for S3: "don't lock yourself into your architecture, because two or three orders of magnitude of scale and you will have to rethink it." The idea of an evolvable architecture was revolutionary. A key difference between Amazon Retail and AWS, he noted, is that retail can experiment and switch features off if customers don't like it, but in AWS, customers build their businesses on top of you—you can't just pull the plug on something because you think something else is better.

A look at NASA's Ingenuity helicopter on Mars highlighted a unique architectural trade-off. The space-hardened processors weren't fast enough for real-time sensor fusion and flight control, forcing the team to use a faster, more susceptible Snapdragon processor. Their solution for potential radiation-induced flips was to hold two copies of memory and double-check operations. If a difference is detected, the system simply reboots. The craft might start to fall, but it can complete a full reboot in a few hundred milliseconds and continue flying—a stark reminder that even the most advanced systems sometimes need to be turned off and on again.

The Human and Strategic Elements

Beyond technical details, there were reflections on processes and people. One insider claimed that Google has all the building blocks for great products, which can be shipped in under six months if you have an executing team and the political will, but they noted that politics and consensus-building are where the real roadblocks lie. Trying to make high-level decisions with ten stakeholders across different orgs is a serious exercise in patience.

A study on driver assistance systems found that drivers inexperienced with Level 2 automation were more than twice as likely to not remember a giant teddy bear speeding past them compared to other groups. Also circulating was a quote on the value of experience, echoing sentiments that a 50-year-old developer feels they've continued to improve at the craft, which is supposedly supported by research. Finally, a poignant tribute highlighted the long tail of influence, as one person remembered his 85-year-old grandmother who taught him BASIC on an Atari and was the first person to debug his mistakes, a reminder that engineering skills are often passed down across generations.

Pearl Harbor, Passwords, and Platform Blame Games

Cybersecurity’s favorite scapegoat narrative continues. Following the SolarWinds breach, the company’s former CEO has pointed a finger at an intern for the now-infamous solarwinds123 password leak. The story feels less like a technical post-mortem and more like a case study in accountability diffusion. While the specifics of the attack’s origin remain a point of contention, the anecdote underscores a persistent theme: complex system failures often end with a search for a single, human point of failure, even when the technical architecture was the real enabler.

Stack Overflow’s Heresy: Best Practices as a Starting Point, Not a Destination

Stack Overflow, the site that codifies best practices for millions, openly admits to deviating from them in its own production codebase. Their approach, detailed in a recent engineering post, prioritizes raw performance and a scale-up architecture over the modern dogma of scale-out and microservices. The team leans heavily on static methods and fields to minimize allocations and reduce garbage collection stalls, deliberately trading testability for speed in hot paths. They build with a mix of memoization and multi-level caching (in-process and Redis) to support a high-performance micro-ORM, Dapper, which translates SQL to objects efficiently. This is a conscious choice; they acknowledge that this design is harder to unit test. The takeaway isn’t that best practices are wrong, but that they are a default state to be evaluated, not a mandated rulebook when performance is the primary metric.

From Pixels to Physical: LaMetric, Dials, and the API of Everything

Troy Hunt is at it again, using Cloudflare Workers and KV as a creative glue layer to build a custom LaMetric app that displays his Have I Been Pwned subscriber count on a physical device in his office. This is a prime example of bridging the digital and physical worlds, making real-time data tangible without significant infrastructure overhead.

This theme extends to the industrial sector. A fascinating project on GitHub, jomjol/AI-on-the-edge-device, demonstrates how to convert a traditional, analog water meter into a web API. By using a cheap ESP32 camera and TinyML, the project can visually interpret dials and numbers, effectively modernizing "dumb" devices that are ubiquitous in the real world. As the author notes, waiting for a full replacement cycle of legacy hardware could take decades; an incremental approach of adding a smart camera to an existing dial is a far more immediate path to remote monitoring and data analysis.

The Serverless Cost-Benefit Analysis

A deep dive into building a high-scale chat server on Google Cloud Run reveals both the promise and the price of serverless. The architecture shows how to use WebSockets, Redis PubSub for state synchronization, and stateless containers to handle up to 250,000 concurrent connections. The system scales automatically based on load, spinning up and down containers as needed. This scalability, however, comes at a significant cost.

The burn rate is starkly illustrated: running 1,000 instances to serve 250K clients 24/7 would cost an estimated $87/hour, or $62.6K/month. The commentary surrounding the post, however, defends the model. One developer notes that for workloads with low baseline traffic but massive, unpredictable surges, it’s an ideal fit. They moved a CPU-intensive worker service from Heroku, where they were forced to pay ~$3k a month for peak capacity, to Cloud Run, where they now pay ~$50 a month, letting the platform handle the scaling autonomously. This highlights a crucial nuance: serverless isn’t about being cheap; it’s about paying for what you use and trading operational complexity for financial flexibility. The argument is that nobody should run a serverless stack for a constant, high-volume load — that’s a misapplication of the model.

Performance Wars: Wix’s Paradigm Shift to Server-Side Rendering

Wix has managed a massive infrastructure overhaul that led to a threefold increase in the percentage of sites passing all Core Web Vitals (CWV) metrics. A key part of the evolution was a dramatic pivot back to server-side rendering (SSR) from client-side rendering (CSR) to improve performance on slower devices and connections. Yet, they didn’t stop there. To mitigate the costs of generating unique HTML on the fly for every request, they engineered a hydration process using a slim JSON payload delivered to a new endpoint on each page load. Combined with an array of optimizations, this strategy shows the cyclical nature of web architecture.

The full list of their tactical wins shows a deep, multi-pronged approach:

  • Serving ~13% of HTML requests directly from browser cache, saving bandwidth and improving repeat-view load times.
  • Enabling HTTP/2 on all domains, which cuts down connection overhead.
  • Using brotli compression instead of gzip, leading to a 21–25% reduction in median file transfer size.
  • Integrating with their DNS provider to automatically select the best-performing CDN for each client’s network and origin.
  • Actively working to serve entire sites directly from CDN locations to improve global response times.

Data Model Circles and Queue Evolution

The eternal struggle with data modeling is on display in a look at how Tumblr stores post content. The journey often starts with the rigidity of a relational model, adding new attributes is painful, prompting a move to a flexible, schemaless JSON-like format. But that newfound flexibility creates querying difficulties, which then necessitates a move back to a relational model to derive value from the data. It’s an acknowledged "data model circle of life." This same pattern is echoed in Uber’s evolution of its Schemaless system into a distributed SQL database and Pinterest’s Manas retrieval architecture for hierarchical documents.

Meanwhile, in the world of message queues, a new scalable approach comes from Facebook with FOQS. It addresses the problem of scaling a distributed priority queue, a common bottleneck. Alongside this, a comprehensive guide to message queues serves as a solid overview of the broader topic, covering concepts beyond the basics.

Evolving Infrastructures: Recipes from Etsy, Netflix, and Mux

Several engineering case studies emerged this week, illustrating how top-tier companies evolve their infrastructure to meet specific challenges.

Etsy’s Prepping for the Holiday Slush

Etsy’s preparation for peak holiday traffic pivots on a few key principles. Internally, they enforce a “Slush” period — a few weeks before the rush where they limit deployments that could disrupt sellers or infrastructure. Their planning is highly communicative; they share upper-bound estimates of resource usage with Google Cloud many weeks in advance, ensuring the right machine types are available. To validate their readiness, they engage in multi-team "game days" for macro load testing. A recent test exposed bottlenecks in components like Memcache clusters and StatsD relays before the real event, confirming that their scaling tools work under pressure in a controlled environment.

Netflix Cosmos and the Power of "Always-on" Workflows

Netflix offers a glimpse into its Cosmos Platform, the internal system for media processing. Their architecture moves away from a monolithic distributed application to a “platform plus applications” model. The core paradigm is “microservices that trigger workflows that orchestrate serverless functions.” The system communicates via Timestone, a high-scale, low-latency priority queue. A key piece is Plato, a forward-chaining rule engine that allows workflows to be "always on." For instance, as better encoding algorithms are developed, these rules automatically trigger a re-encoding of existing videos without manual intervention, a distinct advantage over a procedural workflow engine. When a new title arrives, it can trigger a massive cascade, launching a single workflow that results in hundreds of requests to other services and thousands of serverless function invocations for encoding, subtitling, and packaging.

Mux’s Dual-Cloud, Phoenix-Powered Video Stack

Mux, which handles billions of video views a month, runs its entire operation with a ~45-person team. The public API is a standard Phoenix app (Elixir), running on AWS, while the core video infrastructure is hosted on Google Cloud. The API and billing are handled in Elixir with PostgreSQL, but for heavy analytics, they use ClickHouse, which lets them store billions of rows and query the data quickly. They rely on Kubernetes and Docker for orchestration and use a mixture of manage services — Buildkite for CI/CD, SendGrid for email, Sentry for errors, and Stripe for all payments, including a custom implementation of metered billing.

The Realities of Benchmarks and Scripts

A perspective from Suade argues that most "super-fast" Python benchmark results are misleading. These tests usually are hello-worlds or echo servers that rely on hand-crafted C code. Once you introduce real Python logic, performance plummets. The site claims that the real bottlenecks will be schema design, database choice, and architecture, not the selection of a fully async framework, which can lead to "decision paralysis." The recommendation is that using PyPy and an async Gunicorn worker could be more effective.

This same tension between ideal and practical is at the heart of the discussion on serverless. A video called "Serverless Doesn't Make Sense" explores the pitfalls, such as cold starts and latency, especially with image resizing on synchronous requests. The counter-argument, made by Forrest Brazeal, reframes the debate entirely. His "3 Counter-Intuitive Reasons Why Serverless is the Awesomest" suggests that:

  1. Serverless Violates the Second Law of Thermodynamics because services passively get better over time (e.g., DynamoDB introducing on-demand billing that saved someone $120k/year). You get free, automated improvement.
  2. Serverless is Expensive, and that’s a good thing. It forces you to use managed services, trading dollars for engineering time, which allows you to focus on your core business. These are opportunities to efficiently delegate work.
  3. Serverless Lock-in is Good — you’re always locked in with a provider, and it’s better to be locked into something good.

Brazeal’s talk positions serverless as a manifestation of the API economy, aiming to let you "own less; build more."

Let’s Encrypt’s Impactful Hardware Upgrade

Let’s Encrypt is preparing for the issuance of 200 million certificates in a single day. To achieve this, they’re overhauling their hardware vastly. Key upgrades include:

  • New Database Servers: Rack of dual AMD EPYC 7542 CPUs with 64 physical cores and 2TB of RAM. Critically, the new CPU architecture provides 128 PCIe4 lanes each, letting them pack in 24 NVME drives for massive I/O and use ZFS for data protection.
  • A 25G Fiber Network: An upgrade that was not much more expensive than 10G, providing massive bandwidth improvements.
  • New HSMs (Hardware Security Modules): Generously donated by Thales, offering about 10x the performance with roughly 10,000 signing operations per second per device, 20,000 between the pair. This means they can perform 864,000,000 signing operations in 24 hours from a single data center.

Cutting Through the Complexity

The common thread across these stories is a spectrum of approaches—from embracing serverless’s convenience and cost to meticulously engineering around best practices for peak performance. Yet, even high-profile companies sometimes find simpler solutions are best. One article highlights how a cloud provider architect solved a Lambda cold-start performance problem (~23 seconds) not by tweaking the configuration, but by processing data in a scheduled function once a minute and then serving it from S3 behind CloudFront — entirely avoiding the need for a real-time compute trigger.

Further, an analysis of Adrian Cockcroft’s architecture predictions notes that AWS is systematically addressing objections to a serverless-first world with improvements like container support and more powerful Lambda functions. While the talk is part pitch deck, it highlights the trend with the statistic that 50% of all new AWS services are built on top of Lambda. However, some argue that these architectural trends, particularly those around huge memory machines and GPU databases, have historically been beaten by commodity approaches. For good or ill, the commodity cloud servers often win out.

The Intricate Dance of Stream Processing and Reporting

At Reddit, reporting on events flows through a multi-stage pipeline. A client fires an event to a server, which is placed onto Kafka. A Spark job then validates these events and writes them to S3 as parquet files. Another Spark job transforms those files before ingesting them into Druid, a columnar database designed for high-volume event ingestion. The reporting service responds to user interface requests by querying Druid. Thanks to a simple schema, complex use cases like an advertiser generating a six-month report for all their ads can be easily resolved with a single SQL COUNT query with a GROUP BY clause on date and ad.

Wayfair’s engineering team describes a simpler optimization journey. By shifting focus from averages to percentiles for performance tracking, they found a series of significant wins in their request-processing pipeline. Raising the default log level from DEBUG to WARN bought a 3ms improvement, and removing DataDog APM saved another 2-3ms. However, their biggest win was in the dependency resolution layer: by changing stateless class registration to single instances, they saw a 15-20ms improvement, bringing their p50 from 27ms to 6ms and their p99 below their previous p50.

Finally, Etsy’s story demonstrates the value of proactive, cross-team testing to ensure a complex, distributed system is ready for the biggest shopping day of the year, where sharded MySQL peaked at 1.5 million queries per second and Memcache throughput peaked at 20M requests per second.

Biological Systems as a Lesson in Coordination

A brief but thought-provoking excerpt from a podcast with Andrew Huberman frames an intriguing systems design question using the human body. How does your body coordinate its entire circadian rhythm? The master clock is controlled by body temperature. Tissues measure temperature locally to make global decisions in a timely manner. This decentralized approach, using a common physical signal for coordination, is contrasted with a message-passing approach in distributed systems. The author suggests that relying on a localized, continuous signal like temperature is more robust and reliable for timing-sensitive, system-wide coordination than attempting to pass explicit messages across all components.

Beyond the Code: Platforms and Ecosystems in the Real World

In the music world, an "Instagram for DJs" was built using Azure Functions and the FaunaDB database. The developer from a .NET background chose Azure for its easy configuration and integrated HTTP REST layer (no separate API Gateway needed). They skipped CosmosDB due to a lack of cross-collection transactions; FaunaDB handled everything within a single transaction and offered a true serverless, consumption-based pricing plan, along with a C# client that can serialize classes directly into the database. This pairing enabled a cheap development cycle and an infinitely scalable data layer.

On the topic of security scaling, an article looks at turning an internal API into a key driver for a physical dashboard. It showcases the sheer creativity possible when using simple tools like Cloudflare Workers for serverless compute and its integrated key-value store (KV) for caching or state management.

And to close the loop on an old lesson: “How Etsy Prepared for Historic Volumes of Holiday Traffic” brings it all back to readiness through communication. They not only have a code freeze but they also share upper-bound infrastructure projections with their cloud provider weeks in advance to ensure capacity and location readiness, and they use game days to preemptively hunt down bottlenecks.

Getting Quoted: The Search for the Single Point of Blame

The week’s narrative arc, however, may come from a slightly tongue-in-cheek editorial note drawing parallels between historical battles and modern security. The piece lambasts the military's handling of Pearl Harbor, blaming a low-ranked private for a poorly thought-out defense plan, while noting no higher-ups could be held responsible. This satirical framing becomes a lens for the real-world story of SolarWinds’ security breach. Instead of examining the systemic weaknesses (such as weak password policy and poor architecture), the company’s former leadership has decided to place the onus on an intern. The editorial suggests a far too common theme: pointing fingers downward lets entire organizational and technical structures avoid accountability. The irony is not lost when a security failure is attributed to a low-level human error, after the fact.

Low-Level Failure Analysis in SSD Data Centers

A new study from Alibaba, based on nearly one million SSDs across 11 drive models, examines correlated failures in production data centers using SMART logs, trouble tickets, physical locations, and application data. The findings challenge assumptions that underpin typical redundancy schemes.

  • Intra-node and intra-rack failures account for 12.9% and 18.3% of SSD failures respectively, and failure group sizes can exceed what typical redundancy protection tolerates.
  • The probability of an additional failure within a group increases with the number of existing failures in that group.
  • Failure distribution varies by drive model; mixing too many drives of the same model in one node or rack increases correlated failure percentages. Annual failure rate and environmental factors like temperature also shift the balance.
  • Higher-density MLC SSDs show lower percentages of correlated failures, though the relative share of intra-node and intra-rack failures grows with drive age.
  • SMART attributes correlate weakly with correlated failures, and write-heavy workloads raise overall failure rates without being the sole determinant of AFR.
  • In this dataset, erasure coding outperforms replication, and redundancy schemes designed for independent failures often fail under the observed correlated failure patterns.

New Runtimes for Distributed Applications

Several open-source projects aim to simplify building scalable services. suborbital/atmo abstracts away infrastructure concerns so developers can assemble a server application without managing networking or scaling. One proposal notes that combining Atmo with a content-addressed network like IPFS could create a global serverless platform: if peers share a common runtime and every function and data object has a unique network identity, bundles could execute wherever resources exist, effectively making the network its own CDN.

CondensationDB/Condensation is a zero-trust distributed database that draws on blockchain, email, and git versioning for its architecture. It focuses on data ownership and security, with synchronization as a primary strength. AbstractMachinesLab/lam offers an actor-model VM for native and WebAssembly code, inspired by Erlang and Lua and compatible with the Erlang VM. bastion-rs/bastion provides a fault-tolerant runtime with lightweight, dispatch-oriented processes, guaranteeing at-most-once message delivery while using all system resources efficiently.

Also of note: the Apollo 10 lunar module flight software (virtualagc/virtualagc, Luminary 69 Rev 2) has resurfaced after being lost; it is now available in the project repository.

Papers and Resources

  • Foundational distributed systems papers — a compilation focusing on core distributed systems topics, deliberately excluding networking, security, distributed ledgers, verification, and distributed transactions.
  • Cloud gaming network analysis — a study of Stadia, GeForce Now, and PS Now finds that the first two use RTP (Stadia via standard WebRTC APIs) and can consume up to 45 Mbit/s depending on network and quality. PS Now uses only undocumented protocols and stays under 13 Mbit/s.
  • HHVM Jump-Start — Facebook describes a technique that reduces server capacity loss during warmup by 54.9% while improving steady-state performance by 5.4% on the production website.
  • FirePlace — Amazon research on placing Firecracker microVMs in AWS Lambda shows that microVM usage is spiky and short-lived, making forecasts ineffective. The proposed FirePlace algorithm, based on hindsight imitation, improves on baseline placement algorithms by 10% in production traffic traces of 100K microVMs.
  • Silent data corruptions at scale — Facebook reports that after running silent error test scenarios across hundreds of thousands of machines for over 18 months, hundreds of CPUs were flagged. The findings indicate SDCs are a systemic, cross-generational problem requiring hardware resiliency, production detection, and fault-tolerant software design rather than any single mitigation.
  • Minesweeper — a scalable root cause analysis technique that mines patterns from telemetry alongside bug and crash reports. Evaluated on Facebook's apps, it processes tens of thousands of reports in under three minutes and identifies regression root causes with over 85% accuracy.
  • Apple Platform Security — the official guide documents implementation details for security technology across Apple platforms, intended to help organizations align their policies with these features.
  • Algorithms — a free electronic textbook with accompanying lecture notes from the University of Illinois, Urbana-Champaign.
  • Cloud Native Transformation — an O'Reilly book covering the technology, culture, and strategy of moving organizations toward cloud-native operations.
  • Reading and Writing the Morphogenetic Code — a Tufts whitepaper on the information structures by which cellular networks represent target morphology and compute cell activities over time.