A real-time window into the network
Dropbox’s production network spans multiple data centers and a legacy AWS footprint, with each segment managed by separate teams using separate tooling. Before the rollout of the Magic Pocket infrastructure, the network team lacked a unified, real-time view of traffic flows and the interactions between its parts. Much of the log data from routers is semi-structured or unstructured, which does not map cleanly to relational models. While HBase and Hive can store large volumes of logs, they do not readily support real-time indexing.
The need for a real-time view became acute during the migration of 500 petabytes of data—more than 4 exabits of network traffic—into the exabyte-scale Magic Pocket environment in under six months. That pace required the network to sustain high throughput over an extended period without failure. Monitoring the traffic profile on the backbone between AWS and the datacenter was one thing; mapping NetFlow data back to internal infrastructure made it possible to pinpoint the root cause of anomalies faster. NetFlash was built to address both the scale and the latency requirements.
NetFlash collects NetFlow datagrams from production routers, augments them with relevant infrastructure metadata, and pushes them into Elasticsearch for indexing. Kibana provides the web UI for querying and dashboarding in real time. For example, when multiple clusters show a sudden, brief drop in traffic at 9 a.m.—with outbound traffic to AWS dipping in parallel—the system allows engineers to drill down to the specific team that made the change responsible for the impact. Previously, diagnosing such events from stored logs took hours; with NetFlash, it takes a quick message to the right team.
System components and definitions
NetFlow is a Cisco-proposed, industry-standard datagram format that carries source and destination IP addresses, source and destination ports, IP protocol, and the next-hop IP address. Collecting NetFlow from all routers yields a concise view of traffic origin, destination, and path.
Elasticsearch is an open-source distributed search engine, used widely in enterprise deployments ranging from Azure Search to The New York Times archive. In high-volume environments, ES runs on multiple clusters to enable near-real-time querying.
Kibana is an open-source visualization tool that renders dashboards from content indexed in an Elasticsearch cluster.
Why Elasticsearch for NetFlow
Dropbox generates roughly 260 billion NetFlow records per day—terabytes of aggregated flow data. The original pipeline stored these logs in Hive/Hadoop clusters for permanent retention, analyzed via HiveSQL. That pipeline serves long-term and historical analysis well, but new data typically takes anywhere from two to twelve hours to become queryable, ruling out real-time monitoring.
NetFlow records are sent from production backbone, edge, and datacenter routers to geographically distributed collector pairs, with the second copy serving as failover. From the collectors, records are written to two destinations: the HDFS/Hive clusters for durable storage and a Kafka/Elasticsearch cluster providing near-real-time indexing.
The collector
Each datacenter runs about six collectors, each receiving billions of log entries per day. The collectors are written in Go and take advantage of multi-core processors and large amounts of RAM; increasing the I/O buffer size was a key lever in reducing packet loss.
Raw NetFlow data requires enrichment to be useful. Since IP address lookups at the per-record rate would overwhelm the production database, collectors maintain a local cache mapping IPs to machine, router, load balancer, and GeoIP attributes. Processors annotate each datagram with these attributes—server names, project, owner, roles, locations—before sending the enriched record onward to both the Hive and Elasticsearch pipelines.
Kafka as a buffer
Testing showed that the Elasticsearch cluster’s throughput degraded when the cluster was unhealthy. A Kafka cluster was inserted as a buffer between collectors and ES to add fault tolerance. Kafka increases end-to-end latency only marginally (typically under one second) while ensuring that undelivered messages persist for a defined period, allowing a recovered ES cluster to catch up on unprocessed messages.
LogHub
Transferring data from Kafka to Elasticsearch required a dedicated mechanism. LogHub serves that role as a general-purpose pipeline component with a built-in Kafka consumer, an ES encoder, and an ES connection pool, making it straightforward to map new Kafka topics into the ES index.
Elasticsearch cluster tuning
A prototype with one master and three data-node ES nodes crashed almost immediately under the data load. Production deployment required several changes:
- Scale to three master nodes and ten data nodes.
- Use bulk requests to ES—each containing more than thirty logs—to reduce I/O overhead.
- Keep only the last eight days indexed, deleting expired shards daily.
- Register ES data nodes in a NetFlash Zookeeper quorum and use a connection pool to evenly distribute requests across nodes.
- Tune indexing parameters and keep a single copy of indexed data rather than three.
- Downsample logs by a factor of four at the collectors, while retaining full fidelity in the permanent store.
Stability lessons
Early on, the ES cluster proved fragile. ES masters could become overloaded trying to re-allocate shards after a data-node failure, causing other nodes to lose their master connections—triggering a re-allocation loop that could crash the whole cluster.
The fix was overallocating data nodes: deployment now runs more than twice as many data nodes as there are shards. With 10 shards, 22 nodes are allocated—two copies of each shard and two idle nodes. On failure, a duplicate node seamlessly takes over, the replacement shard is copied from the new primary to an idle node, and the failed node is ejected without impacting the others.
After upgrading ES to 2.0.0, the team plans to use HDFS as an index backup to weather turbulent cluster states and to extend search beyond the current eight-day rolling window.
Adoption and workflow
NetFlash required close collaboration between the engineering team and the network engineers who use it. Kibana’s query language differs from the SQL-like syntax familiar to network specialists. To ease adoption, the neteng team pre-saved the most useful queries, graphs, and dashboards for colleagues and re-indexed tokens so that the displayed frequency is reported in bits per second over each five-minute interval, aligning with existing network terminology.
NetFlash in Practice
The payoff is immediate access to network telemetry. Dashboards built on the pipeline render the traffic matrix at both the cluster and metropolitan scale. The backbone-flow graphs shown earlier in this series track the block storage system, one of Dropbox's heaviest traffic generators.
Beyond day-to-day monitoring, the NetFlow stream informs capacity decisions. The physical-infrastructure team watches cross-region data movement to decide where new hardware will have the most impact, and to flag underserved markets for efficient expansion planning. That bird's-eye view previously required painstaking manual data collection; now it is available on demand. Real-time speed is not strictly necessary for capacity planning, but it lets production engineers answer questions immediately rather than waiting on batch reports. Traffic spike patterns also help identify candidate networks for new peering relationships.
Reuse Beyond Networking
The heavy lifting of the data migration is complete, but NetFlash continues to support high-volume projects. The pipeline has been repurposed to monitor other operational streams, including marketing analytics logs, smart sensor data, and production operation logs. The same collection, enrichment, search, and analysis machinery now applies to any log data the company needs to observe in real time.
The lesson from the effort is that the scaling problems were the difficult part. With those solved, adapting the platform to new data types is straightforward — and a durable advantage of building tooling at scale.
