Monitoring at Scale: Inside Dropbox's Vortex

Dropbox's monitoring and alerting infrastructure operates across 1,000 machines and continuously evaluates 60,000 alerts. In 2018, the company rebuilt that infrastructure from the ground up. The result—Vortex—caches more than 200,000 queries and, in some cases, delivers query results up to 10,000x faster than the previous system. The redesign also eliminated many of the manual recovery and repair operations that had become a regular part of keeping the old system alive.

This article covers the core design goals behind Vortex, its basic metric types, and the architectural decisions that let it scale horizontally without requiring human intervention.

Why the Old System Had to Go

Dropbox's original server-side monitoring system was built in 2013. Over more than five years, it grew to process over 600 million metrics per minute. The architecture relied on Kafka for queuing ingested metrics and a series of processes that performed aggregation across tags and downsample over time. Data was then stored in three places: in-memory for alert evaluation, on-disk RocksDB for data less than a day old, and HBase for older data.

That design accumulated operational debt. Each data store had its own query capabilities, so some queries only worked in select contexts. Kafka and HBase issues routinely demanded manual recovery, and deployments of the monitoring infrastructure itself could cause ingestion problems and data loss. Individual teams could emit high-cardinality metrics that threatened the entire ingestion pipeline. The query system also had no isolation: expensive queries could take down the frontend, and users retrying stalled queries made the problem worse. Manual sharding of the alerting system meant reconfiguring alerts could cause outages.

The query layer was fragmented into three separate frontends: one that imperfectly emulated Graphite and two others with a custom query language. Alerts, graphs, and programmatic access each required a different query language, each with its own set of limitations and bugs. Patching these problems became a losing battle, pushing the team toward a full reevaluation of the architecture.

Design Goals

  1. Completely horizontally-scalable ingestion. The only architectural limit on the ingestion pipeline should be cost. Scaling should be as simple as adding nodes to the deployment.
  2. Silent deployments. With hundreds of independent services in production, a single on-call engineer cannot babysit the monitoring system through a rollout. Updates must not require human attention.
  3. No single points of failure, no manual partitioning. Machine failures happen daily. The system must recover on its own.
  4. Well-defined ingestion limits. No single service should be able to disrupt others by logging too many metrics.
  5. Multitenant query system. Expensive individual queries must not degrade the system for other users.
  6. Metrics scale with service scale. A metrics setup that works for a three-node service should work for 300 or even 3,000 nodes—without requiring those developing the new service to redesign their monitoring right at the moment of broad rollout.

Vortex Metric Types

Nodes

A node in Vortex is any source of metrics: a physical host, a container, a virtual machine, a network device, and so on.

Counters and Gauges

Counters measure the number of occurrences of an event over a time period. Dropbox's RPC layer (Courier), for example, logs counters for every request, tagged with the RPC service and method names, result status, and client and server discovery names.

Gauges record a value at the current time—for example, the number of actively running workers in a process. A gauge retains its value until a new value is set, the gauge is explicitly cleared, or the reporting process terminates.

Topologies

A topology is a special gauge that identifies which node is logging metrics. Vortex treats these metrics differently in ingestion and queries, allowing a join between a topology and any other metric using the node's ID as the key. A tag on a topology effectively applies to all metrics emitted by that node, which reduces total cardinality—the key mechanism that lets Vortex scale beyond other systems Dropbox evaluated.

Three examples show how topologies are used in practice:

  • The deployment system emits a topology with the service name and Git revision of the running code, so any query joined on this topology can be broken down by service and revision.
  • The node exporter (which measures machine resources like memory, disk, and CPU) emits a node topology identifying the hardware instance type and kernel version, enabling comparisons of service behavior across heterogeneous hardware pools.
  • Databases emit a topology identifying which database and shard each node serves.

Histograms

Histograms record observations and compute summary statistics, providing approximate percentile distributions of observed values.

System design and data flow

Vortex stores metrics in a ring buffer within each local process, and collection is poll-based across two tiers. On every node, a NodeCollector process polls local processes and aggregates their metrics into per-node totals. Per-process statistics are retained, but the default query API exposes only the node-level view. Because NodeCollector polls processes only when it is itself polled, it remains stateless, which keeps operations simple.

A separate service, MetricCollector, polls each NodeCollector every 10 seconds and buffers incoming data for four minutes before flushing to storage. This buffering serves two purposes. First, it allows the data streams to be remuxed so that metrics are grouped by metric name rather than by node, which makes sequential reads of a single metric across all nodes far more efficient at query time. Second, it substantially reduces write load on the Cassandra storage layer.

Tags and availability

Every emitted metric is automatically tagged with two system-defined tags:

  • Node ID, which identifies the emitting node. When the node corresponds to a physical hostname, queries can also filter on node-derived tags such as datacenter, cluster, or rack. This supports troubleshooting of issues that vary by network path, data center, or availability zone.
  • Namespace, which defaults to the name of the task that emitted the metric. This simplifies instrumenting shared libraries (for example, RPC or exception reporting), since it lets operators scope a query to just the core library metrics of a given service.

The MetricCollector service is split into two groups (G0 and G1), and each NodeCollector communicates with one collector from each group. Every collector attempts to flush data to storage if it is not already there, so the system tolerates individual machine failures without service interruption or data loss.

To protect the ingestion pipeline, limits are enforced at several points. Each node is limited in the number of per-node and per-process metrics it can export, and the NodeCollector drops metrics that exceed those limits. The drop algorithm prefers to discard high-cardinality metric names, protecting users who deploy as part of the legacy monolith, and favors processes reporting fewer metrics so that global services are not starved by a misbehaving application.

Query serving and caching

The query subsystem has two components: a partitioned query cache split into two groups for high availability, and a frontend query API that handles routing. Each query cache process keeps both a live cache and a range cache. The live cache streams results directly from the MetricCollectors for low-latency real-time alerting, while the range cache holds aggregated rows read from storage.

Most queries are served from cache and return in milliseconds. For a cold query—one that was never issued or has been evicted—data must be assembled from three sources:

  1. All relevant historical data loaded from storage.
  2. Unflushed data (up to four minutes) held in the MetricCollector buffer, which has not yet reached storage.
  3. An intent to receive streamed updates for the query's metrics from the MetricCollector, after which the query cache remains continuously updated and future queries become cache hits.

Streaming is essential because the dominant query workloads are auto-refreshing dashboards and alerts, which both demand fresh data every few seconds.

Query language and topology joins

Vortex implements its own query language to cleanly handle system topology concepts. It supports grouping by tag values, aggregating away tags, regex filters, time shifting, and Unix-style function chaining. To keep the language concise, we also prewarm cached queries for dashboards that engineers have open, so that investigation is not slowed by cold loads.

The @ operator performs a join on a topology metric. In a joined query, tags from both the topology metric and the main metric are used interchangeably, as if reported together on one metric. Tags originating from the topology are prefixed with @ as well, to avoid ambiguity. Topology examples include node-level metadata (kernel version, hardware class, revision) and the YAPS deployment topology, which lists packages installed on a node. Joining on such topologies allows comparisons of latency by kernel version or of exception rates by the Git revision of the running code during rollouts.

Streaming and storage optimization

Each query cache holds thousands of queries, and receives forwarded metric buckets from the MetricCollectors on behalf of most nodes in the fleet. Incoming data is grouped per node and timestamp, which matters because topology tags are needed to route each query. Tags not referenced in a group-by or filter are aggregated away as the data is placed into each cached query.

Over time, Vortex downsamples historical data from the native 10-second granularity to 3 minutes, then 30 minutes, and finally 6 hours, with each coarser granularity retained for a progressively longer window (six-hour data is kept forever). Query time range selects a sampling rate that returns at most roughly 1,000 data points, keeping query performance roughly constant regardless of the requested time window.

The system has already scaled past the limits of its predecessor. Vortex ingested more than 99.999% of reported buckets across recent months, and automated host replacement has handled hundreds of failed nodes with no human intervention or disruption. Planned improvements include recording rules, selectively exported high-cardinality metrics, clients for desktop and mobile reporting, a new set-cardinality estimator metric type, and additional query language features.