Building for scale: the Dropbox backbone
Dropbox’s edge network improved performance for users worldwide, but it was only half the story. The other half was a global backbone connecting North American data centers to each other and to edge nodes around the world. As the company migrated hundreds of petabytes of customer data into its own data centers, the network had to grow quickly to keep up with both that migration and a growing customer base.
In early 2015, Dropbox launched a network expansion initiative to accommodate 10X scale, provide 99.999% reliability, and improve performance. The plan involved overhauling routing architecture and deploying new technologies including quality of service (QoS), Multiprotocol Label Switching (MPLS), and IPv6.
Routing architecture overhaul
The existing routing architecture was built on Open Shortest Path First (OSPF) as the Interior Gateway Protocol (IGP), with route reflectors (RR) for Interior Border Gateway Protocol (iBGP). That setup created problems for the planned expansion.
The biggest issue with OSPF was IPv6 rollout complexity. OSPFv2 only supports IPv4, and OSPFv3 lacked full multi-address-family support from vendors at the time. Running two versions of OSPF to handle both IPv4 and IPv6 was operationally complex. The team replaced OSPF with IS-IS, a protocol-agnostic architecture that runs at OSI layer 2 and supports all address types. IS-IS uses Type Length Value (TLV) structures to carry information in Link State Packets, which makes it extensible for future protocols. The migration from OSPF to IS-IS across the entire backbone completed in Q2 2015.
The iBGP design also needed rethinking. The initial single-hierarchy route-reflector model limited path diversity: RRs advertise a single best path to peers, so peers see only one path per prefix. That caused traffic for a prefix to concentrate on one next-hop instead of spreading across equal-cost paths, resulting in unequal load balancing. The team tested Add-Path to announce multiple paths, but it was still new and buggy in vendor implementations.
Two alternative designs were considered:
- Full mesh iBGP across all routers. This solves path diversity but doesn't scale. As the network grows, millions of routes in the control plane can cause memory issues, and route churn can impact CPU and RIB/FIB memory.
- Full mesh iBGP within regions, route reflectors across regions. This reduces the number of routers in full mesh, but RR limitations persist.
The final design was a hybrid: full mesh iBGP across all routers, but with regionalization into smaller groups with different routing policies. Transit-provider routes, which constitute the bulk of traffic, stay confined to their region of origin. All other peering routes and internal traffic are announced across regions. This eliminates RR limitations while avoiding route-scaling issues from full mesh.
MPLS-TE for dynamic traffic handling
MPLS with RSVP-TE was rolled out in early 2015 to handle failures and demand spikes without manual intervention. When sufficient bandwidth exists, traffic follows the shortest path via Label Switch Paths (LSPs) between source and destination. Dropbox deployed multiple LSPs at different priorities: user traffic takes high-priority paths; internal traffic takes low-priority ones.
When demand rises or capacity drops due to an outage, RSVP-TE moves LSPs to alternate higher-metric paths with sufficient bandwidth. Because multiple LSP priorities exist, RSVP-TE keeps user traffic on the shortest route and first moves less critical internal traffic to longer paths. This yields redundancy and efficient utilization without over-provisioning.
Quality of Service program
Dropbox’s network carries both latency-sensitive user traffic and high-volume batch traffic from data migrations and server provisioning. A QoS program launched in 2015 classified traffic into four categories, each mapped to a queue:
Network_Control: Routing protocol hellos and keepalives, the highest priority since losing these packets jeopardizes network operation.Premium: All traffic serving end users, treated with high priority.Default: Non-user-impacting traffic important for internal service communication.Best_Effort: Non-critical traffic, first to drop under congestion and re-sendable later.
Application owners marked their services on host machines based on priority. Bandwidth is provisioned to support all traffic types at all times, but QoS protects critical services during unexpected network failures by prioritizing Premium traffic.
Re-architecting for 10X scale
In 2016, the network was re-architected with new hardware and role separation. The backbone consists of three router types:
- Data center routers (DR): connect data centers to the backbone.
- Backbone routers (BB): terminate long-haul circuits and aggregate DRs in regions with data centers.
- Peering routers (PR): connect Dropbox to external BGP peers for Internet connectivity.
Previously, all three roles ran on the same hardware platform in a single network layer handling both user traffic (between Dropbox and the open Internet) and data center traffic (between Dropbox data centers). As scale increased, horizontal growth became expensive and operationally complex, so the design was rethought.
Two-tier architecture
The new architecture splits traffic into two independent network domains. A new data center (DC) tier carries data center traffic, connected via full mesh MPLS (RSVP) LSPs and built on dense backbone routers scaling to multi-terabit capacity. The original DR tier transports user traffic from Dropbox to the Internet. Each tier has its own BGP and MPLS LSP mesh but shares the same backbone (BB) routers and physical transport.
Data center traffic—internal services communicating or copying data between data centers—is about twice the volume of user traffic. The two profiles have different characteristics. User traffic always moves from DRs to points of presence and is premium. Splitting internal traffic onto its own tier enables traffic profiles and topologies unique to each type.
Optical infrastructure
To support growth and maintain service level agreements, Dropbox invested in dark fiber connecting data centers to PoPs. Leasing dark fiber and running its own optical systems allows capacity to be added faster than purchasing bandwidth or leased line capacity from optical transport vendors. The company deployed the latest bleeding edge optical gear for quick, easy scaling.
Scaling the backbone to 100G
By 2016, the pattern of adding more 10G circuits to handle growth was becoming unwieldy. Dropbox had multiple link aggregation bundles (LAGs), each containing more than ten individual 10G links. Provisioning, managing, and troubleshooting those bundles added real operational overhead. The team decided to simplify by consolidating multiple 10G circuits into single 100G links.
That simplification depended on a new generation of backbone (BB) routers. Dropbox spent about eight months in 2016 qualifying products from different vendors and was one of the first to deploy the winning platform in production infrastructure. The new routers provided the scale and density to support tens of terabits of throughput, and they made the WAN link migration possible.
The rollout moved quickly. By June 2017, Dropbox had migrated all US and EU WAN links — including transatlantic links — from multiple 10G LAG bundles to 100G. The change increased cumulative WAN capacity by roughly 300%.
Dual-stack routing with a single topology
Dropbox began rolling out IPv6 across its entire network in Q4 2016, with a design goal of parity between IPv4 and IPv6 for both routing and forwarding. To keep routing consistent across both address families, the team chose IS-IS single-topology rather than multi-topology.
For forwarding IPv6 traffic, Dropbox reused the same set of MPLS-TE LSPs already tunneling IPv4 traffic. That approach relied on IGP shortcuts as defined in RFC 3906, so both v4 and v6 traffic traversed the backbone over identical MPLS LSPs. The IPv6 rollout across data centers, backbone, and edge was complete by the end of Q1 2017.
Building for ten times the scale
Dropbox carries hundreds of gigabits of traffic and continues to grow rapidly. The Network Engineering team's guiding principle is to "build for scale" — but not in the sense of simply adding more capacity, nodes, or devices. Instead, the team periodically updates the architecture and constantly designs for operating the network at ten times today's scale.
That mindset means planning two to three years ahead, ensuring the tooling, automation, and monitoring are in place before the traffic arrives. The same principle applies across the entire network footprint, from data center to backbone to edge.
