Courier: Dropbox's production gRPC layer
Dropbox runs hundreds of services across multiple languages, exchanging millions of requests per second. The foundation of that Service Oriented Architecture is Courier, a gRPC-based RPC framework that integrates gRPC with Dropbox's existing infrastructure for authentication, authorization, service discovery, stats, logging, and tracing. Courier supports code generation in Python, Go, Rust, and Java.
Why gRPC
Dropbox's previous RPC framework was HTTP/1.1-based with protobuf-encoded messages. Before Courier, services also used Apache Thrift in some pipelines and custom manual serialization in others. When it came time to build a new framework, the team weighed evolving the legacy system, building on Thrift, or adopting gRPC. gRPC won for three practical reasons: existing protobufs could be carried forward, HTTP/2 multiplexing was attractive, and bi-directional streaming was supported. Notably, had fbthrift existed at the time, Thrift-based solutions may have gotten a closer look.
Courier is not a new RPC protocol; it is how Dropbox plugs gRPC into its own infrastructure. All common service features ship in the framework by default, controlled by command-line arguments and dynamically toggleable via feature flags. The majority of services communicate with no proxy in between to minimize latency impact, though Bandaid is used as a gRPC proxy for specific use cases.
Security: TLS identity and mutual authentication
Every Courier server and client has a TLS certificate issued by Dropbox's internal Certificate Authority. Each certificate encodes a service identity, used for mutual authentication—both server verifies client and client verifies server. Because Dropbox controls both ends, TLS defaults are restrictive: perfect forward secrecy is mandatory, the TLS version is pinned to 1.2+, and algorithms are restricted to a secure subset with ECDHE-ECDSA-AES128-GCM-SHA256 preferred.
After identity is confirmed, servers check permissions. ACLs and rate limits apply at both service and method granularity, and can be updated via Dropbox's distributed config filesystem (AFS), letting service owners shed load in seconds without process restarts. Courier handles config update subscriptions. An example ACL/rate-limit configuration for Dropbox's OCR service:
limits:
dropbox_engine_ocr:
# All RPC methods.
default:
max_concurrency: 32
queue_timeout_ms: 1000
rate_acls:
# OCR clients are unlimited.
ocr: -1
# Nobody else gets to talk to us.
authenticated: 0
unauthenticated: 0
Dropbox is considering adopting the SPIFFE Verifiable Identity Document (SVID) from the Secure Production Identity Framework for Everyone (SPIFFE), which would make Courier compatible with more open source projects.
Observability: stats and tracing
Given a service identity, engineers can find standard logs, stats, traces and other operational data. Code generation produces per-service and per-method stats for both client and server, with server stats broken down by client identity. Out of the box, any Courier service has granular load, error and latency attribution.
Stats include client-side availability, client-side latency, server-side request rates and queue sizes, plus breakdowns like per-method latency histograms and per-client TLS handshakes. Because Dropbox owns code generation, these data structures—including histograms and tracing spans—are initialized statically, keeping the performance impact minimal.
Tracing replaced the legacy RPC system's request_id propagation with an API based on a subset of the OpenTracing specification. Client libraries are custom-written; server-side tracing runs on Cassandra and Jaeger. Tracing data also enables runtime dependency graph generation, useful both for understanding transitive dependencies and as a post-deploy check to catch unintended new dependencies.
Reliability: deadlines and circuit-breaking
Courier gives Dropbox a single place to implement client-side reliability features common across languages. Many of these features were added as action items from postmortems.
Deadlines
Every gRPC request carries a deadline indicating how long the client will wait for a response. Courier stubs automatically propagate known metadata, so the deadline crosses API boundaries with the request. In-process deadlines convert to native representations—for example, in Go, a context.Context from the WithDeadline method. Requiring engineers to define deadlines in service definitions has fixed entire classes of reliability problems.
The deadline context can propagate outside the RPC layer entirely. Dropbox's legacy MySQL ORM serializes the RPC context, deadline included, into a SQL comment. SQLProxy parses those comments and KILLs queries that exceed the deadline, gaining per-request attribution in database debugging as a side benefit.
Circuit-breaking
Legacy RPC clients handled retries with custom exponential backoff and jitter to prevent cascading overloads. Courier solves circuit-breaking generically with a LIFO queue between the listener and the workpool.
Under service overload, the queue acts as an automatic circuit breaker. It is bounded by both size and time: a request can only spend so long in the queue. LIFO reorders requests, however. For order-preserving use cases, CoDel provides similar circuit-breaking behavior without reordering.
Introspection endpoints
Though not part of Courier itself, debug endpoints are widely adopted across Dropbox. Security guidance: expose them on a separate port (loopback-only if possible) or a Unix socket with file permissions, and consider requiring developers to present TLS certificates to access them, especially non-readonly ones.
Useful introspection categories include:
- Runtime — heap and CPU profiles exposed as HTTP or gRPC endpoints, e.g. Go pprof. Planned use includes automating CPU/memory diffs between old and new code during canary verification. Runtime state can be mutable too—Go services can dynamically adjust GCPercent.
- Library — library authors can export library-specific RPC endpoints, e.g. jemalloc internal stats dumps, or a read/write endpoint to change service log levels at runtime.
- RPC — binary and encrypted protocols are hard to troubleshoot, so RPC-layer instrumentation should be as rich as performance allows. The gRPC channelz proposal is a recent example.
- Application — a generalized info endpoint with build/source hash, command line and similar data, letting orchestration systems verify deployment consistency.
Performance tuning at scale
Rolling out gRPC across Dropbox surfaced several performance bottlenecks that only become visible at significant traffic volume. The most impactful optimizations touched TLS handling, memory management, and the transport layer itself.
TLS handshake cost
For a service that manages a high volume of connections, the aggregate CPU cost of TLS handshakes adds up quickly—especially during mass service restarts. We moved from RSA 2048 keypairs to ECDSA P-256 to reduce the cost of signing operations, as the following BoringSSL benchmarks illustrate (note RSA retains an edge for signature verification):
𝛌 ~/c0d3/boringssl bazel run -- //:bssl speed -filter 'RSA 2048'
Did ... RSA 2048 signing operations in .............. (1527.9 ops/sec)
Did ... RSA 2048 verify (same key) operations in .... (37066.4 ops/sec)
Did ... RSA 2048 verify (fresh key) operations in ... (25887.6 ops/sec)
ECDSA:
𝛌 ~/c0d3/boringssl bazel run -- //:bssl speed -filter 'ECDSA P-256'
Did ... ECDSA P-256 signing operations in ... (40410.9 ops/sec)
Did ... ECDSA P-256 verify operations in .... (17037.5 ops/sec)
RSA 2048 verification is about 3x faster than ECDSA P-256, so you might consider RSA for root or leaf certs from a pure throughput standpoint. The security trade-offs are more involved, though, since chaining different primitives means the overall security properties are limited by the weakest link. The same logic argues against RSA 4096+ certificates.
TLS library selection and compile flags also mattered significantly. On identical hardware, a macOS Mojave LibreSSL build trailed a Homebrew OpenSSL build notably:
LibreSSL 2.6.4:
𝛌 ~ openssl speed rsa2048
LibreSSL 2.6.4
...
sign verify sign/s verify/s
rsa 2048 bits 0.032491s 0.001505s 30.8 664.3
OpenSSL 1.1.1a:
𝛌 ~ openssl speed rsa2048
OpenSSL 1.1.1a 20 Nov 2018
...
sign verify sign/s verify/s
rsa 2048 bits 0.000992s 0.000029s 1208.0 34454.8
The cheapest handshake, however, remains the one you skip. We patched gRPC-core and gRPC-python to support session resumption, which cut the CPU load during rollouts substantially.
Encryption is cheap; copies are not
A common assumption is that encryption is performance-heavy. In practice, modern hardware handles symmetric encryption at remarkable speed—a single desktop core can encrypt and authenticate data at roughly 40 Gbps:
𝛌 ~/c0d3/boringssl bazel run -- //:bssl speed -filter 'AES'
Did ... AES-128-GCM (8192 bytes) seal operations in ... 4534.4 MB/s
We still needed to adjust gRPC for our 50 Gb/s storage boxes, where the data path approaches memory-copy speeds. In that regime, minimizing memcpy calls had a bigger impact than any cipher choice, so we contributed changes back to gRPC to reduce copies. Note that authenticated, encrypted protocols have a side benefit: they frequently expose subtle hardware corruption from processors, DMA engines, or network devices. Even outside gRPC, using TLS for internal traffic is strongly recommended.
High bandwidth-delay product links
Dropbox connects multiple data centers over a backbone network, and cross-region RPCs (for instance, replication traffic) traverse links with a high bandwidth-delay product. TCP’s kernel window controls limit in-flight data per connection, but gRPC/HTTP-2 imposes its own flow control on top. In grpc-go, the BDP upper bound is hardcoded to 16 Mb, which can throttle a single connection on such links.
Server and serializer choice in Go
Our initial Go services exposed both HTTP/1.1 and gRPC through the same net.Server. Maintaining one entry point simplified code, but throughput and memory suffered. Splitting the protocols onto separate servers—gRPC onto a dedicated grpc.Server—improved both metrics noticeably.
Serialization also became a larger cost under gRPC’s load. Migrating our Go services from golang/protobuf to gogo/protobuf reduced CPU usage on the busiest Courier servers. As with any dependency, there are caveats; staying within a pragmatic subset of gogo’s features avoids the sharp edges.
Within the Courier stubs
Delving into Courier’s implementation, we’ll use the Test service (used in our integration tests) to illustrate protobuf schemas and generated stubs across our languages.
Defining the service contract
Courier enforces that all methods declare deadlines. The service definition sets a default application-wide via a protobuf option:
option (rpc_core.service_default_deadline_ms) = 1000;
Methods can override this with their own, tighter deadline:
option (rpc_core.method_default_deadline_ms) = 5000;
In cases where a deadline is meaningless—such as a resource-watch method—developers may explicitly disable it:
option (rpc_core.method_no_deadline) = true;
Service definitions also carry extensive API documentation, frequently with usage examples. A snippet from the Test service shows the overall shape:
service Test {
option (rpc_core.service_default_deadline_ms) = 1000;
rpc UnaryUnary(TestRequest) returns (TestResponse) {
option (rpc_core.method_default_deadline_ms) = 5000;
}
rpc UnaryStream(TestRequest) returns (stream TestResponse) {
option (rpc_core.method_no_deadline) = true;
}
...
}
Why Courier generates stubs
Courier eschews interceptors in favor of generated stubs (Java is the exception, since its interceptor API suffices). The main motivation is flexibility, which is visible when comparing a default Go gRPC stub with a Courier one. The default handles processing inline—decoding protobuf, running interceptors, then invoking the handler:
func _Test_UnaryUnary_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
in := new(TestRequest)
if err := dec(in); err != nil {
return nil, err
}
if interceptor == nil {
return srv.(TestServer).UnaryUnary(ctx, in)
}
info := &grpc.UnaryServerInfo{
Server: srv,
FullMethod: "/test.Test/UnaryUnary",
}
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
return srv.(TestServer).UnaryUnary(ctx, req.(*TestRequest))
}
return interceptor(ctx, in, info, handler)
}
Courier’s generated code, by contrast, does substantially more up front:
func _Test_UnaryUnary_dbxHandler(
srv interface{},
ctx context.Context,
dec func(interface{}) error,
interceptor grpc.UnaryServerInterceptor) (
interface{},
error) {
defer processor.PanicHandler()
impl := srv.(*dbxTestServerImpl)
metadata := impl.testUnaryUnaryMetadata
ctx = metadata.SetupContext(ctx)
clientId = client_info.ClientId(ctx)
stats := metadata.StatsMap.GetOrCreatePerClientStats(clientId)
stats.TotalCount.Inc()
req := &processor.UnaryUnaryRequest{
Srv: srv,
Ctx: ctx,
Dec: dec,
Interceptor: interceptor,
RpcStats: stats,
Metadata: metadata,
FullMethodPath: "/test.Test/UnaryUnary",
Req: &test.TestRequest{},
Handler: impl._UnaryUnary_internalHandler,
ClientId: clientId,
EnqueueTime: time.Now(),
}
metadata.WorkPool.Process(req).Wait()
return req.Resp, req.Err
}
Walking through that generated code:
- Panic handling deferred: the stub registers a handler that forwards all uncaught exceptions to centralized aggregation and reporting:
defer processor.PanicHandler()
A custom handler also lets us abort the application on panic, rather than following the default Go net/http behavior of continuing to serve requests from a potentially corrupted process.
Context propagation follows, pulling values from incoming request metadata:
ctx = metadata.SetupContext(ctx)
clientId = client_info.ClientId(ctx)
For finer-grained attribution, the stub creates (and caches) per-client statistics, ultimately keyed by TLS identity:
stats := metadata.StatsMap.GetOrCreatePerClientStats(clientId)
Per-client stats are created at runtime, but Courier also pre-creates per-method stats at codegen time. Since the generator sees every method, it can avoid any runtime overhead for those.
After building the request structure, the stub submits work to a pool and waits:
req := &processor.UnaryUnaryRequest{
Srv: srv,
Ctx: ctx,
Dec: dec,
Interceptor: interceptor,
RpcStats: stats,
Metadata: metadata,
...
}
metadata.WorkPool.Process(req).Wait()
At this stage, almost nothing has happened: no protobuf decode, no interceptor logic. The work pool enforces ACLs, prioritization, and rate limiting before those heavier steps run. For unusually early ordering, gRPC’s Tap interface allows intercepting requests before the workpool processes them, which suits building low-overhead rate limiters.
App-specific error semantics
The stub generator exposes custom protobuf options to declare application-level error codes:
enum ErrorCode {
option (rpc_core.rpc_error) = true;
UNKNOWN = 0;
NOT_FOUND = 1 [(rpc_core.grpc_code)="NOT_FOUND"];
ALREADY_EXISTS = 2 [(rpc_core.grpc_code)="ALREADY_EXISTS"];
...
STALE_READ = 7 [(rpc_core.grpc_code)="UNAVAILABLE"];
SHUTTING_DOWN = 8 [(rpc_core.grpc_code)="CANCELLED"];
}
Within a service, both gRPC-native and app-specific errors propagate. Across API boundaries, though, the stubs substitute UNKNOWN for all errors—preventing an error from one service from being forwarded to another with a different meaning than originally intended.
Python-specific conventions
Courier’s Python stubs explicitly expose a context parameter to every handler:
from dropbox.context import Context
from dropbox.proto.test.service_pb2 import (
TestRequest,
TestResponse,
)
from typing_extensions import Protocol
class TestCourierClient(Protocol):
def UnaryUnary(
self,
ctx, # type: Context
request, # type: TestRequest
):
# type: (...) -> TestResponse
...
The explicit ctx initially looked unusual, but adoption mirrors the familiarity developers already have with self. Stubs are fully typed for mypy, which proves valuable during broad refactors and integrates cleanly with IDEs like PyCharm. Typing extends to the protobuf schemas themselves:
class TestMessage(Message):
field: int
def __init__(self,
field : Optional[int] = ...,
) -> None: ...
@staticmethod
def FromString(s: bytes) -> TestMessage: ...
These annotations catch entire bug classes early, such as assigning None to a string field. The tooling is open source at dropbox/mypy-protobuf.
Migrating without breaking services
Building a new RPC stack is less operationally risky than moving an entire infrastructure onto it. To control risk, we planned a staged migration:
Stage 0: Freeze the legacy API
First, the legacy RPC feature set was frozen. This stops it from being a moving target and creates a natural draw toward Courier, since new capabilities like tracing and streaming are exclusive to services on the new stack.
Stages 1-2: Interface alignment
Next, we defined a common interface to be satisfied by both legacy and Courier stub variants—generated code implementing it either way:
type TestServer interface {
UnaryUnary(
ctx context.Context,
req *test.TestRequest) (
*test.TestResponse,
error)
...
}
With that interface in place, each service switched to it while continuing on legacy RPC. These diffs are large since they touch every method and client; changing one variable at a time during this step reduces risk greatly. Teams with ample error budget and a low-profile, small service may combine this with the next step in a single move.
Stage 3: Client cutover
We ran legacy and Courier servers side by side in the same binary, each on its own port. Flipping a client to the new implementation then becomes a one-line change:
class MyClient(object):
def __init__(self):
- self.client = LegacyRPCClient('myservice')
+ self.client = CourierRPCClient('myservice')
This lets us migrate clients incrementally, beginning with lower-SLA workloads such as batch and async jobs.
Stage 4: Cleanup
After every client migrates, we verify statically and at runtime (via legacy server stats) that the old protocol is gone, then remove the legacy implementation entirely.
Observations and where we go next
Unified RPC has made services quicker to develop, simpler to operate, and more reliable. The effort distilled into a few recurring lessons:
- Observability is a product feature; embedded metrics and breakdowns prove invaluable during debugging.
- Standardization reduces cognitive load and eases both operations and maintenance.
- Minimize boilerplate with code generation.
- Plan migration carefully and expect it to consume more time than building the new system. It’s finished only after cleanup is done.
- The RPC framework itself can enforce service-wide reliability invariants—deadlines, load shedding, and similar—identified by reviewing incidents quarterly.
Courier is an evolving project. Near-term work includes a proper resolver API for Python gRPC, switching Python/Rust bindings to the C++ core, and adding full circuit-breaking and fault injection. Beyond that, we’re evaluating ALTS and relocating TLS handshakes entirely out of service processes (and possibly out of service containers).
Courier teams open up hiring as gRPC migration wraps
With the Courier/gRPC rollout complete, Dropbox is scaling up the teams that built and now run the runtime layer. Open roles span software engineering and site reliability positions on the Traffic, Runtime, and Reliability teams, working on TCP/IP packet processors, load balancers, HTTP/gRPC proxies, and the internal service mesh runtime: Courier/gRPC, Service Discovery, and AFS. The company also lists wider engineering openings across San Francisco, New York, Seattle, Tel Aviv, and other offices.
The work described in this migration series was a joint effort. Contributors include Ashwin Amit, Can Berk Guder, Dave Zbarsky, Giang Nguyen, Mehrdad Afshari, Patrick Lee, Ross Delinger, Ruslan Nigmatullin, Russ Allbery, and Santosh Ananthakrishnan, with additional support from the gRPC team at Google.
