Why service limits matter

Running a large-scale service means learning, often the hard way, that every resource needs a ceiling. At Heroku, we once stored a JSON blob per app release that held configuration variables for the environment. Nobody had thought to cap its size. Months later, customers were using that field as a makeshift database, storing multi-megabyte blobs that bloated our database and slowed app start times as the data had to be transferred and set up at runtime.

The same pattern emerged at Stripe, where text fields accepted by the API were unbounded. Before long, users were encoding large JSON payloads into strings and using the API as a cold storage service. The lesson is universal: on the Internet, anything without an explicit limit will eventually be exploited — an app, a free TLS certificate, an API endpoint. A resource is anything you offer, and it needs a boundary.

At Stripe, our core services now rate limit across 20+ dimensions. A few examples:

  • Inbound requests from a single user.
  • The origin IP address of a request.
  • The number of in-flight requests per user, which prevents a few expensive requests from saturating a server.
  • A fleet-wide limit on non-critical requests, so a flood of analytical queries can't block someone trying to create a charge.

Most of these were added reactively as holes appeared. Operating large production systems is a game of cat and mouse — constantly reinforcing the perimeter as new cracks show up.

Designing limits that respect users

Limits exist to prevent abuse, but they affect legitimate users too. Hitting a rate limit by accident is frustrating, so it's worth considering a few principles:

  • Keep limits moderate. Unless there's a strong reason, don't be overly restrictive.
  • Be transparent. Publish your limits so users hit by them can adjust their behavior.

I recently built a small app that automatically creates playlists on Spotify. Once a week it needs to resolve 30 to 40 song names into Spotify IDs. Initially, I had it running up to 5 parallel fetches with 0 to 1 second sleeps between requests — too aggressive for Spotify's limits. I had to back off to sequential fetches with 1 to 2 second sleeps. The task is modest, and it's a bad sign that such a small integration trips limits so easily.

Amazon is the gold standard here. For a light or moderate user, you rarely brush against a limit. When you do, their service limits page documents everything in detail — from the maximum number of DynamoDB tables to how many VPCs an account can have per region. Spotify, on the other hand, is the counterexample: stringent limits paired with a long rate-limit page that contains no concrete numbers and very little useful information.

Implementing rate limiting

Most rate limiting uses a token bucket algorithm. I've written previously about GCRA (generic cell rate algorithm), a particular implementation that works very well. Along those lines, my redis-cell project distributes GCRA as a Redis module, exposing rate limiting as a single, language-agnostic command:

CL.THROTTLE <key> <max_burst> <count per period> <period> [<quantity>]

The response is an array of integers that indicate whether the action should be limited, along with metadata that can be used for response headers like X-RateLimit-Remaining:

127.0.0.1:6379> CL.THROTTLE user123 15 30 60
1) (integer) 0   # 0 is allowed, 1 is limited
2) (integer) 16  # X-RateLimit-Limit
3) (integer) 15  # X-RateLimit-Remaining
4) (integer) -1  # Retry-After (seconds)
5) (integer) 2   # X-RateLimit-Reset (seconds)

Being a Redis module gives redis-cell a speed advantage. Informal benchmarks put it at under twice as slow as a plain Redis SET command — roughly 0.1 ms per execution, or about 10,000 operations per second from a client's perspective. It's written in Rust, using the language's FFI to talk to Redis, which keeps the codebase safer and easier to contribute to than typical C implementations. If you prefer pure Go, the throttled project offers another GCRA implementation.