Why Security Insights needed a scalability overhaul

Security Insights delivers automated security recommendations across every Cloudflare account, zone, and DNS record. But the scanning system behind it had two critical limitations: scans ran too infrequently—only every one to two weeks—and automatic scanning was opt-in for many free plans, leaving numerous accounts completely unscanned. With automated attacks accelerating, undetected misconfigurations become riskier with each passing day.

Closing that detection gap required roughly a 10x increase in average scanning throughput—from 10 to 100 scans per second. The existing system was already strained under its current load: millions of events sat in backlog, the internal API frequently timed out, and processes were crashing. We needed architectural fixes, not just more resources.

Bottleneck: Kafka's ordered consumption model

Scans are triggered by a scheduler that publishes messages to Apache Kafka. Specialized Go microservices—each acting as a checker—consume those messages and send findings to an internal API, which persists them to Postgres.

Kafka is a partitioned event stream, not a queue. Messages within a partition must be consumed and processed in order, meaning each partition can only have one active consumer per consumer group. This created two constraints:

  • A slow message blocks the consumer from progressing to the next one
  • Each checker can only run as many concurrent consumers as it has partitions

Adding partitions to the shared Kafka broker was a last resort due to resource costs, so we looked at code-level changes first.

Batched consumption with goroutines

Ordered consumption doesn't prevent consuming multiple messages simultaneously. We modified the checkers to pull messages in batches and process each message in its own goroutine. The trade-offs—more reprocessing work on crashes and slightly higher memory usage—were acceptable in our case.

Slow and fast lanes

Some messages were inherently slow: an account or zone with far more assets could take minutes or hours to process versus milliseconds for typical messages. Rather than let those stragglers block the pipeline, we split consumer groups into two lanes. The fast lane checker skips messages it can quickly identify as slow, sending them to a dedicated slow lane with appropriate resources. Fast messages no longer wait behind slow ones.

Fixing pathological database write patterns

Every insight found by a checker is written to Postgres via a single API endpoint that accepts a list of insights. The original implementation made a separate database round trip per insight. With observed batch sizes up to 500,000 insights, a single API call could generate half a million queries and transactions.

The standard bulk-insert approach—COPY into a temporary table—caused bloat in Postgres system tables. The hybrid solution we adopted uses UNNEST for smaller batches and COPY above a threshold. This delivers millisecond inserts for typical calls and keeps large batches down to seconds.

Root cause of API timeouts: geography

Three symptoms pointed to a shared root cause: widespread client-side timeouts, checkers spending 20-90% of processing time on a single API call, and throughput that started high then deteriorated during scan bursts. The culprit was latency.

The primary database lived in Portland, Oregon. The API ran active-active in both Portland and Amsterdam, and the speed-of-light round trip between those cities is 50 milliseconds. Queries from Amsterdam held client connections far longer, exhausting connection pools and causing timeouts to cascade. Average API call time was 10 ms in Portland but nearly 3 seconds in Amsterdam.

The throughput degradation followed directly: Kafka partitions assigned to processes that connected to the Amsterdam API lagged far behind those connected to Portland. Switching the API to active-passive, with the active instance colocated with the primary database, eliminated the latency problem overnight.

Making the scheduler produce uniform scans

Even with the pipeline scaled, we couldn't simply queue all scans at once. Kafka's time-based retention would delete queued scans before they could be processed. The scheduler needed to distribute scans roughly uniformly in time—but it didn't.

The scheduler triggered scans on fixed recurring periods. Inspecting the database revealed a key issue: last_scheduled_at timestamps were clustered for many accounts, causing hundreds of thousands of scans to fire within minutes of each other. Increasing scan frequency would compound the problem—shifting from 15-day to 7-day intervals would make 53% of accounts immediately due for a scan. Large accounts with many zones also caused cascades that saturated Kafka partitions and delayed smaller accounts.

Three changes fixed the scheduler:

  • Independent zone scheduling: Each zone now tracks its own last_scheduled_at field instead of inheriting the account's schedule.
  • Randomized timestamps: Existing accounts and zones got randomized last_scheduled_at values to break up the existing clustering (without delaying any due scans).
  • Adaptive rate limiting: A scan scheduling rate limit is recalculated every half hour based on the total number of accounts and zones. If the account count grows, the limit rises accordingly, ensuring all accounts are still scanned within their frequency window.

Results: 10x throughput, and then some

The combined fixes raised sustained throughput to over 120 scans per second during peak scheduling—surpassing the original 10x target. The API no longer times out, Kafka lag metrics are healthy, and the stability gains enabled two major product improvements: automatic security scanning is now enabled by default for all free accounts and zones, and scan frequency has doubled for everyone.

  • Free: every 7 days (previously every 14 days)
  • Pro and Business: every 3 days
  • Enterprise: daily

System stability also unlocked granular on-demand scans, letting any user manually rescan a specific account, zone, insight, or insight type from the Security Overview page.

The core lesson: understand the existing system before discarding it. Scrutinizing code, SQL queries, logs, and metrics—especially metrics that looked wrong—allowed us to multiply capacity without adding brokers, partitions, or pods. The easy shortcuts, like raising client-side timeout values, would have masked the underlying problems rather than solving them.