Scout: Cloudflare’s Python-Based API Testing Layer
Cloudflare runs many products on the same core APIs, which makes broad, reliable testing of those live APIs critical. To catch regressions, prevent incidents, and keep those APIs healthy, the team built Scout: an automated system that periodically runs Python tests against API endpoints to verify end-to-end behavior.
Scout evaluates APIs in production-like environments, acting as a gate before production deployments. It also runs continuously in production as a monitoring tool.
Why replace the old test system?
Cloudflare’s previous testing setup relied on the Robot Framework, which introduced several operational bottlenecks:
- Difficult JSON assertions: Matching JSON responses against expected keys was not straightforward, limiting test coverage.
- Resource conflicts: It was hard to assign test suites to specific accounts or zones. When multiple suites ran on shared accounts, false negatives resulted from interference.
- Weak validation: Only API responses were checked against JSON schemas, and validation failures did not fail the test. Requests were never validated.
- Serial execution: Test suites ran in a queue, which delayed assessment of new features and meant tests could run against outdated API versions. There was no parallelism for test steps.
- No environment isolation: Test suites could not be split across environments, so writing tests for unreleased features required those features to be live in production first.
Scout was designed to solve these problems while giving developers an easy, fast, and reliable workflow.
A minimal Scout test
Scout is built in Python and extends Pytest. Here is an example test targeting the Rulesets API:
from scout import requires, validate, Account, Zone
@validate(schema="rulesets", ignorePaths=["accounts/[^/]+/rules/lists"])
@requires(
account=Account(
entitlements={"rulesets.max_rules_per_ruleset": 2),
zone=Zone(plan="ENT",
entitlements={"rulesets.firewall_custom_phase_allowed": True},
account_entitlements={"rulesets.max_rules_per_ruleset": 2 }))
class TestZone:
def test_create_custom_ruleset(self, cfapi):
response = cfapi.zone.request(
"POST",
"rulesets",
payload=f"""{{
"name": "My zone ruleset",
"description": "My ruleset description",
"phase": "http_request_firewall_custom",
"kind": "zone",
"rules": [
{{
"description": "My rule",
"action": "block",
"expression": "http.host eq \"fake.net\""
}}
]
}}""")
response.expect_json_success(
200,
result=f"""{{
"name": "My zone ruleset",
"version": "1",
"source": "firewall_custom",
"phase": "http_request_firewall_custom",
"kind": "zone",
"rules": [
{{
"description": "My rule",
"action": "block",
"expression": "http.host eq \"fake.net\"",
"enabled": true,
...
}}
],
...
}}""")A Scout test is essentially a series of request/response roundtrips against an API. Two Pytest mechanisms form the core: marks carry extra metadata about a test, while fixtures provide context and methods used across tests. Together, they build the harness for running a suite.
The cfapi fixture lets a test target specific resources, such as an account or zone. The @requires mark declares the characteristics those resources must have—for example, an account with a flag that permits two rules in a ruleset. This ensures tests run only where the setup is valid.
The @validate mark enforces that both requests and responses conform to a specified OpenAPI schema (here, the rulesets schema). Any mismatch is reported as a test failure. Payloads are written as f-strings, and responses can be described in a “semi-json” format:
response.expect_json_success(
200,
result=f"""{{
"name": "My zone ruleset",
"version": "1",
"source": "firewall_custom",
"phase": "phase_http_request_firewall_custom",
"kind": "zone",
"rules": [
{{
"description": "My rule",
"action": "block",
"expression": "http.host eq \"fake.net\"",
"enabled": true,
...
}}
],
...
}}""")Scout supports partial JSON matching by treating the ellipsis (…) as a wildcard: it tells Scout to ignore any further fields at that nesting level. This lets tests focus on the critical parts of a response without being brittle about unrelated additions.
Once a suite finishes, results are pushed to Cloudflare Workers KV and visualized through a Cloudflare Worker.
Architecture: three components
Scout combines three Python-based components:
The Scout plugin (Pytest plugin)
This is the heart of Scout, enabling descriptive tests with strong OpenAPI compliance checking. Its internal design has three layers:
- Setup: Contains the Registry, which holds information about a pool of test accounts and zones, including their feature entitlement flags. This lets tests run against very specific configurations. Validators are registered here per OpenAPI schema and are selected via the @validate mark. The config reader supplies URLs and authentication details.
- Resource allocator: This factory consumes the setup and provides specialized runners (account, zone, or default) via the cfapi fixture. When code invokes a method on this fixture—like a
requestcall—the matching runner for the target resource handles it. - Runners: These execute the actual HTTP requests, manage test expectations, and invoke schema validators. Any failures—whether expectation mismatches, validation errors, or exceptions—are recorded in a shared stash. The stash logs the full timeline of execution and retries, which later feeds into the suite report.
Parallelism is handled by associating each resource pair (account and zone runner) with a Pytest-xdist worker. This setup allows multiple test steps to run at once, and a separate default runner handles APIs and URLs that don’t require a specific account or zone.
Testing Scout itself was non-trivial. To reach and maintain a high degree of confidence (close to 90% test coverage), Cloudflare built a fake API to verify the plugin behaves correctly in varied situations.
The Scout service (scheduler)
The service schedules test suites on a timer rather than relying on cron jobs. A scheduled component offers better observability: if a job finishes before Prometheus scrapes its metrics, those metrics could be lost entirely. A purpose-built scheduler avoids this blind spot, exposing metrics for network failures, test outcomes, reporting problems, and test lag.
Each scheduled thread launches a separate Pytest process running the Scout plugin, followed by a reporting step that publishes results. Reports go to Workers KV, and chat notifications fire on failures. The reporting step also tracks coverage across all API endpoints and HTTP methods, which is a critical element for achieving full visibility of live API behavior. If metrics or reporting fail, logs from the service and the plugin run serve as the fallback diagnostic source.
The service is configured via a small YAML file and can be tailored per environment: different suites can run in different environments, publication to Workers is togglable, and retry behavior is configurable.
The Scout Worker (presentation)
A Cloudflare Worker pulls the latest report from Workers KV and renders it. Since the Scout service publishes results as JSON, the Worker parses that JSON and displays this data based on the test run’s status—including clear visual flags for issues like authentication failures.
What Scout unlocks
With Pytest and Cloudflare Workers at its core, Scout is a configurable, robust, and reliable test system. It offers request and response validation against OpenAPI schemas, lets tests target specific resources, and provides multiple out-of-band alerting paths. Test definitions live alongside the service’s YAML configuration within the same codebase.
Beyond pure API checks, tests can certify that edge configuration is valid and that a zone reacts appropriately to security threats. Scout has become Cloudflare’s permanent live API monitor and pre-deployment gate: after a rollout to a production-like environment, it takes only minutes to determine whether the new feature is safe to push to production.
