Cloudflare’s URL Scanner, new features, and the story of how we built it

From Radar to Dashboard: Cloudflare’s URL Scanner Gains an API and Security Center Home

Cloudflare’s URL Scanner, which launched on Cloudflare Radar in March, has crossed nearly one million scans. The tool, built on Cloudflare Workers, Durable Objects, and the Browser Rendering API, is now expanding beyond its Radar origins with deeper integration into the Cloudflare Dashboard and the release of an official API for developers.

The scanner’s core value is straightforward: it lets security teams and individual users analyze suspicious websites before engaging with them. Scans check for phishing, malware, and other threats, and they return a detailed report on what the site does when loaded. The new features push that capability into more hands and more workflows.

Security Center Integration for Dashboard Users

Cloudflare’s Security Center is designed as the single pane of glass for attack surface mapping, risk identification, and mitigation. It now includes the URL scanner within its Investigate Portal. This means dashboard users can launch scans directly from their security workflow, with all historical scans stored in one place for later review.

Scans initiated from Security Center are unlisted by default, preserving privacy during investigations. They also automatically capture screenshots at multiple screen sizes, which adds context beyond a single desktop view. Cloudflare notes that customers with dashboard access benefit from higher API rate limits and faster response times than the public tier.

The URL Scanner API and Its New Capabilities

For developers, the URL Scanner API opens a path to programmatic website assessment: custom scans, phishing or malware detection, and technology stack analysis. This iteration of the API lands with several meaningful additions.

Submitting Scans with More Control

Setting up a scan is a single API call:

curl --request POST \
	--url https://api.cloudflare.com/client/v4/accounts/<accountId>/urlscanner/scan \
	--header 'Content-Type: application/json' \
--header "Authorization: Bearer <API_TOKEN>" \
	--data '{
		"url": "https://www.cloudflare.com",
	}'

Beyond the basic submission, developers can now attach custom HTTP headers — including User-Agent and Authorization — which are useful for testing sites that behave differently based on client identity or authentication state. They can also request screenshots from multiple device types, such as mobile and desktop, and set the scan’s visibility to “unlisted.” That unlisted mode, which effectively marks a scan private, was a frequent request from developers wanting to keep investigations confidential. Public scans remain searchable by anyone, which suits researchers sharing findings with the community.

Reading Results with New Signals

Once a scan completes, the report and the full network log can be fetched through the API. Two recent additions to the response are worth flagging. The verdict property provides an assessment of whether the scanned site is malicious. The securityViolations section reports Content Security Policy (CSP) or Subresource Integrity (SRI) policy breaches — a practical tool for developers who want to audit their own sites against Cloudflare’s recommendations. Verdict accuracy, Cloudflare says, will continue to improve as the system is refined.

Search That Digs Deeper

Search functionality has been extended. Developers can now search scans by hostname, a specific partial URL, or even any URL the page connected to during the scan. That last option enables some surprisingly precise queries — for instance, finding every scanned website that loads a JavaScript library named jquery.min.js with the query ?path=jquery.min.js. Future plans include search by IP address, autonomous system number (ASN), and malicious site classification.

Use Cases Beyond Threat Hunting

The scanner’s flexibility supports a range of applications. Teams can capture a site’s changing state over time, such as tracking how an online newspaper’s homepage evolves. Others might use it solely for technology analysis. It is also suited to preemptive risk checks, like expanding a shortened URL to see where it actually leads. And for sustained investigations, the scanner can identify which sites are serving a known malicious file — a useful technique for mapping the scope of a distributed attack.

Inside the URL Scanner’s architecture

The URL Scanner is built entirely on Cloudflare’s own developer platform, which serves as both a demonstration and a stress test of what the platform can do. The system runs on Cloudflare Workers for the public API, Durable Objects for orchestration, R2 for primary storage, and Queues for batch operations. The key enabler, though, is the Browser Rendering API, which removes the need to build and manage a fleet of Chrome browsers from scratch. Instead, the Scanner requests a browser instance and drives it with the familiar Puppeteer library.

High level overview of the Cloudflare URL Scanner technology stack

From request to report

Each scan proceeds through four phases:

1. Queue a scan
2. Browse to the website and compile an initial report
3. Post-process: add additional information and build the final report
4. Store the final report for serving and searching

The scan is orchestrated by a unique Durable Object, called the Scanner, which lives for the duration of the scan. When a user submits a URL, the Scanner saves the request to its transactional key-value storage, schedules an alarm to fire roughly a second later, and immediately responds with an acceptance notice. When the alarm triggers, the browsing phase begins.

Three Durable Objects cooperate in this phase: the Scanner, the Browser Pool, and the Browser Controller. The original version launched a fresh browser for each scan, but that proved slow and wasteful. Reusing browsers across scans required the new components. The Browser Pool tracks which browsers are open, their last heartbeat, and whether they are free to take on a new scan. The Browser Controller keeps a launched browser alive and manages the entire browsing session via Puppeteer. A simplified version of the controller code:

export class BrowserController implements DurableObject {
	//[..]
	private async handleNewScan(url: string) {
		if (!this.browser) {
			// Launch browser: 1st request to durable object
			this.browser = await puppeteer.launch(this.env.BROWSER)
			await this.state.storage.setAlarm(Date.now() + 5 * 1000)
		}
		// Open new page and navigate to url
		const page = await this.browser.newPage()
		await page.goto(url, { waitUntil: 'networkidle2', timeout: 5000, })

		// Capture DOM
		const dom = await page.content()

		// Clean up
		await page.close()

		return {
			dom: dom,
		}
	}

	async alarm() {
		if (!this.browser) {
			return
		}
		await this.browser.version() // stop websocket connection to Chrome from going idle
		
		// ping browser pool, let it know we're alive
		
		// Keep durable object alive
		await this.state.storage.setAlarm(Date.now() + 5 * 1000)
	}
}

Launching a browser and maintaining a connection to it is abstracted away by the Browser Rendering API, which handles all the infrastructure for a fleet of Chrome browsers. That abstraction, plus the ability to use Puppeteer over the DevTools protocol, was a major factor in the Scanner’s quick development and release.

The initial report consists of a network log captured in HAR (HTTP Archive) format, which is a JSON-based industry standard that can be exchanged and inspected with existing tools. The scan data is also enriched with metadata such as base64-encoded screenshots taken at scan time.

Post-processing and storage

After the browsing phase, the Scanner Durable Object calls several other Cloudflare APIs to expand the report: a phishing scanner runs over the page’s DOM, DNS records are fetched, and categories and Radar rank are looked up for the main hostname. The final report is assembled and stored as a JSON file in R2, with Postgres powering scan searches.

Sending each finished scan to PostgreSQL immediately worked initially, but as scan volume grew, batching became necessary. Worker Queues handle that:

This approach smooths the write load on Postgres. Scans reach the requester as soon as they’re ready, while appearing in search results a bit later—anywhere from seconds to minutes, depending on load. When a user later requests a scan by ID, the API Worker simply fetches it from R2.

Building the API itself

The API Worker is written in Typescript and uses itty-router-openapi, a router that generates and validates Open API 3 schemas. It was originally built for Cloudflare Radar and has continued to evolve with community contributions.

import { DateOnly, OpenAPIRoute, Path, Str, OpenAPIRouter } from '@cloudflare/itty-router-openapi'

import { z } from 'zod'
import { OpenAPIRoute, OpenAPIRouter, Uuid } from '@cloudflare/itty-router-openapi'

export class ScanMetadataCreate extends OpenAPIRoute {
  static schema = {
    tags: ['Scans'],
    summary: 'Create Scan metadata',
    requestBody: {
      scan_id: Uuid,
      url: z.string().url(),
      destination_ip: z.string().ip(),
      timestamp: z.string().datetime(),
      console_logs: [z.string()],
    },
  }

  async handle(
    request: Request,
    env: any,
    context: any,
    data: any,
  ) {
    // Retrieve validated scan
    const newScanMetadata = data.body

    // Insert the scan

    // Return scan as json
    return newScanMetadata
  }
}

const router = OpenAPIRouter()
router.post('/scan/metadata/', ScanMetadataCreate)

// 404 for everything else
router.all('*', () => new Response('Not Found.', { status: 404 }))

export default {
  fetch: router.handle,
}

In the example above, the ScanMetadataCreate endpoint validates the incoming POST request against the defined schema before the handle function is called. This ensures that by the time your code runs, the data argument is guaranteed to be properly validated and formatted.

Planned improvements

Upcoming work on the URL Scanner includes location customization for scans, which should offer more insight into region-specific security threats and content compliance; broader scan details, such as fuller headers and security information; and ongoing performance work to return results faster. The basic scan feature on Cloudflare Radar remains available to the public, while more advanced scanning capabilities are integrated into Security Center for existing Cloudflare customers.