How we built an open-source SEO tool using Workers, D1, and Queues

An open-source SEO monitor built on Workers, D1, and Queues

Cloudflare’s developer platform is often used for prototyping, but it also powers many of our production services. For internal tooling, it's a natural fit. When our SEO team needed a better way to track internal linking opportunities across our blog and marketing site, we built a tool called Prospector using Workers, D1, and Queues. The project is open-source and available in the cloudflare/templates repository.

The problem: finding new internal linking opportunities

Internal links help search engines understand site structure and content relevance. Unlike external links, which depend on third parties, internal links are fully under a site owner's control, making them a key SEO lever.

At Cloudflare’s scale, with thousands of published pages and constant new content, manually tracking where new relevant internal links should be added is impractical. Existing third-party tools fell short: some only scanned the first 2,000 characters of a page, some couldn't limit scans to specific site sections, and others required manual operation. We needed an automated solution that would monitor a defined set of URLs and notify SEO experts when specific keywords appear, so they can act quickly on new linking opportunities.

Application architecture

Prospector comprises a user-facing API built with Workers, a real-time data store in D1, and a job queue handled by Queues. The front end is plain HTML, CSS, and JavaScript. Notifications are sent via the MailChannels integration, requiring just one API call.

The API layer uses the Hono framework, an Express-style router for Workers, to define REST endpoints concisely. The application entry point uses the Module Workers syntax to export event handlers:

import { DBUrl, Env } from './types'

import {
  handleQueuedUrl,
  scheduled,
} from './functions';

import h from './api'

export default {
  async fetch(
	request: Request,
	env: Env,
	ctx: ExecutionContext
  ): Promise<Response> {
	return h.fetch(request, env, ctx)
  },

  async queue(
	batch: MessageBatch<Error>,
	env: Env
  ): Promise<void> {
	for (const message of batch.messages) {
  	const url: DBUrl = JSON.parse(message.body)
  	await handleQueuedUrl(url, env.DB)
	}
  },

  async scheduled(
	env: Env,
  ): Promise<void> {
	await scheduled({
  	authToken: env.AUTH_TOKEN,
  	db: env.DB,
  	queue: env.QUEUE,
  	sitemapUrl: env.SITEMAP_URL,
	})
  }
};

This structure shows how the fetch event routes HTTP requests, the queue event processes jobs in the queue, and the scheduled event drives the recurring scrape process.

With Hono, API routes are quick to define. A few example routes wrap CRUD operations for managing notifiers and URLs:

const app = new Hono()

app.get("/", (context) => {
  return context.html(index)
})

app.post("/notifiers", async context => {
  try {
	const { keyword, email } = await context.req.parseBody()
	await context.env.DB.prepare(
  	"insert into notifiers (keyword, email) values (?, ?)"
	).bind(keyword, email).run()
	return context.redirect('/')
  } catch (err) {
	context.status(500)
	return context.text("Something went wrong")
  }
})

app.get('/sitemaps', async (context) => {
  const query = await context.env.DB.prepare(
	"select * from sitemaps"
  ).all();
  const sitemaps: Array<DBSitemap> = query.results
  return context.json(sitemaps)
})

Data model and storage with D1

Prospector relies on four D1 tables. The notifiers table stores the email address and keyword to monitor. The urls table holds specific page URLs and their sitemaps. Each notifier can have many associated URLs.

The sitemaps table records sitemap URLs that have been discovered. Large sites often split content into multiple sitemaps — for instance, the Cloudflare blog's primary sitemap contains four sub-sitemaps. When the application is configured with a primary sitemap, it parses it to find and index all additional ones.

Finally, the notifier_matches table tracks which keyword–URL pairs have already produced a match. Once a notifier has matched a URL, that URL is skipped on future scans. This prevents duplicate emails from cluttering the recipients' inboxes.

Orchestrating jobs with Cloudflare Queues

Cloudflare Queues acts as Prospector's work queue. When a notifier is added, a job is created for it. The queue event handler may run these jobs across multiple Workers, distributing the work as needed. When a job runs, Prospector scrapes the target URL, looks for the keyword, and sends the email notification if there's a match.

Scheduled scanning is set up with Cron Triggers, defaulting to daily execution. This keeps the data current. The schedule can be tuned by the end-user to control email frequency; for instance, you could configure it to run at the start of the workday.

Development benefits from the continuously generated, workerd-based TypeScript bindings for Workers. The environment type definition shows typed bindings for D1Database and Queue, which provide editor-level checks for API method usage.

export interface Env {
  AUTH_TOKEN: string
  DB: D1Database
  QUEUE: Queue
  SITEMAP_URL: string
}

Deployment and setup

Prospector is designed to deploy quickly with Wrangler. Start by cloning the repository, then install Wrangler and authenticate. Next, create a D1 database and a queue:

wrangler d1 create $DATABASE_NAME
wrangler queues create $QUEUE_NAME

Your wrangler.toml needs the corresponding bindings to connect the application to these resources:

[[ d1_databases ]]
binding = "DB"
database_name = "keyword-tracker-db"
database_id = "ab4828aa-723b-4a77-a3f2-a2e6a21c4f87"
preview_database_id = "8a77a074-8631-48ca-ba41-a00d0206de32"
	
[[queues.producers]]
  queue = "queue"
  binding = "QUEUE"

[[queues.consumers]]
  queue = "queue"
  max_batch_size = 10
  max_batch_timeout = 30
  max_retries = 10
  dead_letter_queue = "queue-dlq"

A prepared script handles the schema creation for both local and production databases:

bin/migrate

This also creates a local SQLite file at .wrangler/state/d1/DB.sqlite3, which you can inspect directly with the SQLite CLI to see the expected tables.

Deploying the worker completes the setup:

npm run deploy

Once running, the Workers URL serves the tool's interface. From there you can create notifiers and add URLs, and start receiving email alerts whenever a keyword match appears on new pages.

Why D1 beat KV for this workload

For sites with a large number of URLs to crawl, the choice of storage layer was decisive. Workers KV, Cloudflare's key-value store, would have made it awkward to model, retrieve, and update the data as the scraper needs to. D1, on the other hand, supports relational data models and lets us query only the exact rows needed for each queued processing task.

Putting the pieces together

Prospector demonstrates that applications once difficult to build on Workers without relational storage or background task tooling are now straightforward. D1 and Queues allow us to combine real-time user interfaces, geographically distributed data, and background processing using the same developer ergonomics and low latency Workers is known for.

With these tools, internal tools and applications for companies become more powerful and scalable than before. Pairing them with Cloudflare's Zero Trust suite keeps applications secure by default, while deployment lands them on Cloudflare's global network. The result is fast, secure, and reliable software without the operational burden of managing infrastructure.

Availability

The open-source code for Prospector shows how quickly such an application comes together — the full stack, including real-time data handling and background processing, was built in only a few hours. Feedback on the project is welcome, and questions can go to @cloudflaredev on Twitter or the Workers Discord community, which now has more than 20k members.