Conversational Search Is Becoming a Baseline Expectation

Keyword search has been the default way users navigate websites for decades. The model assumes a visitor will type a query, scan a list of links, and click through pages to assemble an answer. But that assumption no longer holds. People who are used to asking ChatGPT, Claude, or Copilot a direct question now expect the same from any website they visit. And they aren’t the only new kind of visitor: AI agents are arriving at websites too, and they face the same friction with keyword-based navigation.

For agents, the problem is compounded. They must issue keyword queries, crawl through links, and scrape pages to extract reliable answers. What they need instead is a structured way to ask questions and receive consistent, trustworthy responses. That means websites need to offer controlled, agent-friendly access to their content — not just a better search box.

A Protocol and a Pipeline for the Agentic Web

Meeting the expectations of both humans and agents requires more than incremental upgrades to search. It calls for a model where conversational access is treated as a native part of the web. Cloudflare is pursuing that goal by combining an open standard — NLWeb — with its managed retrieval engine, AutoRAG.

NLWeb is an open project from Microsoft that defines a standard protocol for natural-language queries on websites. Each NLWeb instance also functions as a Model Context Protocol (MCP) server. Cloudflare is building to the NLWeb spec and collaborating with Microsoft to extend the standard, with the aim of letting any site behave like an AI application that can answer natural-language questions about its contents.

AutoRAG handles the infrastructure side. It crawls a website, stores the crawled content in R2, and embeds it into a managed vector database on Vectorize. The index stays current through continuous re-crawling and re-indexing. Model inference and embeddings can run through Workers AI, and each AutoRAG instance is paired with an AI Gateway for observability into model usage. The result is a complete, managed RAG pipeline that removes the need for custom infrastructure.

R.V. Guha, creator of NLWeb and CVP and Technical Fellow at Microsoft, described the combination this way: “Together, NLWeb and AutoRAG let publishers go beyond search boxes, making conversational interfaces for websites simple to create and deploy. This integration will enable every website to easily become AI-ready for both people and trusted agents.”

One Click from Domain to Conversational Site

The setup is designed to be minimal from the user’s perspective. After selecting a domain in AutoRAG, the system crawls and indexes the site for semantic queries. A Cloudflare Worker is then deployed as the access layer, implementing the NLWeb standard and UI from the NLWeb project. That Worker exposes content in two ways:

  • /ask endpoint — serves the conversational UI at the root / and an embeddable preview at /snippet.html. It supports chat history and automatic query decontextualization to improve retrieval quality.
  • /mcp endpoint — implements an MCP server that trusted AI agents can connect to for structured access.

Site content becomes immediately available for experimentation through a conversational interface for visitors and a structured MCP interface for trusted agents. For those who prefer to run their own deployment of the NLWeb project, AutoRAG can serve as the retrieval engine powering that instance.

What Happens Behind the Single Click

Turning a site conversational involves a full retrieval pipeline running in the background:

  1. Crawling and ingestion — AutoRAG respects sitemap.xml and robots.txt files, then follows the sitemap to discover pages (up to 100k). Browser Rendering loads each page to capture dynamic JavaScript content. Crawled pages are stored in an R2 bucket in the user’s account before ingestion.
  2. Continuous indexing — Content is parsed and embedded into Vectorize for semantic search, going beyond keyword matching. Re-crawls and re-indexing run automatically to keep the knowledge base aligned with the latest site content.
  3. Access and observability — A Worker is deployed in the user’s account as the access layer implementing the NLWeb protocol. Workers AI powers summarization and query decontextualization. Support for connecting models from any provider — selectable directly in the AutoRAG dashboard — is planned via the AI Gateway and Secret Store BYO keys.

Making Websites a First-Class Data Source

AutoRAG previously supported only R2 as a data source, which worked for structured files but treated websites as second-class citizens. Building website crawling into AutoRAG meant strengthening the system to handle large, dynamic web sources reliably.

The first step was improving visibility into data syncs. Prior AutoRAG users couldn’t see when indexing runs happened or whether they succeeded. A new Job module tracks all syncs, stores history, and provides logs, supported by two new Durable Objects:

  • JobManager — runs a complete sync, including queuing files, embedding content, and keeping Vectorize up to date. Only one JobManager runs per RAG at a time, enforced by the existing RagManager, which cancels running jobs before starting new ones — whether triggered manually or by a scheduled sync.
  • FileManager — addresses scalability issues from Workers running out of memory during parallel processing. Originally a single Durable Object handled multiple files, but that hit a 128MB memory limit quickly. Now JobManager distributes files across many FileManagers, each responsible for a single file. Processing 20 files in parallel through 20 FileManagers expands effective memory from 128MB to roughly 2.5GB per batch.

The website crawler was built by reusing the existing R2-based queuing logic. A JobManager starts by reading the sitemaps associated with the RAG configuration. Instead of listing objects from an R2 bucket, it queues each website link into the existing queue, using the full URL as the R2 object key. A FileManager picks up the job, checks whether website parsing is configured, crawls the link, and places the page’s HTML into the user’s R2 bucket. Indexing then proceeds as with any other data source. This approach maximizes code reuse, so improvements to HTML-to-Markdown conversion benefit both file-based and website-based RAGs.

Getting Started

Enabling conversational search on a site requires just a few steps:

  1. In the Cloudflare Dashboard, go to Compute & AI > AutoRAG.
  2. Select Create, then choose the NLWeb Website quick deploy option.
  3. Select the domain from your Cloudflare account to index.
  4. Click Start indexing.

Once indexing completes, the site is available through an NLWeb search experience via a provided link, plus an embeddable snippet for testing how it looks on the live site.