AI Agents Are a New Kind of Web User
Web developers have long optimized sites for both human visitors and non-human traffic like crawlers and bots. AI agents now represent the latest type of audience that benefits from careful site design. An agent is fundamentally a system that receives input, interprets it, and then plans and executes actions on behalf of a user, whether that user is a person or another agent. These systems are built from multiple components, potentially including models, APIs, and other tools.
While agents have been used for automation and information gathering for some time, their capabilities have expanded. They can now click links, fill in fields, scroll pages, and complete entire workflows on behalf of users. These tasks range from something simple like submitting a contact form to something complex like booking flights for a family. Crucially, a well-designed agent acts as a companion to its human user, asking for confirmation at critical junctures, such as when making a purchase or submitting a form with sensitive data.
The Core Loop of an Agent
From a high level, an agent’s operation follows a consistent four-step process:
- Receive the query.
- Process and plan how to address the query.
- Execute the plan.
- Save any lessons learned to memory.
In a web development context, it helps to think of agents through several defining characteristics: they are autonomous (operating without direct human intervention), interactive (conversing with humans and other agents), reactive (perceiving their environment and responding to changes), and proactive (taking initiative to meet specific goals). For example, an agent for an online bookstore could recommend a book, navigate a user to the product page, and even begin the checkout process. If the book is out of stock, the agent might take the user to another store.
Agents are particularly well-suited for tasks that span multiple origins. While an agent completes a shopping task on your site, it may be simultaneously navigating similar sites to achieve its user’s goal. Making your site easy for an agent to navigate increases the likelihood that it will complete the task with your origin.
Understanding Agent Infrastructure
An agent functions as a contained unit composed of several interconnected pieces:
- Model: A large language model (LLM) serves as the agent’s foundation, providing reasoning capabilities, a base of knowledge, and the ability to process and generate language.
- Rules: Constraints like a persona, specific instructions, and defined goals help the agent perform tasks consistently.
- Memory: Both short-term and long-term memory help the agent manage context, improve efficiency, and perform better for the user.
- Tools: Agents draw on a wide array of tools, including APIs, functions, databases, and even other agents. One example is WebMCP, currently in origin trial, which proposes a way to support structured interactions on your website.
When agents interact with web pages, they typically do so in one of two ways. Visual interaction involves the agent taking a snapshot of the rendered page and using a vision model to identify content and interactive elements. Semantic interaction involves the agent analyzing the DOM and reading text directly, a common approach for automated task completion. Both methods benefit from sites that are well-designed, intuitive to navigate, and have a clear content hierarchy.
Much of the early discussion around AI and the web focused on crawlers scraping data to train foundation models, with open datasets like Common Crawl helping to prevent sites from being overwhelmed by crawlers. However, now AI systems may target specific pages based on a specific user’s request, much like an agent acting on the user's behalf. The AI scrapes the content to help with related tasks, such as summarizing or aggregating data.
Data and the Three Agent Types
There is also a useful way to categorize agents based on their relationship to data: whether the agent's owner and the data's owner are the same. This distinction determines the required layers of authentication and the overall task complexity.
Zero-party agents
A zero-party agent is a browser-based agent acting in a local context with local data. Because browsers store custom user preferences that could be considered personally identifiable information (PII), this type of agent can prevent operations that would share this data with third parties.
First-party agents
A first-party agent exists when both the tool and the information it accesses are owned by the same party. In this scenario, the developer can own and support the tools while managing access to the information and configuration. An example is an agent for a mapping service that uses the service’s own map data, along with the preferences of a logged-in user, to build a suggested list of places to visit in a city.
Third-party agents
A third-party agent is developed by an external party and offers functions and data from external services. For instance, you might want a third-party calendar provider to enable event-based features on your website. You could offer specific tools to these agents or integrate them into your workflows, assuming they meet your security requirements.
Third-party agents can also complete tasks that rely on several different data sources. For instance, an agent designed to pull the best restaurant recommendations from local newspaper websites would need read access to each publication’s site, plus read and write access to the list creation tool it uses. Facilitating such tasks requires several layers of consent and permissions, along with specialized tools to interact with sites. It is increasingly likely that your site will function as a third-party information provider to an agent. In that case, establishing a clear permissions structure becomes key to letting both agents and humans complete tasks with your website.
Practical Next Steps for Developers
Though agents are a relatively new web user, your site can start preparing for them now. Many of the recommended practices overlap with general usability and will benefit all of your users.
- Review the WebMCP proposal and consider signing up for the origin trial.
- Prioritize building an accessible website, as this is a cornerstone of agent-friendly design.
- Take the Learn AI course to gain a deeper understanding of how AI systems integrate with your existing infrastructure.
