Threat Modeling for AI Agents

Shipping an AI agent means shipping more than just a useful tool—it means shipping a program that can take real actions, which expands the attack surface considerably. The more an agent can do, the higher the stakes if it fails. In building our hosted agents, we focus on three main categories of risk:

  • Data exfiltration: An agent with internet access might be tricked into sending repository data, or even credentials, to an unintended endpoint due to direct malicious intent or indirect manipulation.
  • Impersonation: It can be unclear whose permissions an agent should use. For instance, when an issue is assigned to the Copilot coding agent, is the directive from the person who filed the issue or the maintainer who assigned it? And if the agent makes an error, you need a clear record for accountability.
  • Prompt injection: Because agents pull context from various sources like issue comments and repository files, it’s vital that the initiating user sees all the instructions the agent is acting on. Hidden instructions in that context could otherwise manipulate a maintainer into running an agent under false pretenses.

Six Rules for Secure Agent Design

To counter these risks, we’ve codified six principles that apply to all our hosted agentic products.

1. Visible Context Only

If an agent is acting on information you can’t see, you can’t trust the outcome. A malicious user might, for example, embed invisible Unicode prompt injection instructions in a GitHub Issue. To prevent this, our coding agent filters out invisible or masked information via Unicode/HTML tags before processing, and displays the files it uses as context to the user.

2. Firewall the Agent

Unrestricted network access creates two major risks: data exfiltration and external prompt injection. We mitigate this for the Copilot coding agent with a firewall that lets users restrict network access. For usability, MCP interactions can be configured to bypass the firewall automatically. In contrast, features like Copilot Chat avoid the issue entirely by not auto-executing code—for example, rich HTML previews require a manual user action to enable execution.

3. Limit Access to Secrets

The most effective way to prevent data leaks is to not hold the data in the first place. Our agents only receive the context strictly required for the task. CI secrets and files outside the current repository are excluded by default, and even the agent’s own GitHub token is revoked at the end of a session.

4. Human Approval for Irreversible Actions

Mistakes happen, so we prevent agents from making changes that are costly to reverse without human intervention. For instance, the coding agent can open a pull request but cannot push directly to the default branch. These AI-generated pull requests will not run CI automatically; a human must review and manually trigger GitHub Actions. Similarly, MCP tool calls in Copilot Chat require explicit user approval.

5. Clear Attribution for Who Is Doing What

Every agent interaction is logged with a clear chain of responsibility. User-initiated actions are attributed to that user, while the agent’s own actions are labeled with its specific identity. When the Copilot coding agent creates a pull request, the commit is co-authored by the initiating user, but the PR itself is clearly marked as AI-generated by the Copilot identity.

6. Context Only from Authorized Users

An agent must operate with the lowest possible privileges. Our coding agent can only be assigned to issues by users with write access to the repository. Furthermore, to secure public repos, it only relies on comments from users who also have write access, preventing low-privilege users from feeding it instructions.

These rules are designed to be transparent and unobtrusive for end users. While they are built into our products, we hope this framework is useful for evaluating your own AI features. More details are available in our public documentation for the Copilot coding agent.