Why Secret Protection Was a Good Test Case for Copilot

GitHub's Secret Protection generates alerts when sensitive credentials appear in code. The feature spans several capabilities, from blocking commits with push protection to pointing out which leaked credentials are still active via validity checks. The partner program notifies providers of exposed tokens in public repositories, while Copilot secret scanning catches generic secrets like passwords, and custom patterns let teams define their own detection rules.

The validity check workflow became the focus of an experiment with coding agents. Validity checks work by testing a leaked credential against a low-impact API endpoint belonging to the token's provider. The feature launched in 2023 with coverage for the most common leaked token types, including AWS keys, GCP credentials, and Slack tokens, eventually validating around 80% of new alerts. The long tail of less common token types remained unsupported, however, and as agentic AI gained traction in late 2024 and 2025, the team asked whether coding agents could close that gap.

A Framework-Driven Workflow

Adding validation for a new token type follows a repeatable four-step process:

  1. Research the provider to identify an appropriate validation endpoint.
  2. Write a validator that implements the change.
  3. Darkship the validator to observe real-world behavior without writing validity inferences to the database.
  4. Fully ship the validator by removing the darkship configuration.
A diagram showing the framework-driven workflow of research, code, darkship (observe), and release. As needed, the workflow can repeat the "code" step after "darkship."

The code-writing and release steps were the natural first targets for automation. The research phase posed more of a challenge. Engineers typically prefer /me endpoints when available, but those are not always easy to locate in documentation. Copilot struggled to identify the same least-intrusive endpoints a human would pick, and tasks like creating live test tokens and interpreting subtle API changes stayed with engineers.

The team adapted by keeping research human-driven, then feeding results into a manually dispatched GitHub workflow that produced a detailed issue for the coding agent. That issue acted as a comprehensive prompt containing project background, links to API documentation, and examples. Since the agent sometimes failed to follow documentation links, an extra notes field was added to capture additional context.

Screenshot of the GitHub Actions “Run workflow” form to create a new validator. Form shows a dropdown for branch, and text fields for Token Type, Token Name, Provider, Documentation URL, Endpoint URL, Other Notes for Validator all with sample input using a PAT from Example.comas demonstration.

Once assigned, Copilot generated a pull request automatically, turning research into reviewable code. Agents' output went through the same pipeline as human code: automated tests, human review, and engineer-led deployment. Review feedback was handled by commenting on the pull request. The agent was not flawless — it occasionally missed details or failed to follow documentation as intended. Darkshipping caught drift between API documentation and actual behavior, and releasing a validator fully was simply a small configuration change requested from Copilot.

Measured Results

Before this experiment, the team had onboarded 32 partner token types over several months, with engineers balancing this work against day-to-day feature development. With Copilot, engineering interns supervising the agents onboarded nearly 90 additional types in just a few weeks. The key was parallelization: running multiple agents simultaneously against separate research outputs.

Coding agents proved viable for accelerating this kind of framework-driven, repeatable task. Copilot functioned as a force multiplier, but it did not replace engineering judgment. Agent-generated code still required review, testing, and verification, and the team deliberately grafted Copilot into narrow segments of the broader workflow rather than automating the entire process.

Operational Takeaways

  • Automation amplifies repeatability: Well-defined processes with clear steps are strong candidates for agent-assisted scaling.
  • Treat Copilot like a team member: Its contributions need the same careful review, testing, and feedback as any human's code.
  • Prompt quality drives results: Detailed prompts with examples, plus supplementary notes, produced higher-quality pull requests.
  • Iterate on your process: Prompts required refinement, and the workflow itself improved through small adjustments based on what worked.
  • Parallelization is a superpower: Many tasks could be dispatched at once, unblocking work that otherwise would have queued behind one engineer.

The team is now exploring similar agentic automation in other onboarding workflows within the project. For teams eyeing this kind of automation, the pattern is straightforward: find what is already repeatable in your engineering process, invest heavily in prompt quality, and build in review and verification at every stage.