AI-Generated Fixes for CodeQL Alerts

GitHub’s code scanning autofix, now generally available, uses AI to suggest fixes for security vulnerabilities identified by CodeQL analysis. The feature emerged from work announced in November 2023 and targets alerts detected in pull requests for JavaScript and TypeScript projects. When an alert is raised, autofix provides a natural-language explanation of the problem, displays a suggested patch in the pull request UI, and lets developers commit, dismiss, or edit the proposal.

Code scanning itself analyzes repository code on a schedule or when events like branch pushes or PR openings occur. GitHub’s first-party alerting uses CodeQL, a semantic analysis engine that treats code as queryable data. Queries developed by in-house security experts detect vulnerabilities across multiple languages. Autofix extends this detection by sending the affected code and a problem description to a large language model (LLM), asking for edits that resolve the vulnerability without altering code functionality.

Anatomy of the Prompt

Constructing the LLM request involves pulling information from the CodeQL alert, which points to the problematic code location and any related flow paths—for example, showing how untrusted data reaches a SQL query without sanitization. The prompt includes:

  • General vulnerability information and a canonical example fix, drawn from the CodeQL query help.
  • The source location and content of the alert message.
  • Relevant snippets from flow-path locations and other referenced code.
  • A specification of the expected response format.

The model replies with Markdown structured in three parts: natural-language remediation instructions, a precise list of code edits in a defined format, and any external dependencies the fix requires (such as a third-party sanitization library). GitHub renders the explanation alongside the alert and builds a diff patch from the edit specification, which users may review, tweak, and commit.

Prompt Engineering and Post-Processing

A simple prompt-to-patch pipeline works for demos but not for production. Practical deployment demands careful context selection and error-correction heuristics.

Choosing the Right Code Snippets

CodeQL alerts include the sink location and, for data-flow issues, the path from source to sink, plus any extra spots mentioned in the alert message. Each of these may need edits. Region-selection heuristics choose a surrounding snippet that preserves necessary context—including file-level imports and definitions, which frequently require augmentation—while minimizing token count. If multiple alert locations share a file, the tool combines them into one contextual snippet. Code is presented with line numbers so both prompt and response can reference specific lines. To limit hallucination, the model is constrained to edit only the code shown in the prompt.

Handling New Dependencies

Fixes sometimes require adding a package not already listed in the project’s configuration. Rather than rely on the LLM to find and edit manifest files (at a heavy token cost), the system asks the model to name required dependencies as plain text. Language-specific heuristics then locate the manifest, check whether the package is already declared, and add the appropriate entry to the patch.

Edit Format That Avoids Diff Arithmetic

Asking the model for a classic unified diff proved unreliable—LLMs make arithmetic mistakes, especially with line numbers, and the output lacks the context needed for correction. The format that worked best uses paired “before” and “after” code blocks, showing the original snippet (with context lines) and the proposed replacement.

Catching and Fixing Imperfections

Model output rarely arrives clean. Post-processing applies fuzzy matching to reconcile “before” blocks with actual source code, tolerating whitespace, semicolons, and comment discrepancies while silently adjusting misreported line numbers. A parser verifies syntax, and semantic checks confirm names resolve and types are consistent. Failures that can’t be corrected automatically mark the suggestion as suspect. Dependency suggestions are validated against the ecosystem’s registry, including a scan for known vulnerabilities and malicious package names.

Measuring What Works

Moving from prototype to product required testing fix quality over a large sample. The evaluation framework combines an offline collection pipeline and a live GitHub Actions workflow.

The collection phase scans open source repositories, yielding over 1,400 JavaScript/TypeScript alerts with test coverage, spanning 63 CodeQL queries. The workflow component then applies autofix to each alert in a fork, runs CodeQL and the repo’s test suite, and judges a fix successful only if it:

  • Eliminates the original alert without introducing new ones.
  • Causes no syntax errors.
  • Leaves all preexisting test outcomes unchanged.

This harness supported rapid iteration on the prompt, edit format, and heuristics, paired with periodic manual triage to target recurring failure modes and validate the automation’s accuracy. The results were substantial: the success rate tripled while LLM compute costs fell by a factor of six.

Serving Fixes to Users

The front end integrates with the existing code scanning pull request view. A fix suggestion may span multiple files, including locations outside the PR’s diff, along with an attached plain-English explanation. Developers can apply the changes as a commit or send the suggestion to an IDE or GitHub Codespace for additional editing before committing.

Under the hood, the fix generator runs as a CLI tool that the code scanning backend invokes after a SARIF upload, provided the alert language is supported. The tool assembles the prompt using SARIF data and source snippets, calls an internally hosted Azure-based LLM API, and passes the response through a filter against harmful content. Post-processing yields the final suggestion, which is stored and served alongside the alert. Suggestions are cached when possible, reducing repeated LLM calls.

Beyond standard security procedures, the feature was deliberately hardened against AI-specific threats such as prompt injection. Red-team testing was conducted against the response filters and other safeguards, examining risks from harmful output and model bias.

Measuring Autofix in Production

Before opening autofix up broadly, the team needed visibility into how the feature performs under real-world conditions. Prompt and model-response content stays out of telemetry because it may include private user code. Instead, the system tracks anonymized, aggregated interaction signals: the share of alerts that produce a suggestion, how often a suggestion gets committed to the branch exactly as offered, whether it is applied through the GitHub CLI or Codespace, how often it is dismissed, and the fix rate for alerts with suggestions versus those without. During the beta, these metrics will indicate whether the generated patch is genuinely useful to developers.

The service is also monitored for operational problems — specifically overload on the Azure model API and activations of content-safety filters. The goal is a stable, consistent user experience before autofix moves beyond the beta into unlimited usage and, later, general availability.

Current Status and Next Steps

With autofix available to a growing beta audience, the immediate priorities are collecting user feedback, resolving rough edges, and reviewing the metrics to validate the suggestions in live security scenarios. In parallel, the rollout is extending to additional languages and use cases, with UX improvements in progress. Developers interested in joining the public beta can sign up here, and further updates are planned as the feature evolves.