Machine Learning Extends CodeQL’s Reach in GitHub Code Scanning
GitHub code scanning now uses machine learning to flag potential security vulnerabilities. The new capability sits on top of the existing CodeQL analysis engine, which has long detected issues by executing manually written queries against a relational database built from source code. The ML component is designed to catch problems that those hand-crafted queries miss, particularly when vulnerable code relies on uncommon libraries or private code that the manual models have never seen.
CodeQL queries encode expert knowledge about likely sources of untrusted user data and high-risk sinks—such as functions that execute SQL queries or write to a web page. Each query targets a specific vulnerability pattern, like SQL injection or cross-site scripting. But manually modeling every possible library and framework is impractical. The ML system trains deep learning networks on examples surfaced by the manual queries to recognize code snippets that constitute risky sinks, including those involving unfamiliar or closed-source libraries.
Turning Manual Queries Into a Training Corpus
Supervised learning required a large labeled dataset, and security experts can’t feasibly hand-label millions of snippets. Instead, GitHub uses the manually written CodeQL queries themselves as ground-truth oracles. Sinks detected by each query become positive training examples; snippets not flagged by the queries serve as negatives—noise that is offset by the sheer volume of data.
The training pipeline extracts tens of millions of code snippets from over one hundred thousand public repositories, labels them by running the CodeQL queries, and feeds the result into a classifier that scores snippets for each vulnerability type. Notably, the training set is built using an older, less-comprehensive version of each query. That setup simulates the real deployment scenario: the model trained on current query results can be evaluated by its ability to recover alerts the current manual query would have missed.
Feature Extraction and Model Design
The snippets are not treated as raw text strings for naive NLP classification. Instead, CodeQL extracts a rich set of structural features from the underlying code, capturing the kind of context a security expert would consider: the enclosing function body, the access path and API name, or even the argument index for a snippet passed to a function. Some features aren’t obviously useful to humans, but the neural network can identify hidden patterns in them.
These features are tokenized and sub-tokenized, with adjustments for code-specific syntax. A vocabulary is generated from the training data, and indices into that vocabulary are fed into a deep learning classifier with several layers of per-feature processing, followed by concatenation and combined layers. The output is a probability that the snippet is vulnerable for each query type. Offline labeling, feature extraction, and training run at scale in the cloud using GPUs, but inference requires no GPU.
Running Inference Standard Action Runners
When ML-generated alerts are enabled, CodeQL computes source-code features for the snippets in the codebase and applies the trained classifier. The framework returns a vulnerability probability for each snippet and surfaces the most likely new alerts. The entire process executes on the same standard GitHub Actions runners used by other code scanning jobs; users may notice only a modest runtime increase on large repositories.
ML-generated alerts appear alongside those from the manual queries, each tagged with an “Experimental” label so they can be easily filtered in or out.
Evaluating Against Missed Alerts
GitHub measures the ML framework by focusing only on alerts that the manual queries did not catch. True positives are detected vulnerabilities the older query version missed; false positives are incorrect new alerts. Using the experimental setup that trains on older query labels, the average metrics across queries show a recall of roughly 80% and a precision near 60% on non-training repositories.
The ML-generated alerts are currently being extended to more JavaScript and TypeScript security queries, with ongoing work to improve throughput and runtime. Future plans include support for additional programming languages and broader generalizations to capture a wider range of vulnerabilities. Repository owners can enable the experimental queries to surface more potential issues, and the feedback provided on generated alerts feeds back into improving the algorithms.
