Generic secret detection forces a tradeoff: miss too much and real credentials stay in a repository; alert too often and developers waste time on strings that were never secrets. GitHub trained a classifier for its product needs that distinguishes real credentials from lookalikes while meeting the latency and cost requirements of scanning at scale.
Where scanning runs
GitHub secret scanning operates at two moments:
- Push protection scans changes before a push is accepted, so developers can address findings before the secret enters the repository.
- Post-push scanning runs after code is committed, continuously scanning repository content and alerting developers and security teams.
A fast classifier cut median post-push scan execution time by 86% versus the previous LLM-based pipeline. That efficiency made it practical to extend generic secret detection to push protection, returning results before the push rather than only after the commit.
Two-stage pipeline
Structured credentials often carry a distinctive prefix, length, alphabet, or checksum. Generic secrets may not, and their syntax overlaps with build identifiers, checksums, asset hashes, sample values, and other non-sensitive strings. Context is the deciding signal.
| Likely real credential | Likely harmless lookalike |
|---|---|
| A password assigned beside a production database URL | The same password-shaped value in a test fixture |
| A connection string in deployment configuration | A localhost connection string in a setup guide |
| A high-entropy value bound to an API credential | A high-entropy checksum, build ID, or asset hash |
The new scanner runs two stages:
- Candidate generation. Regex filters scan content with high recall, catching broad families of credential-like patterns quickly and deterministically.
- Context-aware classification. A fine-tuned ModernBERT classifier evaluates each candidate together with its surrounding context. It is trained on the signals that separate real secrets from false positives — placeholders, examples, documentation snippets, generated content, non-sensitive identifiers. Because the model is small and cheap to serve, it scales more simply than an LLM-based pipeline.
Impact:
- 11.5% higher user acceptance, reducing false positives.
- 86% lower median post-push scan execution time.
- More than 98% lower cost versus the LLM-based pipeline.
The model does not replace deterministic scanning; it complements it. Regexes remain the right tool for fast candidate discovery, and the classifier handles the part where machine learning provides the most leverage: distinguishing “looks like a secret” from “is likely a real secret.”
Scanning flow
Regex patterns cast a wide net and flag anything credential-shaped so real secrets aren’t missed at the start. Each candidate then goes to the fine-tuned ModernBERT classifier, which reads the value alongside its file path, variable names, and nearby code and estimates the likelihood that it is a real secret. High-confidence findings become alerts; lookalikes such as placeholders, test fixtures, and checksums are filtered before reaching developers.
Hard negatives
Most false positives are not random. They are hard negatives: strings that match a credential pattern but are not usable credentials — documentation placeholders, toy examples in tests, redacted values, non-sensitive IDs, generated fixtures. Regexes match them easily; purely rule-based systems struggle to suppress them without also losing real secrets. Adding rules can suppress some, but each exception risks hiding a real secret.
ModernBERT reads the candidate alongside nearby code and text, providing contextual understanding comparable to an LLM without the latency and computational overhead of generating natural-language output. A candidate surrounded by words like “example,” “dummy,” “fake,” or “replace-me” may be lower risk than one assigned to an environment variable in a deployment file. The classifier learns these contextual differences from labeled examples instead of requiring engineers to encode every exception as another rule.
The model needed to make a calibrated decision over short candidate windows with high throughput and low latency. A compact, fine-tuned encoder classifier is a better serving shape than a general-purpose LLM for this pipeline stage: it can be trained directly on the decision boundary and evaluated with the recall and precision metrics that matter for secret scanning quality.
Training data and strategy
The challenge was representative data for cases where syntax alone misleads. Training and evaluation pull from three sources:
- Product-derived examples from GitHub repositories, using feedback from the secret detection experience plus additional human validation.
- High-recall regex candidates from GitHub repository data, including human-annotated examples and weak labels used to expand coverage.
- Synthetic examples targeted at sparse and difficult cases: documentation secrets, test fixtures, public keys, checksums, structured lookalikes, near-miss pairs.
Synthetic rows pass deterministic validation and model-based quality review before inclusion. Each input includes the candidate value, file path, secret type, and a bounded amount of surrounding context. Before splitting, rows sharing a repository, candidate value, or example were grouped so near-duplicates could not land in both training and evaluation.
Training emphasized the cases that drive alert quality: real secrets, realistic false positives, and the hard negatives regexes match most often. The principle was to make the model see the confusing examples — candidates that are syntactically similar to secrets but semantically different — so it learns what a false positive looks like in context, not just what a secret looks like.
Thresholds are selected through offline evaluation and then validated on production-like traffic before broader rollout. The trained model is exported to ONNX format with 8-bit quantization for fast inference.
Measured results
| Online result | Change vs. previous LLM pipeline | What it measures |
|---|---|---|
| Repository owner-weighted acceptance | 11.5% higher | Customer acceptance of detected findings |
| Median post-push scan execution time | 86% lower | Production scan execution time |
| Serving cost | More than 98% lower | Cost of the compact classifier compared with remote LLM calls |
Evaluation ran in two stages: first against the production baseline on the same public traffic, then through a customer A/B experiment.
- Offline: on post-push scanning, the new pipeline reached 13% higher precision and 19% higher recall than the previous LLM pipeline on the same public traffic.
- Online: an A/B experiment against the previous LLM pipeline showed repository owner-weighted acceptance up 11.5%. Weighting by repository owner reduces the influence of customers who bulk-resolve large volumes of alerts, which can distort a simple average.
What generalizes
Regex supplies speed, transparency, and broad candidate coverage; ModernBERT supplies learned context understanding and false-positive reduction. For this product constraint, the combination outperforms either approach alone, with fewer distracting alerts, more real secrets caught, and lower cost to keep expanding coverage.
Model size is not the same as model usefulness. A smaller fine-tuned classifier can be the right answer when the task is narrow, the decision boundary is measurable, and serving efficiency is critical.
Two areas are next: building a continuous improvement flywheel that turns production feedback on new false negatives and false positives into higher-quality training and evaluation data, and exploring architectures that perform secret detection and extraction in a single step, reducing pipeline complexity while preserving accuracy and scale.
