Using machine learning to surface beginner-friendly issues

GitHub has shipped an updated version of its good first issues feature that combines label-based recommendations with machine learning (ML) classifiers. The goal is to help new contributors find easy issues in projects they care about, while reducing the manual triaging burden on maintainers.

The initial launch in May 2019 relied entirely on issue labels applied by maintainers. A curated list of roughly 300 label names—synonyms for “good first issue” or “documentation” such as “beginner friendly”, “easy bug fix”, and “low-hanging-fruit”—was derived from popular open source repositories. Documentation issues were included because they are often a good entry point, but they were ranked lower than issues explicitly tagged as beginner-friendly.

That approach had a significant limitation: only about 40 percent of recommended repositories had easy issues that could be surfaced. It also placed the burden of labeling on maintainers. The updated version adds ML-based detection, expanding coverage to roughly 70 percent of recommended repositories.

Coverage comes at the cost of accuracy, a classic precision-recall tradeoff. Since only a tiny fraction of all issues qualify as good first issues, the system is tuned for very high precision even if recall suffers. That keeps the recommendation feed from being flooded with false positives. The exact thresholds and numbers evolve as training data and models improve and as feedback adjusts the tradeoff.

Building the training set without manual labeling

Supervised machine learning normally starts with a hand-labeled dataset, but labeling issues as beginner-friendly requires expertise across many topics, projects, and programming languages. Instead, GitHub took a weakly-supervised approach, automatically inferring labels for hundreds of thousands of candidate samples.

Positive samples start with issues carrying any label from the curated list. That set is too small and narrow on its own, so it is supplemented with issues that are likely beginner-friendly by other signals: issues closed by a pull request from a user who had never contributed to the repository before, and issues closed by a pull request touching only a few lines in a single file.

Because the goal favors missing some good issues over surfacing many false positives, every issue not detected as a positive sample is treated as a negative sample in training. This creates a hugely imbalanced set, handled by subsampling negatives and weighting the loss function. Near-duplicate issues are removed, and training, validation, and test sets are split across repositories to prevent data leakage.

Training classifiers on issue text alone

To detect good issues as soon as they are opened, the classifiers use only issue titles and bodies. They do not rely on conversation or other signals that arrive later. Titles and bodies are preprocessed and denoised—for instance, removing segments likely originating from issue templates, which carry no signal for the problem.

Several model architectures were tried, from classical methods like random forests on tf-idf vectors to deep learning models such as 1D convolutional and recurrent neural networks. The neural models, implemented in TensorFlow, use one-hot encodings of titles and bodies fed into trainable embedding layers, with features from the two inputs concatenated near the top. They consistently outperform the classical methods because they exploit word ordering, context, and sentence structure rather than unordered word counts.

Since both training and inference run offline, the higher computational cost of deep learning is manageable. However, the positive training set is small, so textual data augmentation proved crucial, along with regularization and early stopping.

Ranking and surfacing recommendations

For each open issue in a non-archived public repository, the trained classifier predicts a probability. Issues above the required threshold are slated for recommendation with a confidence score equal to that probability. Issues carrying any label from the curated list also qualify, with label-based confidence derived from label relevance—synonyms of “good first issue” score higher than those of “documentation.” Label-based detections are generally trusted more than ML-based ones.

Within each repository, issues are ranked primarily by confidence score, with a penalty applied for issue age. The data acquisition, training, and inference pipelines run daily using scheduled Argo workflows to keep results fresh. As the first deep-learning-powered product on GitHub.com, the feature required infrastructure designed to generalize to future ML projects.

What is next

Iteration continues on training data, pipelines, and classifier models. GitHub is also improving repository recommendation signals so users find projects aligned with their interests, and plans a mechanism for maintainers and triagers to approve or remove ML-based recommendations in their repositories. Personalized recommendations for contributors who have already made commits are also in the works.