Accelerating researchers and developers building multilingual AI with a new open dataset

A Metadata Map of Multilingual Open Source

Software development is a collaborative act expressed in human language as much as in code. Developers document projects in READMEs, file bug reports and feature requests in issues, and critique changes in pull requests. Much of this activity happens in English, but a significant and growing portion does not. As AI-assisted development tools become standard, understanding how that multilingual collaboration is structured becomes a practical concern for researchers and tool builders.

To address this, GitHub has released the GitHub Multilingual Repositories Dataset, a repository-level metadata collection designed to help locate public repositories containing non-English natural-language content. The dataset is published on GitHub under the CC0-1.0 license, fulfilling a commitment made in 2025 through Microsoft’s European Digital Commitments to broaden access to multilingual data for open source AI development.

Repository Signals, Not Raw Text

The dataset is not a content dump. It aggregates metadata from over 40 million repositories, yielding more than 80 million classification rows. For each public repository, it provides language classifications for three key text sources:

• The README file
• The most-commented issue
• The most-commented pull request

Classification is performed on the first 150 characters of each text sample, with texts shorter than 20 characters excluded. Each sample is scored independently by three language identification tools—fastText, gcld3, and lingua-py—and only classifications with a confidence score above 0.5 are included. The dataset also carries standard repository metadata: creation timestamp, disk usage, star and fork counts, primary programming language, SPDX license, issue and pull request counts, and the snapshot date.

A notable finding from the dataset’s construction is that language distribution varies by text type. Korean is the most common non-English language in issue text but ranks only fifth in READMEs. Portuguese, by contrast, tops the non-English README list with more than 3 million repositories.

The decision to keep the three classifiers separate rather than merging them into a single label is deliberate. Classifiers differ in coverage and confidence calibration, particularly for lower-resource languages. Exposing all three results lets consumers set their own precision-recall tradeoffs—requiring agreement across all classifiers for high-precision subsets, or relying on a single classifier for broader recall in exploratory studies.

Use Cases and Constraints

The dataset is positioned as a discovery tool for work that general web text cannot support. Intended applications include:

• Discovering repositories likely to contain developer documentation or collaboration in specific languages.
• Studying how non-English developer communities engage with issues, pull requests, and READMEs.
• Building evaluation sets for AI coding tools, documentation generators, or review assistants that must perform across languages.
• Measuring the representation of European and other underrepresented languages in open source.
• Informing decisions about language coverage for new developer tools with data-backed arguments about multilingual diversity.

The dataset comes with explicit caveats. Short repository text—often containing badges, templates, commands, code snippets, or mixed-language content—poses a challenge for language identification. A 150-character sample may not represent the whole repository. The creators stress that the dataset is not a ground-truth benchmark for language identification, but a transparent tool whose classifications, confidence scores, and sources can be inspected and filtered. It is also not intended for inferring sensitive attributes about repository owners, contributors, or communities; the signals are repository-level, not person-level.

Context for Multilingual AI

Many European languages remain underrepresented in the text corpora used to build and evaluate AI systems. The resulting risk is that AI tools work unevenly across languages and communities. GitHub’s argument is that developer content—installation instructions, bug reports, feature requests, review comments—carries a distinct context that general web text lacks. Making that content easier to find and analyze gives researchers and model builders a more precise tool for studying language representation in software development and for building systems that understand how developers actually work.

The release will be discussed at the Open Innovation Dialogue Hub in Strasbourg on June 16, an event co-organized by the Microsoft Open Innovation Center, the Council of Europe, and GitHub, focusing on AI, linguistic diversity, cultural heritage, and open data. Feedback and discussion on the dataset are welcome via the project’s GitHub discussions.