A public repo with a private history
When GitHub open sourced docs.github.com last week, it wasn't just the documentation content that went public: the Node.js web application that renders the site, the automation workflows that keep it updated, and the tooling built around it all landed in github.com/github/docs as well. That makes it the first private production service GitHub has migrated into the open.
The motives for going public were practical. The docs team is small, and external contributors bring diverse ideas. Open sourcing the project also gives GitHub a chance to model how enterprises can run production services in the open — complete with automation workflows and contribution channels that other organizations can learn from. There's also a cross-organization angle: the Node.js internationalization working group faces the same localization challenges that GitHub Docs does. Both projects use GitHub repos, GitHub Actions, and Crowdin, and the hope is that sharing tooling and practices will benefit both communities. The @crowdin-node organization already hosts tools maintained by folks from Electron, GitHub Docs, and Node.js.
Opening the repo has an operational benefit for the team as well. When docs engineers file support requests with vendors like Fastly, Crowdin, Algolia, or Heroku, they can now link directly to public code or GitHub Issues instead of describing problems in private support channels. Vendors can clone the repo, test fixes, and even submit pull requests themselves — which leaves a useful public record of how problems get solved.
From Ruby to Node.js
docs.github.com has been around in some form since 2013. It started life as help.github.com, a Ruby on Rails app, then moved to a static site generated by Jekyll and later by Nanoc. Today it runs on a Node.js web service with dynamic routing and content rendering.
Despite the platform changes, several authoring conventions from the Jekyll era have been carried forward:
- Content lives in Markdown files.
- Liquid templates inside Markdown handle dynamic data rendering.
- Structured JSON and YAML files in a data directory are referenced from templates.
- Markdown files carry key-value metadata in YAML frontmatter.
What got open sourced
The Node.js rebuild was done with an eye toward extracting reusable pieces. Alongside the main repo, several components were open sourced as standalone projects:
- github/docs — the content and application that powers docs.github.com
- github/repo-sync — GitHub Actions for syncing git repositories
- github/rest-api-description — OpenAPI description files for GitHub's REST API
- docs/liquid — a Node.js implementation of the Liquid template language
- docs/render-content — a Node.js module for rendering Markdown and Liquid
- docs/frontmatter — a Node.js module for parsing and validating YAML frontmatter
- docs/data-directory — a Node.js module for loading structured data from directories
Keeping public and private in sync
The trickiest engineering problem was figuring out how to keep the docs public while still being able to collaborate privately on upcoming product releases. The solution was two git repositories — one public, one private — kept in sync automatically.
No existing tool in the GitHub Marketplace quite fit the bill. The closest was Pull, a GitHub App that automates keeping forks in sync with their upstream repos. GitHub Docs engineers reached out to Pull's author, Wei He, to see if he'd be interested in building something similar — but for bi-directional syncing. He agreed, and the collaboration produced Repo Sync, an open source tool built with Docker, git, shell scripts, GitHub Actions, and the GitHub Container Registry.
Repo Sync is a set of flexible GitHub Actions. A single workflow file runs on a schedule to keep the main branch of both the public and private docs repos in sync without manual intervention.
Reverse-engineering the REST API docs
When the help.github.com and developer.github.com sites merged into docs.github.com earlier this year, the REST API reference documentation was in rough shape. It was a mix of unstructured Markdown, embedded Ruby, Liquid templates, and hand-pasted cURL output. There were no machine-readable specs for an API that had been around for over a decade — the unstructured docs were effectively the source of truth.
Working with Octokit maintainer Gregor Martynus and contractors from Redoc.ly, the team reverse-engineered the REST API docs into OpenAPI description files. As of earlier this year, those descriptions now live in the github.com codebase itself, where they're used to create, validate, and test the REST API. The same OpenAPI files generate the JavaScript and Ruby Octokit clients and power the new REST API reference docs at docs.github.com/rest. The files are available for download at github.com/github/rest-api-description.
Liquid for Node.js
The docs content relies heavily on Liquid, the template language created by Shopify. It's a Ruby project and widely used in the Rails and Jekyll ecosystems, but there was no complete npm implementation that met GitHub's needs. Migrating thousands of content files to another template language wasn't appealing — the technical writers were comfortable with Liquid, and the engineering effort to convert everything would have been massive.
Instead, the team reached out to the authors of existing Liquid-related npm packages. With contributions from several people, they deprecated older unmaintained packages, rebranded liquid-node as liquid, converted the codebase from CoffeeScript to JavaScript, and improved the tests and documentation. The resulting liquid package now lives at github.com/docs/liquid.
With the site now public, the workflow that produced it is visible too — including the localization pipeline. The docs are currently translated into Japanese, Simplified Chinese, Spanish, and Brazilian Portuguese, with more languages planned. The translation work is done by professional translators today, but opening up the contribution process to external translators is on the roadmap.
Automating the contributor workflow
The Docs team runs on automation. Whenever a task becomes repetitive or error-prone, we try to replace it with a script or a bot. That philosophy extends to how we handle contributions on github/docs: we aim for a workflow where anyone can propose a change, preview it, and ship it without ever leaving github.com.
Following GitHub Flow, every pull request opened against the repository triggers an automatic deployment to an ephemeral staging app. Reviewers can interact with the proposed changes in a live environment instead of checking out the branch locally. Once the pull request merges into the default branch, that temporary app is torn down and the changes go straight to production. This "hands-free" continuous delivery model has been in place for over a year, and we have no interest in going back to managing finite staging instances or relying on chatops commands.
External contributors get the same experience as internal employees. When someone outside GitHub opens a pull request, the same continuous integration tests run and the same staging app is created. There is no separate, reduced pipeline for outside contributions.
Recognizing every kind of contribution
Healthy open source projects depend on more than just code commits. Maintainers need to foster a welcoming environment, enforce a Code of Conduct, and visibly appreciate the people who help in non-technical ways.
To do that, we use All Contributors, a project that provides a specification, a GitHub bot, and a command-line tool for crediting contributors. It supports a wide range of contribution types:
All docs-related open source repositories now maintain contributor lists powered by All Contributors. You can see an example in the github/docs README.
Looking ahead
Opening up this long-lived private project took considerable time and effort, but the hard part is behind us. Working in the open is the most enjoyable way to build software, and we are eager to see how docs.github.com grows as contributors from around the world shape its direction.
