Here’s how academic research is shaping GitHub Discussions

We strive to understand how developers collaborate and work on GitHub, and we sometimes partner with academics to better understand how we can improve our products. Here’s how we did that to build and evolve GitHub Discussions.

EL

Evi LiuGitHub

·July 20, 2022

Here’s how academic research is shaping GitHub Discussions

Research partnerships behind GitHub Discussions

Open source projects and product teams often start with a problem, then move into a research phase to see what solutions exist and how they compare. At GitHub, that research takes two forms: short internal studies that can quickly shape product decisions, and larger academic collaborations that surface patterns worth building around.

Three academic studies have directly influenced how GitHub Discussions—the company’s community forum tool—has been designed and evolved since its beta release in 2020. Here is what those studies found and how the findings translated into product changes.

Validating the case for a separate forum space

GitHub Discussions grew out of an internal hypothesis: developers and maintainers were using GitHub Issues as a catch-all for conversations that lacked a clear, actionable outcome. That made it harder to get work done as communities scaled, and switching between chat apps, external forums, and GitHub fragmented those communities.

A research team led by Dr. Hideaki Hata, an associate professor at Shinshu University in Japan, tested that hypothesis with a study of early GitHub Discussions adopters. Hata’s previous GitHub research covered the Sponsors program and the link between academic papers and GitHub; this study examined whether Discussions actually created a better home for community conversation.

The findings supported the original bet. Discussions did make it simpler to separate community chatter from issue-tracked work. The study also found communities used Discussions to plan upcoming work, and that this planning had a positive effect on community productivity.

Designing dashboards from maintainer feedback

A central goal of GitHub Discussions was making maintainers’ lives easier. Early on, the team built a community insights dashboard meant to surface community health metrics—contribution activity, page views, daily contributors, and new contributor counts. It was a start, but the team knew it needed more.

A collaboration with Dr. Denae Ford, a senior researcher at Microsoft Research, and Mariam Guizani, then a Ph.D. student at Oregon State University interning at Microsoft Research, focused on what would help maintainers attract and retain newcomers. Their paper, co-authored with Tom Zimmermann and Anita Sarma and presented at the 2022 International Conference on Software Engineering, included maintainer-tested design mockups for the dashboard.

A screenshot of the maintainers’ dashboard view in GitHub Discussions.

Those mockups proposed graphs for new contributor activity, overall community activity rates, contributor retention trends, and a way for maintainers to spot “rising contributors” active within a project.

The real scale of duplicate discussions

Early feedback from some maintainers pointed to a recurring pain: answering the same questions over and over. But internal qualitative interviews with contributors did not surface duplication as a widespread problem—likely because of who was interviewed. That gap showed the limits of qualitative research on a platform with millions of developers of wildly different experiences.

Two academic researchers quantified the issue. Márcia Lima, a Ph.D. candidate at Universidade Federal do Amazonas and faculty member at Universidade do Estado do Amazonas, and Dr. Igor Steinmacher of Northern Arizona University, working with Tayana Conte and Bruno Gadelha, applied a Sentence-BERT model to detect related and duplicate posts in OSS communities.

The scale was striking: in a single project, the model found 151 related discussion posts, of which 81 were clear duplicates. Some developers even triplicated their posts on the same day. The motive was attention—duplicating boosted the odds of getting a response, but it degraded the experience for maintainers and other contributors alike.

That finding redirected the product roadmap. GitHub is now shipping features that let maintainers and Discussions administrators quickly identify and mark duplicate discussions, with plans to automate the process further.