Turning support tickets into product signals
GitHub’s support portal receives a steady stream of feedback from its developer community, and the volume makes manual review impractical. Data scientists already spend roughly 80% of their time on data collection and organization rather than analysis, according to a Harvard Business Review study. Manual classification of unstructured text is slow, error-prone, and leaves most insights untapped.
The company’s Customer Success Engineering team set out to change that by pairing data-mining techniques with machine learning algorithms. The goal was straightforward: extract, interpret, and analyze customer feedback at scale, then convert it into actionable guidance for product teams. Rather than rely on off-the-shelf analytics solutions, the team built an internal tool tailored to GitHub’s security requirements and product-specific business metrics.
From BERT embeddings to topic clusters
The first experiments used open-source AI models hosted on GitHub’s own platform. The team turned to BERTopic, a topic modeling framework that combines BERT embeddings with a hierarchical clustering algorithm like HDBSCAN. BERT’s transformer-based language model understands ambiguous text by deriving context from surrounding words, and its multilingual training data let the team analyze feedback across GitHub’s global user base without language-specific preprocessing.
Notably, the team did not train models on customer feedback from support tickets. Instead, pre-trained models were applied directly to the feedback text to generate insights.
BERTopic produced representative words for each cluster, which served as a starting point. But lists of keywords were not enough — the outputs needed to read as coherent, human-understandable statements of customer pain points.
Using GPT-4 to draft summaries
To close that gap, the team used GPT-4 to summarize each topic cluster into clear sentences. The LLM’s summarization capabilities were adapted without retraining, using three levers:
- Optimizing prompts. Refining prompts to steer the model toward relevant summaries.
- Setting parameters. Tuning temperature, max tokens, top-p, and frequency or presence penalties to control output.
- Iterative feedback. Improving results through human review and A/B testing.
Learning from early dashboards
Once the team had generative insights, the next challenge was presentation. Following GitHub’s “ship to learn” philosophy, they quickly built an Azure Data Explorer (ADX) dashboard and shared it company-wide to gather reactions. The experiment showed that generic visualizations without business context were not compelling enough — product teams needed to see stories like “the top 10 customer pain points in support tickets for product X.”
That feedback drove development of a custom internal web application. The team used layout, color, and size to steer attention to the most important findings, released a minimum viable product (MVP), and continued iterating based on how internal product teams actually used the visuals.
Where the insights land
The AI-driven analysis has directly changed how GitHub processes support data:
- Automated trend identification replaces manual classification. The shift scales analysis and improves precision in understanding developer feedback.
- Faster resolution of common pain points. Clustering surfaces recurring problems sooner, minimizing disruption for users.
- Better feature prioritization. Insights reflect what the developer community needs, guiding where engineering effort goes.
- Data-driven decisions. Summarized findings align internal teams with user needs.
- New self-service opportunities. The tool identifies where customers can resolve issues on their own, reducing dependency on direct support.
Closing the feedback loop
The project reflects a broader commitment: keeping developer voices central to product decisions. As GitHub refines its analytics pipeline and incorporates new techniques, the same principle applies — the insights are only as valuable as the action they inspire.
There is also a practical takeaway for users. Detailed feedback in support tickets feeds directly into this system, so specifics help the team make better-informed product choices.
