When analytics self-service actually works
Making data access genuinely self-serve has been a decades-long industry aspiration, but the tooling never quite delivered. At GitHub's scale, dedicated analytics support for dozens of product teams isn't feasible, so those teams historically had to fend for themselves. Product telemetry is abundant, but knowing which data model to query, which grain or filters to apply, and how to validate the result typically required a human analyst.
Qubot is GitHub's internal answer: a Copilot-powered analytics agent that lets any employee ask plain-language questions about any data model in the warehouse and get an answer in seconds. It's not a reporting tool or a dashboard substitute. It targets exploratory queries like retention by user cohort or identifying which product most moved a metric last week. It's also essentially maintenance-free, which makes it viable for teams to ramp up quickly on unfamiliar datasets.
Three moving parts
Qubot's architecture splits into a user interface, a context layer, and a query engine.
Interfaces for different workflows
Qubot runs in Slack, VS Code, and the Copilot CLI. The Slack channel requires no setup and is where most Hubbers interact with it. A question posted there spawns a Qubot instance as a Copilot Cloud Agent on github.com; the answer returns in the thread, where users can iterate on the question or share results. Every run is also saved as a markdown report in a pull request, which users can reference to tweak the query or feed into a dashboard.
For users who want analytics inside their normal dev loop, Qubot installs as a plugin with a single command and shows up alongside other custom agents, skills, and tools in VS Code or Copilot CLI sessions.
Context tuned to the data tier
The warehouse contains three curation tiers: raw bronze events, silver conformed facts and dimensions, and gold datasets curated for specific business use cases. Context is federated by tier:
- Bronze data carries product-team telemetry context, including schema and metadata.
- Silver data has query examples, usage guidance, and mandatory filters maintained by the data and analytics team.
- Gold data comes with business rules and metric definitions from the owning teams.
The ETL pipelines systematically augment context with derived signals. At runtime, the GitHub MCP Server pulls context on demand.
Because GitHub primarily documents in markdown, teams don't have to interface with many tools to contribute. A context agent normalizes contributions from a standardized template or referenced repositories into a structured format that Qubot can consume efficiently.
Evaluating before shipping
Any change to context or agent configuration goes through offline evaluation before deployment. Contributors open a pull request, and a benchmark framework measures response accuracy, latency, and regression risk. The framework has three pieces:
- Test cases — curated prompts with known answers, ground-truth SQL, and metadata for domain and difficulty.
- Run orchestration — a script that launches each case as an agent task via
gh agent-task create, runs parallel trials, and collects detailed JSON results. - Stats aggregation — a reporting script computing per-case completion rate, accuracy, and average/min/max duration.
Picking the right engine
Qubot talks to both Kusto and Trino through MCP servers. Trino uses a custom MCP implementation; Kusto uses a local Fabric RTI MCP server. Kusto is optimized for fast exploratory questions over recent events; Trino handles complex joins and deeper historical analysis. Users don't pick — Qubot defaults to Kusto and switches when the question demands Trino.
What adoption revealed
Qubot now has hundreds of active users who have run thousands of queries. Activity in the data and analytics Slack channels has dropped sharply: people explore independently and only escalate genuinely complicated questions. The zero-configuration Slack interface makes that possible even for Hubbers who previously avoided the warehouse entirely.
The biggest finding is that context quality drives both accuracy and speed. Structurally curated context made Qubot three times faster at arriving at the right answer. That has implications for analytics engineering as a discipline: context becomes part of how data is modeled, not documentation bolted on afterward.
Qubot also reshaped how analytics work gets shared across the company. Product teams own bronze telemetry context and business teams own gold definitions, which relieves the central data team. The agent acts as a focal point that consolidates distributed knowledge into one tool — and gives contributing teams a reason to collaborate rather than building separate, domain-limited solutions.
