Slack’s Internal AI Tooling: How 10,000+ Hours Are Saved Per Year

Slack’s engineering organization has been running a broad internal experiment with generative AI, deploying a set of developer-facing tools that are now estimated to save more than 10,000 hours of engineering time per year. The tools are built on top of an internally managed LLM platform, and they are subject to the company’s usual security, data protection, and compliance requirements.

The effort is not a single product. Slack has shipped a handful of distinct interfaces—an escalation bot inside Slack, a custom in-Slack AI assistant, a web app for deeper LLM experimentation, and work on IDE integrations—plus a platform layer meant to make it easier for internal teams to build their own AI-enhanced tools.

An Escalation Bot That Answers “How-To” Questions

Slack channels are often used as informal ticketing queues for escalations and on-call alerts. To ease that load, Slack built a bot that listens in escalation channels and automatically categorizes posts and responds to support questions. The bot can operate in two ways: it listens for a specific emoji reaction (reacji) and steps in when appropriate, or it actively monitors channel posts and jumps in when it detects a topic it can handle.

For active monitoring, LLMs classify posts into six default categories:

  • ANNOUNCEMENT – A change to a service or a new person joining.
  • HOW_TO_REQUEST – Someone asking for information or contextual help on a topic.
  • HUMAN_ACTION – A request for a human to perform an action.
  • PULL_REQUEST_REVIEW – A request for PR review or questions about a PR.
  • SERVICE_ISSUE_REQUEST – A service is down or not working as expected.
  • UNKNOWN – Posts that are unclear or too vague for the bot to assist with.

To sharpen classification accuracy, Slack provides example posts for each category inside the prompt, wrapped in <EXAMPLE> tags.

Knowledge Retrieval and Customization

For contextually aware answers, the bot pulls knowledge from several sources: technical documentation relevant to a channel, historical posts in the escalation channel via Slack’s message search API, and shared canvases via the canvas search API. Slack uses Amazon Bedrock Knowledge Bases to vectorize and store this data, taking advantage of the managed service’s handling of infrastructure, chunk sizing, vector stores, indexes, and on-demand scalable embedding models.

Each channel can be tuned individually, with controls over prompt wording, response length, documentation filters, and similarity score thresholds before the bot responds. These per-channel configurations help reduce hallucinations.

Effectiveness is measured via user interaction ratings: 30% of interactions were rated five stars, and the bot is enabled in 25+ internal channels. Slack estimates that the bot alone has saved about 3,000 hours of engineering time in a year.

Three Front Doors to an AI Assistant

Beyond the bot, Slack has brought AI into the platforms where engineers already work, trying to minimize context switching:

Custom AI Assistant in Slack

Building on Slack AI’s native search capabilities, internal teams can now deploy a custom AI agent accessible via a dedicated secure UI inside Slack. Slack’s assistant is designed to answer from internal and external knowledge bases, cutting down on the need to gather information manually. It is still in beta, and currently represents about 14% of all internal LLM traffic.

Web App for Advanced Tuning

For engineers who need more control, a custom web application provides a conversational interface to the same underlying agents and LLMs. This app is aimed at advanced users, giving fine-grained control over API parameters such as temperature and token limits. Agents are connected to internal technical documentation and can synthesize answers about Slack-specific systems. The web app accounts for about 37% of internal LLM traffic, making it one of the most-used tools in the portfolio.

IDE Integration

Surveys inside Slack show 65% of developer respondents prefer an IDE as the primary place to interact with generative AI. In response, Slack is evaluating in-house IDE plugins and customized open source plugins that route AI traffic to its approved LLM infrastructure. Target use cases for the pilot include frontend development with TypeScript, backend development with Hack, and Android and iOS development.

Across the Slack and web apps, which have both been generally available internally for five months, Slack logs over 4,000 monthly chat-turns, with a 70%+ positive satisfaction rate.

A Platform for Building AI Features

The tools are the visible portion of a bigger infrastructure play. Slack has developed a unified LLM platform with:

  • Access to multiple approved LLMs, so teams can pick the best model for a task.
  • A defined schema for input/output, giving applications an abstraction layer so they can swap LLMs without rewrites.
  • A derived data system that uses an API-level source attribute to track LLM traffic uniformly.
  • Programmatic APIs for knowledge base creation, management, and querying.

To speed up teams building their own AI features, three companion utilities have been created:

  • CLI tools – For quick experiments at the terminal, sending structured requests to the AI infrastructure.
  • Evaluate – A web-based playground where developers can chat with different models and refine prompts iteratively.
  • Compare – A side-by-side viewer for generated text from different LLMs, helping developers choose the right model for their project requirements.

These supporting tools have made it easier for the 16+ internal teams Slack is partnering with to prototype applications quickly and understand the pros and cons of specific LLMs.

Reality Checks and Limits

Notably, the journey has surfaced consistent limitations of current-generation LLMs: hallucinations and weak multi-step problem solving hamper adoption in complex technical situations and occasionally cost engineering time. Fine-tuning retrieval of third-party knowledge sources also takes considerable systems work, and the rapid evolution of generative AI requires constant learning across the user base.

How the Time Savings Estimate Is Built

The headline metric—10,000+ hours saved by year end—comes from accounting for a focused set of factors:

  • Time spent searching for information from internal or external sources (minutes to a couple of hours per lookup).
  • The estimated 30 to 60 minutes it takes to read a medium-length versus complex internal document.
  • Savings from reduced context switching and less manual translation of information between platforms.
  • Time spent comprehending gathered knowledge, which varies by seniority.
  • Time to turn acquired knowledge into actual code or other action.
  • A deduction for time lost to hallucinations, estimated at about 30% of total interactions.

Slack’s internal rollout underscores a broader point: AI adoption in an engineering org tends to work when tooling is both secure and highly customizable per team. The scoring and tuning infrastructure Slack built alongside the models—not just the models themselves—is what makes each channel context reliable and useful.