Recommend API

One API for Every Recommendation Use Case

Slack surfaces recommendations in many places: suggesting contacts for a DM, pointing workspace creators to likely invitees, flagging channels to leave or archive, and picking the best time to send an invite reminder. Each feature tempts a bespoke machine-learning solution, but building one from scratch per use case doesn’t scale.

The ML services team answered with the Recommend API, an internal framework that standardizes the plumbing around recommendation features so product engineers can bootstrap a recommender without owning the full ML stack. The API reuses shared infrastructure across data processing, candidate generation, model training, and monitoring, and has shipped several distinct recommendation models into the product.

Why the ML Engineering Is the Hard Part

Most of the team’s effort goes into MLOps, not model training. The broader industry reflects this imbalance: the 2021 ML/data landscape survey catalogs a glut of tools per MLOps phase, a sign that standards are still forming, and most corporate AI initiatives (up to 88% by some reports) never get past experimentation. Like Facebook, Netflix, and Uber, Slack builds its own in-house machinery rather than stitching together a vendor stack.

Slack gets a head start from its data warehouse ecosystem, which provides:

• Sequential job scheduling in Airflow
• Data ingestion from databases plus logs from servers, clients, and queues
• Query and dashboard tooling for data tracking and visualization
• Keyed lookups into a Feature Store
• An A/B testing framework for evaluating launches

Infrastructure from other internal teams — Cloud Services, Cloud Foundations, and Monitoring — completes the foundation the Recommend API builds on.

Recommendation Features in the Product

The Recommend API currently powers several live product experiences:

• Composer DMs: suggesting additional people to add to a conversation
• Creator invite flow: recommending top contacts to invite when creating a workspace, based on the creator’s Google Calendar events
• Slackbot channel suggestions: flagging channels to leave or archive from user activity patterns
• Channel browser sort order: the default ordering (⌘ + ⇧ + L) when browsing channels
• Invite reminder timing: personalizing send time per team and inviter activity instead of a fixed default

The bigger win may be the experiments this framework enabled. Roughly as many recommendation use cases are in internal testing or were tried and abandoned as are live in the product. The value of that breadth goes beyond any single feature.

Prototyping the Path

Building ML cold is expensive and uncertain, which historically discouraged speculative use cases. By lowering the upfront cost of a recommendation prototype, the Recommend API aligns with Slack’s core product principle of “prototyping the path.” The result is more ML prototypes across the product, a net increase in both shipped features and learnings from those that didn’t pan out.

One Workflow, Many Recommenders

Slack organizes recommenders along two axes: “corpus” and “source.” A corpus is the type of entity being recommended, such as a channel or user, while a source is a specific part of the product. Different sources can share the same corpus; for instance, Slackbot channel suggestions and the Channel browser recommendation each have their own source but both recommend from the channel corpus.

Regardless of corpus or use case, every recommendation request follows the same basic flow:

• The backend handles the request, takes in a query, corpus and source, and returns a logged list of recommendations.
• The frontend logs user interactions with those results.
• Offline in the data warehouse (Airflow), those logs are combined into training data to train new models, which are then served back to the backend for future recommendations.

That full workflow breaks down into a few distinct phases.

Backend: Standardized Recommendation Steps

Each source has a corresponding “recommender,” which implements an ordered set of steps to produce a list of recommendations:

• Fetch relevant candidates from sources including the embeddings service, where similar entities are close in vector space
• Filter candidates on relevancy, ownership or visibility, e.g. private channels
• Augment features such a sentity attributes and activity from the Feature Store
• Score and sort candidates using ML model predictions
• Rerank candidates based on additional rules

These steps are implemented as standardized classes that are reusable across recommenders. Building a new recommender meant composing a sequence of those classes; often, adding a new recommender simply requires wiring existing steps together:

final class RecommenderChannel extends Recommender {
   public function __construct() {
	   parent::__construct(
		   /* fetchers */ vec[new RecommendChannelFetcher()],
		   /* filters  */ vec[new RecommendChannelFilterPrivate()],
		   /* model    */ new RecommendLinearModel(
			   RecommendHandTunedModels::CHANNEL,
			   /** extra features to extract **/
			   RecommendFeatureExtractor::ALL_CHANNEL_FEATURES,
		   ),
		   /* reranker */ vec[new RecommendChannelReranker()],
	   );
   }
}

Logging and Data Pipelines

The base recommender handles detailed logging beyond serving results—tracking the originating API request, the returned results, and the features the model consumed at scoring time. These logs along with frontend interaction logs like clicks are joined by scheduled Airflow jobs to generate training data for the models.

Model Training

Recurring Airflow tasks train models via Kubernetes Jobs, then serve them on Kubernetes Clusters, completing the cycle of logging, training and serving. Each source tends to be tested with several model types, from Logistic Regression to XGBoost. For example, the people-browser source alone is running experiments with the six models shown here, alongside the piece of Python needed for the XGBoost ranking model.

ModelArtifact(
	name="people_browser_v0_xgbr",
	model=RecommendationRankingModel(
		pipeline=create_recommendation_pipeline(
			XGBRanker(
				**{
					"objective": "rank:map",
					"n_estimators": 500,
				}
			)
		),
		input_config=RecommenderInputConfig(
			source="people-browser",
			corpus=Corpus.USER,
			feature_specification=UserFeatures.get_base_features(),
		),
	),
)

Privacy by Construction

Protecting Customer Data—the content of user messages and files—is a core constraint in model design. Slack's privacy principles mean avoiding any technique that could leak content outside its workspace. The approach combines data refinements and feature engineering:

• All training data is de-identified: teams, users and channels appear as numeric IDs rather than names or text.
• Features rely overwhelmingly on Slack usage metadata, such as interaction counts for the entity being scored, rather than the content itself. Even the most common embedding is derived from factoring a matrix of interaction counts per user per channel—built with no Customer Data.
• Semantic similarity between a user and an entity is incorporated without encoding the content itself. An open source embedding model (not trained on Slack data) embeds messages from both sides, which are aggregated to single embeddings, and the cosine similarity between them becomes a feature. The content-based embeddings themselves are deliberately not exposed to the model.

The resulting training data is free of customer content and personally identifiable information, allowing models to recommend well without ever seeing the content they could otherwise memorize and leak.

Instrumentation and Experimentation

Metric output is built into each stage of the system so performance can be monitored, automatically updating when new models are deployed:

• Reliability metrics: Prometheus counters from the backend tracking request and error counts
• Efficiency metrics: Prometheus metrics from the serving service, such as throughput and latency
• Online metrics: Business metrics shared externally, chiefly clickthrough rate (CTR) and ranking metrics like discounted cumulative gain (DCG)
• Offline metrics: Ranking and classification metrics computed on validation data separate from training data, for comparing candidate models before production
• Feature stats: Distribution and importance metrics that feed anomaly detection on distribution shift

Building models for new use cases presents a classic cold start loop: models need interaction data, but interaction needs a working model. The first iteration therefore leans on a hand-tuned model—simple heuristics like sending invite reminders when a team is active—while logging useful features from the Feature Store. That produces the first training batch to power iterative improvements.

Progression to model-backed recommendations happens through extensive A/B testing. Any switch from hand-tuned rules to ML, or any feature set or model change, is validated against key business metrics before rollout.

Those experiments produce meaningful wins in CTR:

• Composer DMs: +38.86% moving from hand-tuned to logistic regression; a further +5.70% with an XGBoost classifier plus expanded features
• Creator invite flow: +15.31% shifting from hand-tuned to logistic regression
• Slackbot channel suggestions: +123.57% for leave and +31.92% for archive suggestions moving from hand-tuned rules to XGBoost
• Channel browser recommendation: +14.76% moving from hand-tuned to XGBoost classification, visible in the experiment impact over time

Foundations and Future Work

The Recommend API has been serving ML models for several years, but the underlying service infrastructure took far longer to build. Its unified design now lets teams rapidly prototype and ship ML models across the product. Current focus areas are expanding data logging for broader use cases, improving model training infrastructure along the axes of scaling, hardware acceleration and debuggability, and building up model explainability tooling via SHAP. The team is also reaching out across Slack to find more parts of the product ripe for ML improvements.