Context Is the Hidden Cost of Long Prompts
System prompts easily run to thousands of tokens per request. Every additional token slows inference and drives up cost, which means serving the same traffic on dedicated hardware requires more GPUs. Shopify's implementation of gisting, based on the Prompt Compression and Contrastive Conditioning paper (2022), sidesteps that tradeoff: the behavior of a long prompt at the cost of a short one.
The technique was applied to the Sidekick GraphQL agent, compressing its system prompt from roughly 6,000 tokens to about 1,500 gist tokens—a 4:1 reduction—with no loss in prediction quality. The mechanism works by learning embeddings for a set of special tokens via knowledge distillation, then swapping the full prompt for those tokens at inference time.
What a Gist Token Is
A gist token is added to the model's vocabulary, and the sequence of embeddings is trained so the model behaves as if it had seen the full prompt. At 4:1 compression, one gist token replaces every four prompt tokens. The model's weights remain frozen during training; only the gist embeddings are learned.
The embeddings are trained through knowledge distillation. Each trajectory runs a forward pass twice. In the teacher pass, the model sees the complete natural-language prompt, producing teacher logits for every position in the response. In the student pass, the prompt is swapped out for gist tokens and the same model generates student logits for the same response positions. KL divergence between the two logit sets drives the training until the student's predictions align with the teacher's.
Deployment Is Straightforward
Once training finishes, the gist embeddings are written directly into the model's embedding matrix and the new tokens are registered as special tokens in the tokenizer. There is no custom attention mask, no extra encoder, and no distinct serving path. The model loads and runs like any other. The only inference-time change is on the request side: swap the prompt for the string of gist tokens. The compression cost is paid once, during training.
Why Prefix Caching Falls Short
Prefix caching and gisting are complementary. Modern serving engines keep a KV cache of keys and values for previously seen sequences; when a new request matches a cached sequence—often the system prompt—those tensors are fetched instead of recomputed.
Prefix caching does not, however, reduce decode cost. Each generated token attends over every key in the sequence, cached or not. Decode is memory-bandwidth-bound, so every generated token streams the entire KV cache from high bandwidth memory, and that read cost grows linearly with cached sequence length. Gisting cuts both the cost of attention computation and the size of KV cache reads—the latter especially matters as batch sizes grow. The two optimizations compound, and both are used in production.
Autoresearch Finds the Recipe
Hyperparameter tuning was handled by an autoresearch loop pointed at the trainer. The loop proposes a recipe, trains the gist embeddings, evaluates the result, and repeats.
Three optimizations had outsized impact:
- Initialization: Random noise was replaced by splitting the system prompt into chunks of length k (derived from the k:1 compression ratio) and initializing the nth gist embedding with the mean of the nth chunk. Initial loss dropped by a factor of seven.
- Compression ratio: Testing a range of ratios showed that 4:1 was the point beyond which prediction quality degraded for this domain.
- Data quantity and diversity: A large, diverse dataset closed the remaining quality gap.
Autoresearch also drove infrastructure refinements. Loss normalization mattered: averaging per response token caused hallucination, while batch-level averaging preserved signal from long responses and produced stable embeddings. Precomputing teacher logits and pre-tokenizing data reduced a full run from thirty hours to six.
The Measured Payoff
Load tests compared the same model with compressed versus full prompts. At 350 RPM, median time to first token (TTFT) fell from 438ms to 354ms—a 19% drop. End-to-end latency went from 6.8s to 4.2s, about 38% lower. Throughput rose from 20.2 to 23.4 queries per second (QPS), a 16% gain. The gap widened as concurrency increased and batch sizes grew. On the production GraphQL workload, that throughput improvement translated directly to 14% fewer GPUs.
Gisting Fits the Continual Learning Loop
Gisting also works naturally with Shopify's existing continual learning pipeline. Once gist embeddings are distilled for a model, that model becomes a new starting point. Post-training can use the gist embeddings as the prefix and apply gradient updates to both model weights and gist embeddings. Optimizing both on incremental data keeps the model calibrated without re-running distillation from scratch each time.
The Standard Going Forward
Long system prompts deliver real value, but they are expensive. Gisting preserves the advantages of extensive instructions with a fraction of the tokens, cutting latency and GPU spend. With GPU demand outstripping supply, every inference optimization counts—but quality cannot be sacrificed. Gisting delivers both, and it has become the standard for this workload at Shopify. The full method is detailed in Wingate, Shoeybi, and Sorensen's 2022 paper on prompt compression and contrastive conditioning.
