Streaming LLM Responses Without the Jank

Two UX problems tend to plague large language model (LLM) chatbots: markdown rendering jank and response delay. Rendering jank happens when syntax fragments appear as raw text until a complete markdown element forms. Response delay is the consequence of multiple LLM roundtrips while the model consults external data sources—the user stares at a spinner while the backend assembles an answer.

Shopify's Sidekick team tackled both problems together with a buffering markdown parser and an event emitter. The approach multiplexes multiple streams and events into one stream that renders piece-by-piece, preventing markdown jank while streaming the LLM response immediately and merging asynchronously resolved content into the stream as it becomes available.

Gif demonstrating solution from both problems.

The Ambiguity Problem in Streamed Markdown

Streaming makes markdown rendering difficult because character sequences stay ambiguous until an end-of-expression marker appears. Consider two common cases:

  • Emphasis versus unordered list items: A "*" at the start of a line could open either a <strong> or a <li> element. The parser cannot disambiguate until it sees a closing "*" (emphasis) or an immediately following whitespace character (list item).
  • Links: A "[link text](link URL)" pattern cannot render as an <a> element until the closing parenthesis arrives, because the full URL isn't yet known.

Sidekick solves this by buffering characters whenever the parser encounters a markdown candidate sequence. The buffer flushes under two conditions: when an unexpected character appears, the whole sequence renders as raw text (the markdown syntax was a false positive); or when the markdown element is complete, the buffer renders as a single element sequence. This requires a stateful stream processor consuming characters one at a time, passing them through or buffering them depending on the FSM state.

The team uses a Node.js Transform stream running a finite state machine (FSM), fed with individual Unicode characters from stream chunks. Iterating with for..of over the chunk string handles character boundaries properly. Stream chunks from an LLM can be assumed to split at Unicode character boundaries.

Support for additional markdown elements means extending the state machine. Implementing the entire CommonMark specification manually would be impractical; an off-the-shelf parser generator supporting push lexing/parsing would serve better for a full implementation.

Resolving Tool Content asynchronously

LLMs handle general language and culture well, but they're not reliable sources of current, accurate information. Sidekick directs the LLM to request external data via tools. The typical tool integration pattern is: receive user input, ask the LLM which tools to consult, execute the tool calls, then ask the LLM to assemble tool results into a final answer. Users wait through all steps before seeing anything.

Typical tool integration mapping using Shop Pay as an example.

The Sidekick tweak breaks tool invocation and output generation out of the main LLM response. The first LLM roundtrip responds directly to the user, with placeholders that populate asynchronously:

Revised mapping where we have broken the tool invocation and output generation

This means the response is no longer a string the UI renders directly; presentation requires orchestration. Rather than having the UI make additional backend requests to populate tool content, Sidekick multiplexes asynchronously-resolved tool content into the main response stream:

Mapping of a multiplex asychronously-resolved tool content into the main response stream

The UI splits the multiplexed response into components: it renders the main LLM response as it streams from the server, and renders resolved tool content into the placeholder area as it arrives. This works well for requests that contain multiple intents, and Server-Sent Events provide the multiplexing mechanism—each stream is treated as a series of named events.

Gif of a user requests with multiple intents (take me to tax settings and teach me how to use them)

Cards: Piggybacking on Markdown Parsing

Sidekick turns asynchronous multiplexing into an extension of the markdown buffering layer. The prompt tells the LLM to use special markdown links—call them "cards"—whenever it wants to insert asynchronously resolved content. The card links use a card: protocol in their URLs, and the link text is a terse, LLM-paraphrased version of the original user intent.

Because these card links are ordinary markdown, they're buffered and parsed by the same markdown parser. The parser invokes a callback when it encounters any link; Sidekick checks for the card: protocol and fires an asynchronous card resolution task. The main LLM response and card content both flow through the single multiplexed stream. This design means no separate stream parser sits on top of the LLM output looking for tool invocation syntax—the existing markdown parser is already there.

Certain cards resolve entirely at the backend and deliver final content to the UI. Others resolve into an intermediate presentation, which the UI processes and renders locally (for instance, by making a follow-up request to a service). In all cases, everything streams as it's produced, so the user always sees content being generated.

Why Markdown Holds Up as a Transport Format

Markdown beats JSON and YAML in token counts when transporting structure, and it remains human-readable. The Sidekick team uses markdown as the narrow waist connecting backend to frontend, both for transport and rendering, and for LLM-to-backend invocations.

Buffering and joining stream chunks also enable markdown mutation before it reaches the frontend—Sidekick replaces markdown links with a content identifier matching the card content multiplexed into the response stream. The FSM-based approach is relatively easy to implement and unlocks significant UX improvement.