Streaming and longer context lengths for LLMs on Workers AI

Workers AI adds streaming, bigger context windows, and a higher-precision Llama-2

Cloudflare has expanded its serverless GPU inference platform, Workers AI, with several features developers have been asking for since launch. The platform now supports streaming responses for all large language models (LLMs), increased context and sequence lengths, and a new full-precision variant of Llama-2.

Response streaming addresses a fundamental characteristic of LLM inference: models generate output sequentially, one token at a time. While generating a single token takes only milliseconds, producing a complete response of hundreds or thousands of tokens takes seconds. With streaming, the client can begin displaying text as soon as the first tokens are generated, rather than waiting for the entire sequence to complete. This makes applications feel far more responsive and gives users time to read output as it appears.

Consuming event streams with server-sent events

Streaming on Workers AI uses server-sent events, a standardized browser API that is straightforward to implement server-side and broadly supported across platforms either natively or via polyfill. It is well suited for handling a stream of server updates without the boilerplate code typically required to manage an event stream.

To enable streaming for Workers AI text generation models, set the stream parameter to true in the request input. This changes the response format and MIME type to text/event-stream. Streaming is available for any LLM in the Workers AI catalog, including Llama-2, and will apply to future LLM additions as well.

Here is an example using the REST API:

curl -X POST \
"https://api.cloudflare.com/client/v4/accounts/<account>/ai/run/@cf/meta/llama-2-7b-chat-int8" \
-H "Authorization: Bearer <token>" \
-H "Content-Type:application/json" \
-d '{ "prompt": "where is new york?", "stream": true }'

data: {"response":"New"}

data: {"response":" York"}

data: {"response":" is"}

data: {"response":" located"}

data: {"response":" in"}

data: {"response":" the"}

...

data: [DONE]

And here is a Worker script example:

import { Ai } from "@cloudflare/ai";
export default {
    async fetch(request, env, ctx) {
        const ai = new Ai(env.AI, { sessionOptions: { ctx: ctx } });
        const stream = await ai.run(
            "@cf/meta/llama-2-7b-chat-int8",
            { prompt: "where is new york?", stream: true  }
        );
        return new Response(stream,
            { headers: { "content-type": "text/event-stream" } }
        );
    }
}

To consume the event stream from this Worker in a browser, the client-side JavaScript looks like this:

const source = new EventSource("/worker-endpoint");
source.onmessage = (event) => {
    if(event.data=="[DONE]") {
        // SSE spec says the connection is restarted
        // if we don't explicitly close it
        source.close();
        return;
    }
    const data = JSON.parse(event.data);
    el.innerHTML += data.response;
}

This code works with any simple HTML page or more complex single-page applications built with React or other frameworks. Instead of displaying a spinner while the full response is generated, the page updates incrementally as tokens arrive.

Server-sent events occupy a distinct niche compared to other communication methods:

Higher precision and longer model windows

In addition to streaming, Cloudflare has responded to community requests for longer questions and answers with Llama-2. In LLM terminology, context length refers to the number of tokens the model accepts as input before generating a prediction, while sequence length is the number of tokens it produces in its response.

To address these needs, Workers AI now includes a 16-bit full-precision Llama-2 variant in its catalog. The existing 8-bit version of Llama-2 also gets increased context and sequence lengths.

The combination of these changes is shown in the updated model listing:

Streaming, higher precision, and expanded context and sequence windows enable richer, more responsive applications built on Workers AI. Developers can find additional details and configuration options in the Workers AI documentation. Feedback and questions can be directed to the Cloudflare Community or the Cloudflare Discord.