Jake Archibald's 2016 declaration of web streams as the year's big story proved early. The Streams Standard was announced in 2014, and it has taken until now for a consistent streaming API to land in modern browsers (Firefox still pending) as well as in Node and Deno.

How streaming works

Instead of waiting for a full download, a stream splits a resource into chunks that are processed one at a time, starting with the first chunk that arrives. Three stream types make this up: readable streams, the source of chunks (from a file or HTTP connection, for instance); transform streams, which can optionally modify the data in between; and writable streams, which receive the piped chunks.

One API for browser, Node and Deno

Node has long shipped its own stream implementation, one that is widely regarded as difficult to use. The WHATWG standard, which the Node documentation refers to as "web streams," arrived later and is generally seen as an improvement. The original Node streams are not being deprecated or removed — the two APIs will coexist, which makes cross-platform code simpler and leaves developers with a single model to learn.

Deno, Node's original creator's take on server-side JavaScript, has always tracked browser APIs closely and fully supports web streams. Cloudflare Workers, which run at CDN edge locations much like service workers, and Deno Deploy, Deno's serverless offering, support them too.

Reading a fetch() response

There are several ways to construct a readable stream, but fetch() is likely the most common: the response body of fetch() is itself a readable stream.

fetch('data.txt')
.then(response => console.log(response.body));

The console output reveals a set of useful methods on a readable stream. As the spec states, a readable stream can be piped directly to a writable stream with pipeTo(), or routed through one or more transform streams first with pipeThrough().

Node core does not currently implement fetch. The popular node-fetch package, which mimics the browser API, returns a Node stream rather than a WHATWG stream. Undici, an improved HTTP/1.1 client from the Node.js team, is a modern replacement for core http.request — the layer that node-fetch and Axios are built on. Undici implements fetch, and its response.body does return a web stream. It may eventually be folded into Node core and is positioned to become the recommended way to make HTTP requests in Node.

After npm install undici and importing fetch, it behaves exactly as in the browser. The example below pipes the stream through a transform stream; each chunk is a Uint8Array, and Node core supplies a TextDecoderStream for decoding binary data.

import { fetch } from 'undici';
import { TextDecoderStream } from 'node:stream/web';

async function fetchStream() {
  const response = await fetch('https://example.com')
  const stream = response.body;
  const textStream = stream.pipeThrough(new TextDecoderStream());
}

Because response.body is synchronous, there is no await needed. In browsers, fetch and TextDecoderStream live on the global object, so the import statements disappear and the remaining code is identical across Node and the browser. Deno also provides built-in support for fetch and TextDecoderStream.

Async iteration over streams

A for-await-of loop is the asynchronous counterpart of for-of, which walks arrays and other iterables; the async form can iterate an array of promises, for example.

const promiseArray = [Promise.resolve("thing 1"), Promise.resolve("thing 2")];
for await (const thing of promiseArray) { console.log(thing); }

More relevant here, it also iterates streams.

async function fetchAndLogStream() {
  const response = await fetch('https://example.com')
  const stream = response.body;
  const textStream = stream.pipeThrough(new TextDecoderStream());

  for await (const chunk of textStream) {
    console.log(chunk);
  }
}

fetchAndLogStream();

This works in Node and Deno. For-await-of has shipped in all modern browsers, but it does not yet work on streams there.

Other sources of readable streams

Blob and File both expose a .stream() method that returns a readable stream. The code below runs in modern browsers, in Node and in Deno — though Node requires import { Blob } from 'buffer'; first:

const blobStream = new Blob(['Lorem ipsum'], { type: 'text/plain' }).stream();

On the front end, an <input type="file"> in your markup makes it easy to obtain the user's chosen file as a stream:

const fileStream = document.querySelector('input').files[0].stream();

Node 17 ships a .readableWebStream() method on the FileHandle object returned by the fs/promises open() function.

import {
  open,
} from 'node:fs/promises';

const file = await open('./some/file/to/read');

for await (const chunk of file.readableWebStream())
  console.log(chunk);

await file.close();

Promises and stream completion

To run code after a stream finishes, use promises.

someReadableStream
.pipeTo(someWritableStream)
.then(() => console.log("all data successfully written"))
.catch(error => console.error("something went wrong", error))

Optionally, the result can be awaited:

await someReadableStream.pipeTo(someWritableStream)

Building a transform stream

Beyond the built-in TextDecoderStream and TextEncoderStream, you can write your own. The TransformStream constructor accepts an object that may define three methods — start, transform and flush. All are optional, but transform is where the actual work happens.

For illustration, here is an over-simplified reimplementation of TextDecoderStream's behavior (use the real TextDecoderStream in production):

const decoder = new TextDecoder();
const decodeStream = new TransformStream({
  transform(chunk, controller) {
    controller.enqueue(decoder.decode(chunk, {stream: true}));
  }
});

The controller forwards each modified chunk: encoded text arrives, gets decoded, and is passed on. The other two methods look like this:

const transformStream = new TransformStream({
  start(controller) {
    // Called immediately when the TransformStream is created
  },

  flush(controller) {
    // Called when chunks are no longer being forwarded to the transformer
  }
});

A transform stream is a readable and a writable stream paired to transform data. Every object from new TransformStream() has a readable property holding a ReadableStream and a writable property holding a writable stream. Calling someReadableStream.pipeThrough() writes data from the readable stream into transformStream.writable, optionally transforms it, then pushes it to transformStream.readable.

An "identity transform stream" — made by calling new TransformStream() with no argument, or omitting the transform method — performs no transformation at all, passing every chunk straight from writable to readable. The following code logs "hello":

const {readable, writable} = new TransformStream();
writable.getWriter().write('hello');
readable.getReader().read().then(({value, done}) => console.log(value))

Building a readable stream

A custom stream can be filled with your own chunks. new ReadableStream() takes an object that may include a start function, a pull function and a cancel function. start runs immediately when the stream is constructed; inside it, controller.enqueue adds chunks.

A basic "hello world":

import { ReadableStream } from "node:stream/web";
const readable = new ReadableStream({
  start(controller) {
    controller.enqueue("hello");
    controller.enqueue("world");
    controller.close();
  },
});

const allChunks = [];
for await (const chunk of readable) {
  allChunks.push(chunk);
}
console.log(allChunks.join(" "));

A more real-world case, drawn from the streams specification, turns a web socket into a readable stream:

function makeReadableWebSocketStream(url, protocols) {
  let websocket = new WebSocket(url, protocols);
  websocket.binaryType = "arraybuffer";

  return new ReadableStream({
    start(controller) {
      websocket.onmessage = event => controller.enqueue(event.data);
      websocket.onclose = () => controller.close();
      websocket.onerror = () => controller.error(new Error("The WebSocket errored"));
    }
  });
}

Bridging Node streams and web streams

Since Node's original stream API stays alongside the web standard, you may sometimes need to convert between them. The .fromWeb() and .toWeb() methods, added in Node 17, handle both directions.

import {Readable} from 'node:stream';
import {fetch} from 'undici';

const response = await fetch(url);
const readableNodeStream = Readable.fromWeb(response.body);

Where streams still fall short

More browser APIs are migrating into Node.js all the time — ES modules, EventTarget, AbortController, the URL parser, Web Crypto, Blob, TextEncoder/Decoder — and fetch and streams are central to that convergence, with skills carrying over between environments.

Domenic Denicola, a co-author of the streams spec, has written that the API aims to be an efficient abstraction and a unifying primitive for I/O, much as promises became for asynchronicity. For that to pay off on the front end, more APIs need to support streams. A MediaStream, its name notwithstanding, is not a readable stream, and when working with video or audio a readable stream cannot currently be assigned to srcObject. Feeding an image through a transform stream and onto the page is verbose at the time of writing:

const response = await fetch('cute-cat.png');
const bodyStream = response.body;
const newResponse = new Response(bodyStream);
const blob = await newResponse.blob();
const url = URL.createObjectURL(blob);
document.querySelector('img').src = url;    

Adoption is nonetheless moving forward in browsers, Node and Deno. Deno and Chrome already offer a stream API for Web Sockets; Chrome has implemented Fetch request streams; and Node and Chrome have implemented transferable streams, which pipe data to and from a worker so chunks can be processed on a separate thread. Real products are using streams today — the makers of the file-sharing app Wormhole have open-sourced code that encrypts a stream.