Media Source Extensions

Media Source Extensions (MSE) is a JavaScript API that lets you build streams for playback from segments of audio or video.

W

web.devweb.dev

·

Demystifying Media Source Extensions

François Beaufort

Media Source Extensions (MSE) is a JavaScript API that allows you to build streams for playback from segments of audio or video. It's the foundation underneath features like adaptive streaming, ad insertion, time shifting, and fine-tuning download size and performance. To build a mental model, think of MSE as a chain of layers between a downloaded media file and the media element itself:

An <audio> or <video> element to play the media.
A MediaSource instance with a SourceBuffer to feed the media element.
A fetch() or XHR call to retrieve media data in a Response object.
A call to Response.arrayBuffer() to feed MediaSource.SourceBuffer.

In code, that chain looks like this:

var vidElement = document.querySelector('video');

if (window.MediaSource) {
  var mediaSource = new MediaSource();
  vidElement.src = URL.createObjectURL(mediaSource);
  mediaSource.addEventListener('sourceopen', sourceOpen);
} else {
  console.log('The Media Source Extensions API is not supported.');
}

function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  var mime = 'video/webm; codecs="opus, vp09.00.10.08"';
  var mediaSource = e.target;
  var sourceBuffer = mediaSource.addSourceBuffer(mime);
  var videoUrl = 'droid.webm';
  fetch(videoUrl)
    .then(function (response) {
      return response.arrayBuffer();
    })
    .then(function (arrayBuffer) {
      sourceBuffer.addEventListener('updateend', function (e) {
        if (!sourceBuffer.updating && mediaSource.readyState === 'open') {
          mediaSource.endOfStream();
        }
      });
      sourceBuffer.appendBuffer(arrayBuffer);
    });
}

This article walks through each link by building a simple MSE example. If the overview suffices, stop here; for a step-by-step walkthrough, continue reading.

Setting Up the MediaSource

Standard web development practice dictates you start with feature detection. Next, create an instance of MediaSource. A MediaSource object can be passed to a src attribute—which may seem odd since these are usually string URLs, but they can also be blobs. You convert the instance to a URL and assign it to the media element:

var vidElement = document.querySelector('video');

if (window.MediaSource) {
  var mediaSource = new MediaSource();
  vidElement.src = URL.createObjectURL(mediaSource);
  // Is the MediaSource instance ready?
} else {
  console.log('The Media Source Extensions API is not supported.');
}

**Figure 1**: A source attribute as a blob

While URL.createObjectURL() is synchronous, the attachment is processed asynchronously. That means you can't use the MediaSource immediately. The readyState property describes the relation between a MediaSource instance and a media element. Its possible values are:

closed - The MediaSource instance is not attached to a media element.
open - The MediaSource instance is attached to a media element and is ready to receive data or is receiving data.
ended - The MediaSource instance is attached to a media element and all of its data has been passed to that element.

Querying readyState directly can hurt performance. Instead, listen to events like sourceopen to know when to fetch and buffer video. Note the call to revokeObjectURL(). It's safe to invoke immediately once the element's src is connected; it doesn't destroy objects, but rather allows the platform to handle garbage collection at an optimum time.

var vidElement = document.querySelector('video');

if (window.MediaSource) {
  var mediaSource = new MediaSource();
  vidElement.src = URL.createObjectURL(mediaSource);
  <strong>mediaSource.addEventListener('sourceopen', sourceOpen);</strong>
} else {
  console.log("The Media Source Extensions API is not supported.")
}

<strong>function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  // Create a SourceBuffer and get the media file.
}</strong>

Building the Buffer

Next, create the SourceBuffer, the object that shuttles data between media sources and media elements. It must be specific to the media file type. Call addSourceBuffer() with the correct value. The mime type string typically includes the main type and separate codecs for the video and audio portions of the file.

Note that the MSE spec allows user agents to vary on whether they require both a mime type and a codec. Some accept just the mime type, while Chrome, for instance, requires a codec for types that don't self-describe. To keep things portable, just include both.

var vidElement = document.querySelector('video');

if (window.MediaSource) {
  var mediaSource = new MediaSource();
  vidElement.src = URL.createObjectURL(mediaSource);
  mediaSource.addEventListener('sourceopen', sourceOpen);
} else {
  console.log('The Media Source Extensions API is not supported.');
}

function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  <strong>
    var mime = 'video/webm; codecs="opus, vp09.00.10.08"'; // e.target refers to
    the mediaSource instance. // Store it in a variable so it can be used in a
    closure. var mediaSource = e.target; var sourceBuffer =
    mediaSource.addSourceBuffer(mime); // Fetch and process the video.
  </strong>;
}

Fetching the File and Appending Data

The SourceBuffer is ready, but the media still needs to get the data. Use the Fetch API to retrieve the file. In Safari, a fetch() polyfill is necessary for this to work. A production player would maintain multiple file versions (for codec/browser support) and multiple resolutions (for adaptive bitrate streaming)^*, but a basic example loads a single file.

function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  var mime = 'video/webm; codecs="opus, vp09.00.10.08"';
  var mediaSource = e.target;
  var sourceBuffer = mediaSource.addSourceBuffer(mime);
  var videoUrl = 'droid.webm';
  <strong>
    fetch(videoUrl) .then(function(response){' '}
    {
      // Process the response object.
    }
    );
  </strong>;
}

To pass the data from the Response object to the MediaSource, call response.arrayBuffer(). This returns a promise for a buffer, which we then append via appendBuffer():

function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  var mime = 'video/webm; codecs="opus, vp09.00.10.08"';
  var mediaSource = e.target;
  var sourceBuffer = mediaSource.addSourceBuffer(mime);
  var videoUrl = 'droid.webm';
  fetch(videoUrl)
    .then(function(response) {
      <strong>return response.arrayBuffer();</strong>
    })
    <strong>.then(function(arrayBuffer) {
      sourceBuffer.appendBuffer(arrayBuffer);
    });</strong>
}

*While technologies like DASH and HLS handle the realities of network adaptation and chunked media, going deep into them is a separate topic.

Finalizing the Stream

Once all ArrayBuffers are appended and no more media data is expected, call MediaSource.endOfStream(). This changes the readyState to ended and fires the sourceended event:

function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  var mime = 'video/webm; codecs="opus, vp09.00.10.08"';
  var mediaSource = e.target;
  var sourceBuffer = mediaSource.addSourceBuffer(mime);
  var videoUrl = 'droid.webm';
  fetch(videoUrl)
    .then(function(response) {
      return response.arrayBuffer();
    })
    .then(function(arrayBuffer) {
      <strong>sourceBuffer.addEventListener('updateend', function(e) {
        if (!sourceBuffer.updating && mediaSource.readyState === 'open') {
          mediaSource.endOfStream();
        }
      });</strong>
      sourceBuffer.appendBuffer(arrayBuffer);
    });
}

Here is the final, complete example:

var vidElement = document.querySelector('video');

if (window.MediaSource) {
  var mediaSource = new MediaSource();
  vidElement.src = URL.createObjectURL(mediaSource);
  mediaSource.addEventListener('sourceopen', sourceOpen);
} else {
  console.log('The Media Source Extensions API is not supported.');
}

function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  var mime = 'video/webm; codecs="opus, vp09.00.10.08"';
  var mediaSource = e.target;
  var sourceBuffer = mediaSource.addSourceBuffer(mime);
  var videoUrl = 'droid.webm';
  fetch(videoUrl)
    .then(function (response) {
      return response.arrayBuffer();
    })
    .then(function (arrayBuffer) {
      sourceBuffer.addEventListener('updateend', function (e) {
        if (!sourceBuffer.updating && mediaSource.readyState === 'open') {
          mediaSource.endOfStream();
        }
      });
      sourceBuffer.appendBuffer(arrayBuffer);
    });
}

Simplifications & Production Notes

For clarity, this introduction leaves out playback controls (provided by the HTML5 elements themselves) and error handling, and it recommends using a library like Google's Shaka Player for anything serious. Before deploying MSE, consider these production guidelines:

Handle error events and exceptions, and check HTMLMediaElement.readyState and MediaSource.readyState before making API calls.
Verify previous appendBuffer() and remove() calls have finished by checking SourceBuffer.updating before modifying the buffer's mode, timestampOffset, or append window.
Ensure no SourceBuffer instances are updating before calling MediaSource.endOfStream() or setting MediaSource.duration.
Be aware that if readyState is ended, calls like appendBuffer() will change it to open, so you should expect multiple sourceopen events.
When handling HTMLMediaElement errors, the MediaError.message content can help identify the root cause of hard-to-reproduce failures.