Frame-accurate video work in the browser

Video in the browser has long been a black box: developers could draw the current frame, but had no reliable way to know when a new frame was actually presented. The HTMLVideoElement.requestVideoFrameCallback() method changes that by letting you register a callback that fires in the rendering steps when a new video frame is sent to the compositor. From there, you can do per-frame work such as canvas painting, video analysis, or synchronizing with external audio sources.

The API is available in Chrome and Edge from version 83, Safari from 15.4, and Firefox from version 132.

How it differs from requestAnimationFrame

The key difference from window.requestAnimationFrame() is what drives the callback. requestAnimationFrame() fires roughly 60 times per second, tied to the display refresh rate, regardless of what the video is doing. requestVideoFrameCallback(), in contrast, is bound to the actual video frame rate. Operations scheduled through it, such as drawImage() onto a canvas, are synchronized as a best effort with the frame rate of the video playing on screen.

There is one important exception to the frame-rate binding. As the specification notes, callbacks are fired at the lesser rate of the video's rate and the browser's rate. A 25fps video in a browser painting at 60Hz will fire callbacks at 25Hz; a 120fps video in that same 60Hz browser will only fire at 60Hz.

Feature detection

if ('requestVideoFrameCallback' in HTMLVideoElement.prototype) {
  // The API is supported!
}

A polyfill exists, built on Window.requestAnimationFrame() and HTMLVideoElement.getVideoPlaybackQuality(). Check the README for its limitations before relying on it in production.

Registering callbacks

The usage pattern mirrors requestAnimationFrame(): register an initial callback once, then re-register it inside the callback each time it fires.

const doSomethingWithTheFrame = (now, metadata) => {
  // Do something with the frame.
  console.log(now, metadata);
  // Re-register the callback to be notified about the next frame.
  video.requestVideoFrameCallback(doSomethingWithTheFrame);
};
// Initially register the callback to be notified about the first frame.
video.requestVideoFrameCallback(doSomethingWithTheFrame);

The callback receives two arguments: now, a DOMHighResTimeStamp, and metadata, a VideoFrameMetadata dictionary containing these properties:

  • presentationTime: When the user agent submitted the frame for composition.
  • expectedDisplayTime: When the user agent expects the frame to be visible.
  • width / height: Dimensions of the video frame in media pixels.
  • mediaTime: The presentation timestamp (PTS) in seconds, on the same timeline as video.currentTime.
  • presentedFrames: A count of frames submitted for composition, useful for detecting missed frames.
  • processingDuration: Time from submitting the encoded packet with this PTS to the decoder until the decoded frame was ready.

WebRTC sources can additionally expose:

  • captureTime: When the camera captured the frame (estimated for remote sources via clock synchronization and RTCP sender reports).
  • receiveTime: When the last packet of the encoded frame arrived over the network.
  • rtpTimestamp: The RTP timestamp for the frame.

Identifying frames with mediaTime

The mediaTime property is worth special attention. Chromium's implementation drives video.currentTime from the audio clock, but mediaTime is populated directly from the frame's presentationTimestamp. For reproducible frame identification—including determining exactly which frames you missed—use mediaTime, not currentTime.

Watch out for vsync latency

The API runs on the main thread while actual video compositing happens on the compositor thread, so everything is best effort with no strict guarantees. It is possible for the callback to fire one vertical sync (vsync) late relative to frame rendering.

Changes you make in the callback appear on screen one vsync after the frame itself is rendered. If you update a displayed frame counter against numbered video frames, the video can appear one frame ahead. What's happening: the frame is ready at vsync x, the callback fires and the frame renders at vsync x+1, and your callback's DOM changes render at vsync x+2.

You can detect this by comparing metadata.expectedDisplayTime against now. If the two are within about five to ten microseconds, the frame is already rendered. If expectedDisplayTime is roughly sixteen milliseconds ahead on a 60Hz display, your callback is in sync with the frame.

A practical example

In practice, frame-accurate drawing onto a canvas requires exactly this pattern: request the callback, draw the video frame in the callback, then request the next frame. The logged metadata in this example shows how the frame timing properties surface at runtime.

let paintCount = 0;
let startTime = 0.0;

const updateCanvas = (now, metadata) => {
  if (startTime === 0.0) {
    startTime = now;
  }

  ctx.drawImage(video, 0, 0, canvas.width, canvas.height);

  const elapsed = (now - startTime) / 1000.0;
  const fps = (++paintCount / elapsed).toFixed(3);
  fpsInfo.innerText = `video fps: ${fps}`;
  metadataInfo.innerText = JSON.stringify(metadata, null, 2);

  video.requestVideoFrameCallback(updateCanvas);
};

video.requestVideoFrameCallback(updateCanvas);

Beyond time-based hacks

Frame-level video processing previously meant approximating based on video.currentTime polls or requestAnimationFrame(), never knowing exactly which frame was on screen. With requestVideoFrameCallback(), code can finally run in lockstep with the presented video frames rather than guessing at their timing.