Virtual reality comes to the web, part II

Inside the WebXR frame loop

The WebXR Device API runs on an infinite, user-agent controlled loop in which content is drawn repeatedly to the screen. Each pass through this frame loop produces a discrete frame; the rapid succession of frames creates the illusion of movement. This article walks through the parts of that loop and the objects that feed it, using the Immersive VR Session sample from the Immersive Web Working Group as a reference.

Rendering during a frame loop requires WebGL or WebGL2 directly, or a framework that abstracts over them, such as three.js, babylonjs, or PlayCanvas. Some frameworks, including A-Frame and React 360, were designed specifically for WebXR interaction.

The objects in play

Several distinct objects cooperate in each frame. Some are only reachable as properties of others, which makes the flow hard to trace at first.

XRViewerPose

A pose describes the position and orientation of something in 3D space. Viewer and input devices both have poses, but for drawing you need the viewer's pose. Its transform attribute describes position as a vector and orientation as a quaternion, relative to an origin. That origin is determined by the reference space type you pass when calling XRSession.requestReferenceSpace().

Reference spaces vary in meaning. The sample used here requests a 'local' reference space, which sets the origin at the viewer's position when the session is created, without a defined floor and with platform-dependent precision.

XRView

An XRView represents one camera viewing the virtual scene, corresponding to a physical display or a portion of one. Each has a transform with position and orientation, available either as a vector/quaternion pair or as an equivalent matrix. Views are stored in an array on the XRViewerPose object. The number of views varies: a mobile AR scene typically has one view, while headsets commonly have two, one per eye.

XRWebGLLayer and its helpers

Layers provide bitmap image sources and instructions for rendering them to a device. An XRWebGLLayer effectively links the headset hardware to a WebGLRenderingContext. From this layer you obtain two other key objects: a WebGLFramebuffer, which supplies image data to the rendering context, and an XRViewport, which gives the coordinates and dimensions of the rectangular region within that framebuffer.

WebGLRenderingContext

The rendering context is the programmatic entry point to the canvas being drawn on. It requires both a WebGLFramebuffer and an XRViewport to work. The XRWebGLLayer corresponds to the viewer's device; the WebGLRenderingContext corresponds to the web page. The framebuffer and viewport are passed from the former to the latter.

What happens each frame

The frame loop runs at a rate set by the hardware. VR frames per second can range from 60 to 144; AR on Android runs at 30. Code must never assume a specific rate. Each cycle repeats the same steps:

1. Call XRSession.requestAnimationFrame(), prompting the user agent to invoke your XRFrameRequestCallback.
2. Inside that callback:
   1. Call XRSession.requestAnimationFrame() again to schedule the next frame.
   2. Get the viewer's pose.
   3. Bind the WebGLFramebuffer from the XRWebGLLayer to the WebGLRenderingContext.
   4. Iterate each XRView, pass its XRViewport from the layer to the rendering context, and draw to the framebuffer.

Retrieve the viewer's pose

To place anything in AR or VR, you must know the viewer's position and gaze. The XRFrame.getViewerPose() method returns an XRViewerPose for the current animation frame when passed the reference space acquired at session setup. All values returned are relative to that same reference space. One viewer pose represents the user overall—the head, or the phone camera—but actual rendering steps through the XRView objects.

function onXRFrame(hrTime, xrFrame) {
  let xrSession = xrFrame.session;
  xrSession.requestAnimationFrame(onXRFrame);
  let xrViewerPose = xrFrame.getViewerPose(xrRefSpace);
  if (xrViewerPose) {
    // Render based on the pose.
  }
}

Always test whether a pose was returned. The system may lose tracking—for instance, when headset or phone cameras cannot see in low light—or may block poses for privacy while showing a security prompt. In either case, because you already called XRSession.requestAnimationFrame() again, the loop can continue if the system recovers. If it cannot, the user agent ends the session and invokes the end event handler.

Bind the framebuffer

The XRWebGLLayer supplies a framebuffer made for WebXR, replacing the WebGLRenderingContext's default. Binding this framebuffer to the context is done in WebGL's terminology with bindFramebuffer().

function onXRFrame(hrTime, xrFrame) {
  let xrSession = xrFrame.session;
  xrSession.requestAnimationFrame(onXRFrame);
  let xrViewerPose = xrFrame.getViewerPose(xrRefSpace);
  if (xrViewerPose) {
    let glLayer = xrSession.renderState.baseLayer;
    webGLRenContext.bindFramebuffer(webGLRenContext.FRAMEBUFFER, glLayer.framebuffer);
    // Iterate over the views
  }
}

Loop through the views

With the pose and framebuffer in hand, iterate the XRView array from the viewer pose. Each view includes per-display properties such as field of view, eye offset, and optical data needed to render correctly for the device and the user. Drawing for a headset means two views—one per eye—and a distinct image is drawn for each.

For phone-based AR, there is only one view. Keeping the loop even then yields a single rendering path across different immersive experiences, an intentional difference between WebXR and earlier immersive systems.

function onXRFrame(hrTime, xrFrame) {
  let xrSession = xrFrame.session;
  xrSession.requestAnimationFrame(onXRFrame);
  let xrViewerPose = xrFrame.getViewerPose(xrRefSpace);
  if (xrViewerPose) {
    let glLayer = xrSession.renderState.baseLayer;
    webGLRenContext.bindFramebuffer(webGLRenContext.FRAMEBUFFER, glLayer.framebuffer);
    for (let xrView of xrViewerPose.views) {
      // Pass viewports to the context
    }
  }
}

Move the viewport to the context

An XRView describes what is observable on a display, but drawing requires device-specific coordinates. As with the framebuffer, you request the viewport from the XRWebGLLayer and hand it to the WebGLRenderingContext.

function onXRFrame(hrTime, xrFrame) {
  let xrSession = xrFrame.session;
  xrSession.requestAnimationFrame(onXRFrame);
  let xrViewerPose = xrFrame.getViewerPose(xrRefSpace);
  if (xrViewerPose) {
    let glLayer = xrSession.renderState.baseLayer;
    webGLRenContext.bindFramebuffer(webGLRenContext.FRAMEBUFFER, glLayer.framebuffer);
    for (let xrView of xrViewerPose.views) {
      let viewport = glLayer.getViewport(xrView);
      webGLRenContext.viewport(viewport.x, viewport.y, viewport.width, viewport.height);
      // Draw something to the framebuffer
    }
  }
}

The variable shown as webGLRenContext is conventionally named gl in sample code, matching method names in the OpenGL ES 2.0 API used for VR in compiled languages. That naming is convenient for developers coming from OpenGL—less so for newcomers—and the longer name is used here only for clarity.

Draw

Drawing directly to the framebuffer with WebGL is possible but much simpler through a framework listed earlier. The details of actual scene rendering fall outside this article's scope.

Further reading

A complete reference to WebXR's interfaces and members is maintained at MDN. Upcoming interface enhancements are tracked by feature on Chrome Status.