Bringing VR and AR into the browser

The immersive web brings virtual world experiences directly into the browser — either as full virtual reality (VR) environments on headsets like Google's Daydream, Oculus Rift, Samsung Gear VR, HTC Vive, and Windows Mixed Reality, or as augmented reality (AR) experiences on mobile devices. These two poles sit at opposite ends of a spectrum, with mixed reality in between.

Typical immersive web applications include 360° video, traditional 2D or 3D video in immersive surroundings, data visualizations, shopping, art, and anything developers dream up next.

From WebVR to WebXR

The early immersive web ran on the WebVR 1.1 API, available behind an origin trial since Chrome 62 and supported by Firefox and Edge (with a Safari polyfill). That trial ended in July 2018, and the spec has since been superseded by the WebXR Device API, now in its own origin trial.

WebVR 1.1 revealed several design flaws that made it hard to build the applications developers wanted. For one, the API was tightly coupled to the main JavaScript thread. It also gave too many ways to set up invalid configurations. And "magic window" — viewing immersive content in a single rendered view on a flat screen using device orientation — was an accident rather than an intended, first-class feature.

The WebXR Device API redesign aims to fix these issues: it enables simpler implementations and better performance while staying extensible for future AR use cases. The implementors of WebVR have committed to migrating to WebXR.

What the WebXR Device API handles

The WebXR Device API is developed by the Immersive Web Community Group, with contributions from Google, Microsoft, Mozilla, and others. It's available via the origin trial and an accompanying polyfill. At launch, only VR capabilities were enabled; AR support arrived in Chrome 69. The API does not cover rendering — you draw your scenes with WebGL directly or via a framework such as Cottontail (used in the samples) or Three.js.

Starting and stopping a session

The application lifecycle works like this: request a device, request a session, run a render loop at 60 frames per second, then end the session when the user exits. Coding this properly requires handling both feature detection and user gestures for immersive sessions.

  1. Detect WebXR support and request a device.
  2. If you need an immersive session, prompt for a user gesture first. Non-immersive sessions ("magic window") don't require one.
  3. Request a session with a drawing canvas.
  4. Render frames until the user ends the session.
  5. End the session and clean up.

For an immersive session, you'll typically wrap the device and session requests inside a user gesture handler:

if (navigator.xr) {
    navigator.xr.requestDevice()
    .then(xrDevice => {
    // Advertise the AR/VR functionality to get a user gesture.
    })
    .catch(err => {
    if (err.name === 'NotFoundError') {
        // No XRDevices available.
        console.error('No XR devices available:', err);
    } else {
        // An error occurred while requesting an XRDevice.
        console.error('Requesting XR device failed:', err);
    }
    })
} else{
    console.log("This browser does not support the WebXR API.");
}

Once you have a device and the user gesture, create the session and pass it a canvas:

xrPresentationContext = htmlCanvasElement.getContext('xrpresent');
let sessionOptions = {
    // The immersive option is optional for non-immersive sessions; the value
    //   defaults to false.
    immersive: false,
    outputContext: xrPresentationContext
}
xrDevice.requestSession(sessionOptions)
.then(xrSession => {
    // Use a WebGL context as a base layer.
    xrSession.baseLayer = new XRWebGLLayer(session, gl);
    // Start the render loop
})

The term "frame" means two different things here. A frame of reference defines the coordinate system's origin and how it behaves when the device moves. A presentation frame, represented by an XRFrame, holds the data needed to draw a single scene — even though you get it by calling something similar to window.requestAnimationFrame().

xrSession.requestFrameOfReference('eye-level')
.then(xrFrameOfRef => {
    xrSession.requestAnimationFrame(onFrame(time, xrFrame) {
    // The time argument is for future use and not implemented at this time.
    // Process the frame.
    xrFrame.session.requestAnimationFrame(onFrame);
    }
});

Every frame, you need to know where the viewer is pointing. Both viewers and input devices expose a pose — a 4x4 matrix in column-major order stored in a Float32Array. Retrieve the viewer's pose with XRFrame.getDevicePose() and always confirm you got one back before drawing anything:

let pose = xrFrame.getDevicePose(xrFrameOfRef);
if (pose) {
    // Draw something to the screen.
}

After that, get the frames' views via XRFrame. Non-immersive sessions have one view; immersive sessions typically return two, one for each eye. Iterating uniformly matters because it lets one code path serve both types of sessions and a broad range of devices:

for (let view of xrFrame.views) {
    // Draw something to the screen.
}

Here's the full render loop, with a placeholder for handling input devices:

xrSession.requestFrameOfReference('eye-level')
.then(xrFrameOfRef => {
    xrSession.requestAnimationFrame(onFrame(time, xrFrame) {
    // The time argument is for future use and not implemented at this time.
    let pose = xrFrame.getDevicePose(xrFrameOfRef);
    if (pose) {
        for (let view of xrFrame.views) {
        // Draw something to the screen.
        }
    }
    // Input device code will go here.
    frame.session.requestAnimationFrame(onFrame);
    }
}

Ending a session can happen from your code with XRSession.end(), or externally — the headset may disconnect or another app may take over. Your app should monitor the end event, discard the session and renderer objects, and keep in mind that ended sessions cannot be restarted.

xrDevice.requestSession(sessionOptions)
.then(xrSession => {
    // Create a WebGL layer and initialize the render loop.
    xrSession.addEventListener('end', onSessionEnd);
});

// Restore the page to normal after immersive access has been released.
function onSessionEnd() {
    xrSession = null;

    // Ending the session stops executing callbacks passed to the XRSession's
    // requestAnimationFrame(). To continue rendering, use the window's
    // requestAnimationFrame() function.
    window.requestAnimationFrame(onDrawFrame);
}

How user input works

The WebXR Device API treats input with a "point and click" model. Each input device gets a pointer ray — a line from the controller indicating where it points. Your application draws that ray on screen. When the user clicks, three events fire: select, selectStart, and selectEnd.

Retrieving the input device's pose and drawing the pointer ray happens inside your render loop. It looks something like this (simplified from the Immersive Web Community Group's Input Tracking sample):

let inputSources = xrSession.getInputSources();
for (let xrInputSource of inputSources) {
    let inputPose = frame.getInputPose(inputSource, xrFrameOfRef);
    if (!inputPose) {
    continue;
    }
    if (inputPose.gripMatrix) {
    // Render a virtual version of the input device
    //   at the correct position and orientation.
    }
    if (inputPose.pointerMatrix) {
    // Draw a ray from the gripMatrix to the pointerMatrix.
    }
}

Note that the select events only tell you an input device was used — they don't indicate what was selected. Handlers for these events should be attached to the XRSession object right when it becomes available:

xrDevice.requestSession(sessionOptions)
.then(xrSession => {
    // Create a WebGL layer and initialize the render loop.
    xrSession.addEventListener('selectstart', onSelectStart);
    xrSession.addEventListener('selectend', onSelectEnd);
    xrSession.addEventListener('select', onSelect);
});

To find out what the user selected, you again turn to poses — details will depend on your app or framework. For reference, check how Cottontail implements this in the Input Selection example:

function onSelect(ev) {
    let inputPose = ev.frame.getInputPose(ev.inputSource, xrFrameOfRef);
    if (!inputPose) {
    return;
    }
    if (inputPose.pointerMatrix) {
    // Figure out what was clicked and respond.
    }
}

What’s next for WebXR

Augmented reality support is slated to land in Chrome 69, with an initial Canary build expected around June 2018. The current implementation is still early, but testing it now is the best way to shape the API before it stabilizes. Feedback from developers will directly influence how the feature set evolves.

Track the ongoing work on WebXR Hit Test via ChromeStatus.com. Hit testing is what lets virtual objects snap to real-world surfaces, a core primitive for AR interactions. Separately, keep an eye on WebXR Anchors, which aim to improve pose tracking by letting you lock virtual content to a fixed point in physical space. Both features are under active development and represent the next step in making the immersive web practical beyond basic 360° video.