Adding real-world hit testing to WebXR

The WebXR Device API arrived in Chrome 79, and Chrome 81 adds two pieces of that implementation: augmented reality session types and hit testing. This article covers the WebXR Hit Test API, which lets you place virtual objects into a real-world camera view. I'll assume you already know how to start an AR session and run a frame loop.

The code here is based on the Immersive Web Working Group's hit test sample, which places virtual sunflowers on real surfaces. When the app starts, you see a blue circle with a dot in its center. That dot marks where an imaginary line from your device intersects the environment, and it moves with your device. As hit testing finds intersection points, the dot snaps to surfaces like floors, tabletops, and walls. Because hit testing only returns the position and orientation of the intersection point—not information about the surface itself—this temporary reticle image guides object placement. Tapping the screen places a sunflower at the reticle's current location and orientation.

A reticle rendered on a wall, Lax, or Strict depending on their context
The reticle is a temporary image that aids in placing an object in augmented reality.

Setting up for hit testing

The reticle is not provided by the browser or the API; you must create and draw it yourself. The loading and rendering method depends on your framework, so I won't detail the drawing code. I do show its creation below so you know what the reticle variable refers to in later code.

let reticle = new Gltf2Node({url: 'media/gltf/reticle/reticle.gltf'});

Your session request must include 'hit-test' in the requiredFeatures array:

navigator.xr.requestSession('immersive-ar', {
  requiredFeatures: ['local', 'hit-test']
})
.then((session) => {
  // Do something with the session
});

When entering a session, the code below adds a select event listener that places a flower at the reticle's pose when the user taps the screen. That listener is described later.

function onSessionStarted(xrSession) {
  xrSession.addEventListener('end', onSessionEnded);
  xrSession.addEventListener('select', onSelect);

  let canvas = document.createElement('canvas');
  gl = canvas.getContext('webgl', { xrCompatible: true });

  xrSession.updateRenderState({
    baseLayer: new XRWebGLLayer(session, gl)
  });

  xrSession.requestReferenceSpace('viewer').then((refSpace) => {
    xrViewerSpace = refSpace;
    xrSession.requestHitTestSource({ space: xrViewerSpace })
    .then((hitTestSource) => {
      xrHitTestSource = hitTestSource;
    });
  });

  xrSession.requestReferenceSpace('local').then((refSpace) => {
    xrRefSpace = refSpace;
    xrSession.requestAnimationFrame(onXRFrame);
  });
}

Why two reference spaces?

Notice that the code calls XRSession.requestReferenceSpace() twice. This can be confusing at first: why does the hit test code not start the frame loop, and why does the frame loop not seem to involve hit tests? The key is understanding reference spaces, which express relationships between an origin and the world.

Imagine a standalone rig with both a headset and a controller. To measure distances from the controller, you'd use a controller-centered frame of reference. But to draw to the screen, you'd use user-centered coordinates. In this sample, the viewer and controller are the same device. What gets drawn must stay stable relative to the environment, but the "controller" used for drawing is moving.

So the code uses the local reference space for drawing, which is stable relative to the environment, and starts the frame loop with requestAnimationFrame(). For hit testing, it uses the viewer reference space, based on the device's pose at the time of the test. The label "viewer" is confusing here because it refers to the device, not a person—think of it as an electronic viewer. That reference space is used to call xrSession.requestHitTestSource(), which creates the source of hit test data used during drawing.

The frame loop

The requestAnimationFrame() callback gains new logic for hit testing. Since the reticle must move as your device moves, it's redrawn each frame—but only when a hit test succeeds. The reticle's visible property is set to false at the start:

function onXRFrame(hrTime, xrFrame) {
  let xrSession = xrFrame.session;
  xrSession.requestAnimationFrame(onXRFrame);
  let xrViewerPose = xrFrame.getViewerPose(xrRefSpace);

  reticle.visible = false;

  // Reminder: the hitTestSource was acquired during onSessionStart()
  if (xrHitTestSource && xrViewerPose) {
    let hitTestResults = xrFrame.getHitTestResults(xrHitTestSource);
    if (hitTestResults.length > 0) {
      let pose = hitTestResults[0].getPose(xrRefSpace);
      reticle.visible = true;
      reticle.matrix = pose.transform.matrix;
    }
  }

  // Draw to the screen
}

Before drawing anything in AR, verify that hitTestSource and the xrViewerPose are still valid:

function onXRFrame(hrTime, xrFrame) {
  let xrSession = xrFrame.session;
  xrSession.requestAnimationFrame(onXRFrame);
  let xrViewerPose = xrFrame.getViewerPose(xrRefSpace);

  reticle.visible = false;

  // Reminder: the hitTestSource was acquired during onSessionStart()
  if (xrHitTestSource && xrViewerPose) {
    let hitTestResults = xrFrame.getHitTestResults(xrHitTestSource);
    if (hitTestResults.length > 0) {
      let pose = hitTestResults[0].getPose(xrRefSpace);
      reticle.visible = true;
      reticle.matrix = pose.transform.matrix;
    }
  }

  // Draw to the screen
}

Next, call getHitTestResults() with the hitTestSource as an argument. It returns an array of HitTestResult instances, one per surface found, ordered by distance from the camera. The first result is usually the one you want, but the array supports advanced scenarios—for instance, if your camera points at a box on a table on a floor, all three surfaces might be returned, and you'd typically care about the box. If the array is empty, no surface was found, and you simply retry next frame:

function onXRFrame(hrTime, xrFrame) {
  let xrSession = xrFrame.session;
  xrSession.requestAnimationFrame(onXRFrame);
  let xrViewerPose = xrFrame.getViewerPose(xrRefSpace);

  reticle.visible = false;

  // Reminder: the hitTestSource was acquired during onSessionStart()
  if (xrHitTestSource && xrViewerPose) {
    let hitTestResults = xrFrame.getHitTestResults(xrHitTestSource);
    if (hitTestResults.length > 0) {
      let pose = hitTestResults[0].getPose(xrRefSpace);
      reticle.visible = true;
      reticle.matrix = pose.transform.matrix;
    }
  }

  // Draw to the screen
}

To process results, get a pose from the first hit test result, move the reticle to that position, and set its visible property to true. The pose represents the position and orientation of a point on a surface:

function onXRFrame(hrTime, xrFrame) {
  let xrSession = xrFrame.session;
  xrSession.requestAnimationFrame(onXRFrame);
  let xrViewerPose = xrFrame.getViewerPose(xrRefSpace);

  reticle.visible = false;

  // Reminder: the hitTestSource was acquired during onSessionStart()
  if (xrHitTestSource && xrViewerPose) {
    let hitTestResults = xrFrame.getHitTestResults(xrHitTestSource);
    if (hitTestResults.length > 0) {
      let pose = hitTestResults[0].getPose(xrRefSpace);
      reticle.matrix = pose.transform.matrix;
      reticle.visible = true;

    }
  }

  // Draw to the screen
}

Placing an object on tap

The select event handler places an object when the user taps. Because the moving reticle provides a constant source of hit tests, the simplest placement method is to draw the new object at the reticle's location from the last successful hit test:

function onSelect(event) {
  if (reticle.visible) {
    // The reticle should already be positioned at the latest hit point,
    // so we can just use its matrix to save an unnecessary call to
    // event.frame.getHitTestResults.
    addARObjectAt(reticle.matrix);
  }
}

To get comfortable with these concepts, step through the sample code or try the codelab. More immersive web APIs are still in progress, so watch for future articles as the work continues.