Reordering razor cartridges shouldn't require knowing the right product keywords. For many sites, visual search—letting a user point their phone camera at a barcode, product, or landmark—could replace that friction entirely. The open-source Web Perception Toolkit makes this possible, and in many cases without writing custom detection code.

The toolkit routes a device camera stream through detectors that map real-world objects ("targets") to content on your site. You define that mapping using Structured Data (JSON-LD), and the toolkit renders the right information in a customizable UI. The core approach:

  1. Publish JSON-LD describing the targets you want recognized and the content to associate with them.
  2. Include the toolkit's scripts and stylesheet on your page.
  3. Handle (or skip) the lifecycle events if you want more than the default card UI.

If you provide only the structured data, the toolkit automatically identifies targets and shows or hides cards based on that data. For a complete example, see the artifact-map demo.

The default interface is available by using just the linked data.
The default interface.

Declaring targets with JSON-LD

Add your data as a JSON linked data file, included in the page with a <script> tag and the "application/ld+json" MIME type:

<script type="application/ld+json" src="//path/to/your/sitemap.jsonld">

The file defines your targets and the content cards for them:

[
  {
    "@context": "https://schema.googleapis.com/",
    "@type": "ARArtifact",
    "arTarget": {
      "@type": "Barcode",
      "text": "012345678912"
    },
    "arContent": {
      "@type": "WebPage",
      "url": "http://localhost:8080/demo/artifact-map/products/product1.html",
      "name": "Product 1",
      "description": "This is a product with a barcode",
      "image": "http://localhost:8080/demo/artifact-map/products/product1.png"
    }
  }
]

Handling the detection lifecycle

The toolkit's default UI works fine for many cases. To build a custom experience, listen for lifecycle events and use the provided Card and ActionButton objects.

The central event is PerceivedResults, fired whenever the toolkit perceives a target—an object, barcode, or QR code. Like any other event, you can register a handler. Unlike most events, if you don't, the toolkit falls back to its default structured-data-driven UI. To opt into a custom handler, call event.preventDefault() first:

const container = document.querySelector('.container');
async function onPerceivedResults(event) {
  // preventDefault() to stop default result Card from showing.
  event.preventDefault();
  // Process the event.
}
window.addEventListener(PerceptionToolkit.Events.PerceivedResults, onPerceivedResults);

The event object carries arrays of what the camera found and lost since the last fire. Because cameras shake and markers move in and out of frame, event.found and event.lost let you react to transitions, not just a single snapshot:

async function onPerceivedResults(event) {
  // preventDefault() to stop default result Card from showing
  event.preventDefault();
  if (container.childNodes.length > 0) { return; }
  const { found, lost } = event.detail;
  // Deal with lost and found objects.
}

Once you know what's in view, choose and render a card accordingly:

async function onPerceivedResults(event) {
  event.preventDefault();
  if (container.childNodes.length > 0) { return; }
  const { found, lost } = event.detail;
  if (found.length === 0 && lost.length === 0) {
    // Object not found.
    // Show a card with an offer to show the catalog.
  } else if (found.length > 0) {
    // Object found.
    // Show a card with a reorder button.
  }
}

Building the UI is just instantiating objects and appending to a parent:

const { Card } = PerceptionToolkit.Elements;
const card = new Card();
card.src = 'Your message here.'
container.appendChild(card)'

Putting it together, you can offer contextual one-click actions whether the marker is currently detected or was just lost:

async function onPerceivedResults(event) {
  // preventDefault() to stop default result Card from showing
  event.preventDefault();
  if (container.childNodes.length > 0) { return; }
  const { found, lost } = event.detail;
  const { ActionButton, Card } = PerceptionToolkit.Elements;
  if (found.length === 0 && lost.length === 0) {
    //Make a view catalog button.
    const button =  new ActionButton();
    button.label = 'View catalog';
    button.addEventListener('click', () => {
      card.close();
      //Run code to launch a catalog.
    });
    //Make a card for the button.
    const card = new Card();
    card.src = 'We wish we could help, but that\'s not our razor. Would you like to see our catalog?';
    card.appendChild(button);
    //Tell the toolkit it does not keep the card around
    // if it finds something it recognizes.
    card.dataset.notRecognized = true;
    container.appendChild(card);
  } else if (found.length > 0) {
    //Make a reorder button.
    const button = new ActionButton();
    button.label = 'Reorder';
    botton.addEventListener('click', () => {
      card.close();
      //Run code to reorder.
    })
    const card = new Card();
    card.src = found[0].content;
    card.appendChild(button);
    container.appendChild(card);
  }
}

Styling the cards

The toolkit ships a default stylesheet for cards and buttons. To brand the interface, the Card and ActionButton objects expose style properties for custom styling. Include the defaults with a normal <link> element:

<link rel="stylesheet" href="//path/to/toolkit/styles/perception-toolkit.css">

The toolkit's documentation and getting-started guide cover the rest—further API details and a sandbox demo are available at perceptiontoolkit.dev and the I/O Sandbox.