Machine Learning in the Browser: A Conversation with Charlie Gerard

Machine learning has moved from the realm of data science into the everyday toolkit of web developers. In this episode of the Smashing Podcast, Drew McLellan sits down with Charlie Gerard, a developer and author of Practical Machine Learning in JavaScript, to discuss how ML can be applied within web development.

Gerard explains that machine learning is fundamentally about pattern recognition. A model is trained on large datasets to identify relationships — whether that's recognizing objects in images, understanding speech, or predicting user behavior — and the resulting model can then be deployed to make inferences on new data. The key shift for web developers is that these models can now run directly in the browser or in a Node.js environment, opening up a wide range of interactive and real-time possibilities that weren't practical before.

Where TensorFlow.js Fits In

The gateway to this world is TensorFlow.js, which brings the popular TensorFlow library to JavaScript. With it, developers can load pre-trained models, retrain them on custom data, or even build and train neural networks from scratch — all in the browser. This makes it feasible to add features like image classification, pose detection, or natural language processing to a web app without needing a dedicated backend with GPU resources.

The Question of Performance and Privacy

Running models client-side isn't just about convenience; it also has significant implications for privacy and latency. Because all processing happens on the user's device, no data needs to be sent to a remote server for inference. This is particularly compelling for applications dealing with sensitive or personal data — such as health metrics or camera feeds — since the data never leaves the local machine.

Photo of Charlie Gerard

Performance is another consideration. While modern browsers with WebGL acceleration make it possible to train and run fairly complex models, developers must still be mindful of the device's capabilities. Training a large model entirely in a browser can be slow, especially on lower-end hardware. Gerad's advice, therefore, is to lean on pre-trained models whenever possible. These models, often huge in size, have already been trained on powerful infrastructure; in the client context, you are only running the inference step, which is far less computationally demanding.

Bringing AI to Real-World Web Projects

One of the most accessible ways to integrate machine learning today involves leveraging pre-trained models through APIs. A developer can source a model trained by a large organization or research team and use it directly — for instance, one might pull in a model that gauges whether a photo's subject is looking at the camera, and then wire that into an interface.

One of the most pragmatic examples is using the PoseNet model for real-time body tracking. By analyzing video frames through the webcam, it provides joint coordinates that can drive creative or accessibility-focused interactions. Gerard notes that many use cases are artistic or interactive now, but there are real user-experience payoffs too — like detecting whether a user is visibly frustrated, warning against repetitive strain injuries, or simply enabling hands-free controls.

In essence, the market for ML in the web context boils down to two broad playbooks:

  • Using an existing ML API to interpret everyday data — video, audio, or text — that is captured client-side.
  • Running those models directly in the browser to avoid backend dependency, which favors speed and privacy even if edge computing means you might be handling less heavyweight workloads.

However, implementing ML in production brings unique professional challenges. Debugging a model isn't like debugging code — you can't toss in a console.log and inspect what went wrong. Understanding model outputs requires a grasp of the data that went into the model and the confidence scores it returns, which calls for a more analytical, layered debugging mindset than most web developers are accustomed to.

The Path Forward for the Web Developer

Getting started is easier than it seems. Gerard suggests that an understanding of basic JavaScript is enough to start experimenting with some of the higher-level APIs — and she insists that a heavy math background isn't a prerequisite when using pre-trained models. Instead, she advises starting with a prototype-based approach; usually, the difficulty isn't in getting an ML model to work, but in refining it to do exactly what the user needs.

Her final note is that ML shouldn't be justified as a replacement for engineers — rather, it's an augmentation. It's most useful when doing a tedious, mechanical task that a user hasn't fully understood yet or when it introduces an interaction mode that users wouldn't expect from a web page.

Relevant links from the podcast:

Sustained reading on Smashing:

Machine Learning Without the Math Degree

Charlie Gerard, senior front-end developer at Netlify, Google Developer expert, and author of Practical Machine Learning in JavaScript (Apress), joined the Smashing Podcast to demystify machine learning for web developers. Her central message: you don't need to be a data scientist or a math whiz to get started.

The standard definition of machine learning, she explains, is giving computers the ability to generate predictions without being explicitly told what to do. Instead of coding for specific circumstances, algorithms learn from data you provide and then make predictions based on that learning. For front-end developers, this is analogous to using a web API like navigator.getUserMedia — you don't need to know how the browser implements it under the hood, just what it's good for and how to use it. The same logic applies to established ML algorithms and tools.

Three Flavors of Learning

When approaching a machine learning problem, Gerard advises first identifying which type of problem you're trying to solve. She outlines three main categories that cover the vast majority of her projects:

  • Supervised learning: Your dataset includes both features and labels. For example, to predict a house price, you feed the algorithm a dataset of houses with features (number of rooms, garden, neighborhood) and labels (their market price). The model learns the correlation between them and can then predict a price for a new house based on its features. The model becomes specific to its training data; you shouldn't reuse a model trained on Parisian house prices to predict prices in a small Australian town without retraining it with a new dataset of the same structure.
  • Unsupervised learning: You have features but no labels. This approach is for finding trends or clustering entities together. It powers recommendation engines like Spotify's "people like you also listen to this," where the algorithm groups users by listening habits and other traits to suggest new content. It's not the right tool if you want a specific output for a specific question.
  • Reinforcement learning: The third category Gerard mentions, though she admits she uses the first two most often for her own work.

From Prototypes to Production

Machine learning in JavaScript opens up a range of applications beyond standard data analysis. Gerard points to examples such as dynamically generating alt text for images on image-heavy sites, filtering not-safe-for-work content in a Chrome extension, and using gesture recognition with a webcam. She describes a past prototype (a desktop app with Electron) that let her scroll a webpage by tilting her head up and down — a fun experiment with potential accessibility benefits.

The power of doing this in JavaScript, Gerard argues, is that you can stay in your existing ecosystem, avoid learning a new language or toolset, and quickly spin up prototypes to validate ideas and get buy-in. The language unlocks a broad set of platforms: web pages, Chrome extensions, desktop apps via Electron, mobile apps via React Native, and IoT projects with frameworks like Johnny-Five. While a large portion of the ecosystem runs on Python, Gerard notes that with JavaScript you can also build and train models in Node.js for better performance, then use the generated model in the browser.

Three Ways to Start With TensorFlow.js

Gerard outlines three increasingly complex approaches to machine learning, primarily using TensorFlow.js:

  1. Use pre-trained models: The simplest starting point. Load an existing model (trained on different datasets for object or people recognition) and learn the framework's basics. Gerard still uses this approach for many experiments — there's no point reinventing the wheel.
  2. Transfer learning: Retrain a pre-trained model with your own samples. If you need to recognize a left hand raising instead of a right, you retrain the model live in the browser with your own examples. This takes only a few seconds and yields a model well-suited to your specific gestures.
  3. Train everything in the browser: Write and train a model from scratch entirely in the browser. Gerard notes the main application she's seen for this is building visualizations to watch the training process, accuracy going up and down, and parameters being tweaked live.

For those finding TensorFlow.js's documentation intimidating, Gerard recommends ML5.js, an abstraction layer on top with a more beginner-friendly API. Alternatively, cloud APIs from Amazon, Google, and Microsoft can handle image or voice recognition without writing much code at all.

Practical Applications and Known Limits

Beyond fun demos, Gerard identifies real-world use cases: image classification for recycling bins via phone camera, text classification for detecting toxic comments on GitHub issues, sound classification to recognize clapping and automatically send emoji in chat, and voice recognition with accents. Her most compelling suggestion for day-to-day front-end work is predictive prefetching — using site analytics to preload resources on the page most likely to be visited next, improving performance and user experience. It also supports sustainability efforts by not requesting useless resources, though she acknowledges the carbon footprint of training models is a real concern.

Machine learning has definite limitations. A custom project where no pre-trained model exists requires large datasets that are often hard to find or build. Training can be time-consuming and experimental — Gerard recalls an eight-hour training run in Python that failed at the very end. Models can also be several megabytes, which matters on mobile connections, and model size continues to be a barrier to wider adoption.

There's also a matter of liability. "The model is a black box," Gerard warns. It's a function you don't fully understand. If your model makes consequential decisions — who gets a loan, who goes to prison — you need to be able to explain how it reached that conclusion.

Bias Is a Data Problem

On ethical concerns like image recognition failing on dark skin, Gerard stresses that bias originates in data, not in the algorithm itself. A model "doesn't care" about skin color; it only generates results based on what it was given — and that dataset was assembled by people. Developers are responsible for building datasets that are as fair and unbiased as possible, and it's okay to get this wrong initially. The fix isn't scrapping the model; it's retraining it with more diverse data.

She points to positive examples, like Google's work to remove gender bias in translation, but laments that IBM's exemplary Diversity in Faces dataset is no longer available. This is an area where user pressure matters, she says: if companies want machine learning adopted, they must ensure everyone can use it.

What's Next

Gerard is currently studying digital signal processing to pair with machine learning, inspired by noise-canceling headphones. By filtering out unwanted ambient noise and data from hardware like an Arduino, she can feed a model cleaner signals for better gesture predictions.

Her book, Practical Machine Learning in JavaScript, aims to make the topic less intimidating, helping developers understand what algorithms do and which to use, without diving into source-level math. Her parting advice for those curious: take time to explore the fun side first, regardless of whether your end goal is entertainment or enterprise.

For more of Gerard's work, you can follow her on Twitter at @devdevcharlie or visit her website at charliegerard.dev.