Pixel-Level Editing on the Web: CPU vs. GPU

Applying filters and effects to images or video is a core feature of many popular apps. On the web, the process is essentially identical for both media types: you grab the pixel data from the source and create a new image from it. The only interface for doing so is a canvas.

The critical architectural decision is whether to process pixels on the CPU (using a 2D canvas) or on the GPU (using WebGL). The 2D canvas is far simpler; WebGL offers performance but requires working with shaders and triangles.

The 2D Canvas Approach

With a 2D canvas, you start by drawing the image to the canvas, then retrieve an array of its pixel data. This array is a Uint8ClampedArray of length width * height * 4, where each set of four elements describes the red, green, blue, and alpha values of a single pixel, ordered from the top-left corner.

To access a specific pixel at coordinates x, y, the index is a simple formula: (y * width + x) * 4. After modifying this array, you must write it back to the canvas using putImageData(). Note that the array is a copy; the original canvas pixels are not affected until you explicitly write back.

WebGL for GPU Processing

WebGL is fundamentally a triangle-drawing API. To manipulate a 2D image, you describe the image as a rectangle composed of two triangles, upload the image as a texture, and write two shaders:

  • Vertex shader: Calculates the screen positions of the triangle points.
  • Fragment shader: Runs once per pixel and returns the output color, often by sampling a texture.

Whichever method you choose, many effects are applied per-pixel. For operations like adjusting contrast, a fragment shader can perform the same calculations as a JavaScript loop. To sample an image at the correct location in a shader, you use floating-point texture coordinates (from 0 to 1) or pass the image’s pixel dimensions as a uniform vector for conversion.

From Single Pixels to Neighbors

Many common effects—such as Gaussian blur and edge detection—require looking at more than just the current pixel; they are convolution filters that also use the colors of neighboring pixels.

With a 2D canvas, this changes the logic because you must read the original colors while writing new ones. The solution is to make a copy of the original pixel data before processing. In WebGL, no changes are needed since shaders do not write to input textures.

A convolution filter works by applying a weighted matrix (a kernel) to a region. For each output pixel, the values of itself and its neighbors are multiplied by the kernel's corresponding weights, then summed and divided by the sum of the weights themselves, producing the new color.

Whole Image Transformations

Geometric effects come in different levels of complexity. Cropping and scaling are common; the 2D canvas context handles them by drawing only a subsection of the source. Rotation and reflection are also direct features of the 2D context.

For greater control, you can express any 2D transformation as a matrix and apply it with setTransform(), which can combine multiple operations (like rotation and translation) in one step. More complex effects, such as lens distortion or ripples, aren’t simple matrix operations. These require applying a specific offset to each destination coordinate to find the corresponding source pixel location, often based on a wave function.

Video: The Extra Frame Cost

All the pixel-level techniques transfer directly to video if you pass a video element into drawImage() (Canvas 2D) or texImage2D() (WebGL). However, this only processes the current frame. For continuous effects on a playing video, you must call this capture-and-process operation on every browser animation frame.

Performance becomes critical here. For a still image, a 100 ms delay between a button press and the effect rendering is often tolerable. For video, a delay of just 16 ms can create visible and disruptive jerkiness, making the choice of a GPU-based WebGL pipeline highly attractive for real-time applications.