AV1 in the browser: porting a codec to WebAssembly
WebAssembly opens the door to bringing new capabilities into the browser before (or without) native support. It works as a high-performance polyfill, letting you port existing C/C++ or Rust code rather than rewriting it in JavaScript. A practical example: compiling the AV1 video decoder to WebAssembly and playing AV1 video in any modern browser.
The full source for this walkthrough is available at github.com/GoogleChromeLabs/wasm-av1. You can try the live demo with these two 24fps test files: video 1 and video 2.
Why AV1
Video makes up a massive share of web traffic — Cisco estimates around 80%. Reducing that data footprint means better compression. The Alliance for Open Media has been working on the AV1 codec to shrink video data considerably. Browsers will eventually ship native AV1 support, but the open source compressor and decompressor make it an ideal candidate to compile into WebAssembly for experimentation now.
Adapting the code for the browser
Getting AV1 into the browser starts with understanding the existing codebase. Two things stand out:
- The source tree uses
cmakefor its build. - The examples assume a file-based interface.
Most community code that builds on the command line has similar assumptions. The stream-like interface we build here to make AV1 run in the browser can apply to many other command-line tools.
Building with cmake and Emscripten
The AV1 authors have already started experimenting with Emscripten. In the root CMakeLists.txt of the AV1 repository, these build rules exist:
if(EMSCRIPTEN)
add_preproc_definition(_POSIX_SOURCE)
append_link_flag_to_target("inspect" "-s TOTAL_MEMORY=402653184")
append_link_flag_to_target("inspect" "-s MODULARIZE=1")
append_link_flag_to_target("inspect"
"-s EXPORT_NAME=\"\'DecoderModule\'\"")
append_link_flag_to_target("inspect" "--memory-init-file 0")
if("${CMAKE_BUILD_TYPE}" STREQUAL "")
# Default to -O3 when no build type is specified.
append_compiler_flag("-O3")
endif()
em_link_post_js(inspect "${AOM_ROOT}/tools/inspect-post.js")
endif()
The Emscripten toolchain can emit asm.js or WebAssembly. We target WebAssembly since it produces smaller output and runs faster. The existing rules compile an asm.js build for an inspector application. For our purposes, we add lines to get WebAssembly output just before the closing endif() in the rules above:
# Force generation of Wasm instead of asm.js
append_link_flag_to_target("inspect" "-s WASM=1")
append_compiler_flag("-s WASM=1")
With cmake, the first step is generating Makefiles, followed by make to compile. Since we use Emscripten, we need its compiler toolchain instead of the host compiler. That means passing the path to Emscripten.cmake (part of the Emscripten SDK) to cmake:
cmake path/to/aom \
-DENABLE_CCACHE=1 -DAOM_TARGET_CPU=generic -DENABLE_DOCS=0 \
-DCONFIG_ACCOUNTING=1 -DCONFIG_INSPECTION=1 -DCONFIG_MULTITHREAD=0 \
-DCONFIG_RUNTIME_CPU_DETECT=0 -DCONFIG_UNIT_TESTS=0
-DCONFIG_WEBM_IO=0 \
-DCMAKE_TOOLCHAIN_FILE=path/to/emsdk-portable/.../Emscripten.cmake
The path/to/aom parameter is the full path to the AV1 source; path/to/emsdk-portable/…/Emscripten.cmake is the path to the toolchain description file.
For convenience, a shell script locates the file:
#!/bin/sh
EMCC_LOC=`which emcc`
EMSDK_LOC=`echo $EMCC_LOC | sed 's?/emscripten/[0-9.]*/emcc??'`
EMCMAKE_LOC=`find $EMSDK_LOC -name Emscripten.cmake -print`
echo $EMCMAKE_LOC
Running make builds the whole tree including samples, and crucially produces libaom.a — the video decoder ready to incorporate into our project.
Designing the interface
Now we need a way to feed compressed video data into the library and get decoded frames back for display in the browser.
A good reference point in the AV1 tree is simple_decoder.c, which reads an IVF file and decodes it into a series of images.
Our interface lives in decode-av1.c.
The browser can't read files from the file system, so we abstract the I/O using a stream-like interface. On the command line, file I/O is already a stream — we can define our own interface that looks like streams and plug in whatever implementation we need.
DATA_Source *DS_open(const char *what);
size_t DS_read(DATA_Source *ds,
unsigned char *buf, size_t bytes);
int DS_empty(DATA_Source *ds);
void DS_close(DATA_Source *ds);
// Helper function for blob support
void DS_set_blob(DATA_Source *ds, void *buf, size_t len);
The open/read/empty/close functions mirror normal file I/O, mapping naturally onto file operations on the command line and onto something else inside the browser. The DATA_Source type is opaque from JavaScript and just encapsulates the interface. Keeping an API that closely follows file semantics makes it reusable for other command-line oriented codebases (like diff or sed).
We also define a helper DS_set_blob that binds raw binary data to those stream functions, so a blob can be read as if it were a sequential file. The reference implementation in blob-api.c is short:
struct DATA_Source {
void *ds_Buf;
size_t ds_Len;
size_t ds_Pos;
};
DATA_Source *
DS_open(const char *what) {
DATA_Source *ds;
ds = malloc(sizeof *ds);
if (ds != NULL) {
memset(ds, 0, sizeof *ds);
}
return ds;
}
size_t
DS_read(DATA_Source *ds, unsigned char *buf, size_t bytes) {
if (DS_empty(ds) || buf == NULL) {
return 0;
}
if (bytes > (ds->ds_Len - ds->ds_Pos)) {
bytes = ds->ds_Len - ds->ds_Pos;
}
memcpy(buf, &ds->ds_Buf[ds->ds_Pos], bytes);
ds->ds_Pos += bytes;
return bytes;
}
int
DS_empty(DATA_Source *ds) {
return ds->ds_Pos >= ds->ds_Len;
}
void
DS_close(DATA_Source *ds) {
free(ds);
}
void
DS_set_blob(DATA_Source *ds, void *buf, size_t len) {
ds->ds_Buf = buf;
ds->ds_Len = len;
ds->ds_Pos = 0;
}
A test harness outside the browser
Good practice says build unit tests alongside integration tests. With WebAssembly, that means building a test version of the interface that runs on the command line and does real file I/O underneath the DATA_Source API — so you can debug outside the browser.
That stream I/O code is straightforward:
DATA_Source *
DS_open(const char *what) {
return (DATA_Source *)fopen(what, "rb");
}
size_t
DS_read(DATA_Source *ds, unsigned char *buf, size_t bytes) {
return fread(buf, 1, bytes, (FILE *)ds);
}
int
DS_empty(DATA_Source *ds) {
return feof((FILE *)ds);
}
void
DS_close(DATA_Source *ds) {
fclose((FILE *)ds);
}
Abstracting the stream lets us use binary blobs in the browser and real files on the command line. The test harness code is in the example source file test.c.
Buffering multiple frames
Video playback normally buffers a few frames to keep things smooth. Here we buffer 10 frames before starting playback. After each frame displays, we decode another to keep the buffer full, preventing stutter.
In this simple example the entire compressed video is available to read, so the buffer isn't strictly necessary. But if we extend the data source interface to support streaming from a server, the buffering mechanism needs to be in place.
The code in decode-av1.c reads frames from the AV1 library into the buffer:
void
AVX_Decoder_run(AVX_Decoder *ad) {
...
// Try to decode an image from the compressed stream, and buffer
while (ad->ad_NumBuffered < NUM_FRAMES_BUFFERED) {
ad->ad_Image = aom_codec_get_frame(&ad->ad_Codec,
&ad->ad_Iterator);
if (ad->ad_Image == NULL) {
break;
}
else {
buffer_frame(ad);
}
}
The choice of 10 frames is arbitrary. More frames means longer pre-roll; too few risks stalling. Native browser implementations handle buffering far more complexly.
Rendering frames with WebGL
Buffered video frames need to get on the page as fast as possible — that's what WebGL is for.
WebGL treats an image as a texture painted onto geometry, and in WebGL everything is triangles. We can use the built-in gl.TRIANGLE_FAN for a simple rectangle.
One catch: WebGL textures expect RGB images with one byte per channel. The AV1 decoder outputs YUV format with 16 bits per channel, and each U or V value corresponds to 4 pixels in the output. So we need a color conversion before handing images to WebGL.
The function AVX_YUV_to_RGB() in yuv-to-rgb.c handles that conversion. When calling it from JavaScript, the converted image memory must be allocated inside the WebAssembly module's memory — otherwise the module can't access it. The function to get an image out of the module and onto the screen:
function show_frame(af) {
if (rgb_image != 0) {
// Convert The 16-bit YUV to 8-bit RGB
let buf = Module._AVX_Video_Frame_get_buffer(af);
Module._AVX_YUV_to_RGB(rgb_image, buf, WIDTH, HEIGHT);
// Paint the image onto the canvas
drawImageToCanvas(new Uint8Array(Module.HEAPU8.buffer,
rgb_image, 3 * WIDTH * HEIGHT), WIDTH, HEIGHT);
}
}
The drawImageToCanvas() WebGL painting code lives in draw-image.js.
What the demo shows
Running the AV1 decoder against two test clips recorded at 24 f.p.s. points to two conclusions. First, complex, performance-sensitive code bases can be ported to the browser with WebAssembly. Second, even a decode workload as heavy as video can keep up.
That second point deserves some nuance. The current implementation runs entirely on the main thread, which means frame painting and decoding compete for time in the same event loop. Until decode time is predictable, moving that work into a web worker would smooth playback and keep decode stalls from blocking rendering.
Headroom on the CPU
The compiled WASM binary targets a generic CPU, much like a command-line build with the same configuration. Unsurprisingly, those two builds show comparable CPU usage. What's missing is the speed you'd get from the AV1 library's native SIMD paths, which can run about five times faster on typical hardware. The wider WebAssembly ecosystem is still catching up; the WebAssembly Community Group is designing SIMD primitives as a language extension. Once supported by browsers, the impact on decode-heavy tasks should be significant.
The takeaway
Sustained real-time WebAssembly decoding is still gated by a few key improvements, notably SIMD support and off-main-thread execution. But the project shows the baseline is already workable. The example also serves as a practical reference for porting an existing command-line tool to a usable WebAssembly module with modern web APIs.
