Why Envoy needed a pluggable design

In a service mesh, Envoy typically runs alongside each service as a sidecar proxy, handling the messy details of inter-service communication: service discovery, load balancing, retries, and the rest. The services themselves only deal with business logic. But that convenience creates its own problem: users frequently want to customize Envoy itself — for example, by adding custom filters that act as middleware on HTTP traffic.

Envoy's original extension path was to write C++ and link a custom build. That works, but it forces you to distribute your own Envoy binaries, it entangles you with an API that was never designed to be stable, and it demands C++ expertise. A later approach added Lua extensions, which are stable and fully supported today. Lua was built for embedding; it's small, simple, and you can drop filter code directly into the Envoy config or point at a script file. Still, requiring operators to learn yet another language for proxy filters was a hard sell in a cloud ecosystem dominated by Go, Python, and Java. That pushed Envoy to adopt WebAssembly (WASM) as a third extension mechanism.

WASM solves both problems at once. An extension is compiled to a .wasm file, loaded dynamically at runtime — no custom Envoy binary needed. And because the module can be written in any language that targets WASM, teams can write filters in Go and use an SDK like the one from Tetrate instead of reaching for C++ or Lua. Envoy embeds v8 as its WASM VM; the remaining piece was defining an interface between those modules and the proxy host.

The Proxy-Wasm ABI

WASM defines a bytecode format and execution semantics, plus the mechanics for modules to export and import functions and data. Everything else — notably data types — is left to the host. WASM's type system is essentially fixed-width integers and floats; anything richer must be built by the user over addresses into WASM's linear memory.

WASI, which provides OS-like functionality to WASM modules, is useful for exposing modules to the outside world, but its current interface model is too constrained for complex host-module interactions. Envoy therefore needed a custom interface, which it defines in the Proxy-Wasm ABI, hosted as a separate project from Envoy proper.

The ABI is deliberately low level and has two halves. First are functions implemented in the WASM module, exported from the extension and imported by the host. A representative example is proxy_on_request_headers, a callback the module exports to intercept HTTP request headers:

void proxy_on_request_headers(uint32_t context_id, uint32_t num_headers, bool end_of_stream, bool end_stream)

The host side lives in the proxy-wasm-cpp-host project, a C++ host implementation that Envoy depends on. The second half is functions implemented in the host environment, which the module imports. An example is proxy_get_header_map_value, used by an extension to inspect actual HTTP headers:

proxy_get_header_map_value(uint32_t context_id, uint32_t map_type, const char* key, size_t key_size, const char** value, size_t* value_size)

Everything is a pointer (an address in WASM linear memory) or a constant of a predefined type. That's an inevitable consequence of WASM's restrictive parameter types combined with modules and hosts written in diverse languages. The glue code at this level is tedious, which is what motivated high-level SDKs.

The Go SDK in practice

Using the Go SDK for Proxy-Wasm, a filter that snoops on HTTP traffic and logs headers is expressed in plain Go:

func (ctx *httpHeaders) OnHttpRequestHeaders(numHeaders uint32, endOfStream bool) types.Action {
	hs, err := proxywasm.GetHttpRequestHeaders()
	if err != nil {
		proxywasm.LogCriticalf("failed to get request headers: %v", err)
	}
	for _, h := range hs {
		proxywasm.LogInfof("request header: %s = %s", h[0], h[1])
	}
	return types.ActionContinue
}

Underneath, the SDK is a thin wrapper over the ABI. For instance, proxywasm.GetHttpRequestHeaders (used above) simply calls a general getMap helper with an ABI-defined map type, returning a slice of key/value pairs. The core call, internal.ProxyGetHeaderMapPairs, corresponds to the ABI function proxy_get_header_map_pairs imported from the host. It writes raw pointers into its output parameters; the rest of getMap converts those to Go types.

On the module side, callbacks are exported from WASM. The SDK exports proxy_on_request_headers via a //export annotation, which instructs the compiler to place that function in the WASM export table. When the host invokes it, the SDK routes execution to the user-defined OnHttpRequestHeaders context method shown above.

The Go SDK is only one option — Rust and C++ SDKs exist as well. One caveat: the Go SDK currently requires TinyGo, not the standard Go toolchain, because the default toolchain lacks sufficient WASM support. That is changing: Go 1.21 added WASI support, and ongoing work is moving toward capability for Envoy extensions with the standard compiler.

Plugin infrastructure concepts, mapped

Discovery

Discovery is trivial: extensions must be listed explicitly in Envoy's config file. For WASM, the config entry points to a local .wasm file or a remote URL such as a cloud storage bucket.

Registration

Extensions register functionality by exporting well-known function names from the WASM module. When Envoy loads the module, it scans exports for names it recognizes. If proxy_on_request_headers is present, Envoy invokes it for each request's headers; absence means the extension isn't interested in that callback.

A second registration mechanism is the proxy_abi_version_X_Y_Z export. The extension exports this function with the ABI version substituted in the name; Envoy looks for the proxy_abi_version_* prefix and determines from the name which ABI revision the module was built against.

Hooks

Hooks are the exported callbacks themselves. proxy_on_request_headers is one of many such entry points defined by the ABI.

Exposing an application API to plugins

That is the role of the host-implemented ABI functions. proxy_get_header_map_pairs is one example; proxy_log (for emitting messages to Envoy's log) is another. These imports give extensions their capability to call back into Envoy.

The takeaway

The Envoy WASM case study shows what an extension ABI looks like when the requirements exceed what WASI can offer. Custom network filter plugins need deep, frequent interaction with the proxy host, not a narrow interface modeled on standard I/O. The result — a custom ABI plus a family of language SDKs layered on top — is a template for advanced plugin systems in other domains. The low-level ABI does the heavy lifting; the SDKs rescue developers from the dangerous details. Notably, the Go SDK's example code presents its filter logic as idiomatic Go, even though the entire module compiles down to a .wasm blob loaded at runtime.