Why ASAN Can’t See Apache’s Pool Allocations
Address Sanitizer tracks memory through a shadow map, with per-byte state that marks regions as accessible or poisoned. Every instrumented memory access is prefixed with a check: if the target byte is in a poisoned “red zone,” the program aborts with a diagnostic. This is effective at catching classic out-of-bounds reads and writes, but it breaks down when a target manages memory internally in ways ASAN doesn’t understand.
ASAN exposes a manual poisoning API so developers can mark arbitrary regions as poisoned or unpoisoned, usually wrapping calls to malloc and free to create red zones between chunks. That works, but it requires you to rewrite allocation logic for every new target. Custom interceptors offer a cleaner path: hook the exact functions the target uses and let ASAN manage memory as if it were raw heap.
Apache’s Memory Pools Defeat Default ASAN
Apache HTTP uses apr_palloc to allocate from its own memory pools rather than calling malloc directly. These pools are implemented as a linked list of nodes, each with a free-space pointer and a linked list of bookkeeping structs. When a request comes in for, say, 126 bytes (in_size = 126), the allocation is rounded up to meet alignment requirements, and if the current node lacks space, allocator_alloc creates a new node.
That node’s size is rounded up to MIN_ALLOC bytes (typically 8192). Under the hood, the function calls malloc(8192). The result is a mismatch between what your code asked for and what ASAN sees as writable:
- You requested 126 bytes from
apr_palloc. - ASAN sees a 8192-byte allocation from
mallocand marks it fully accessible. - The pool node only “owns” 126 bytes, but ASAN considers all 8192 bytes writable.
The 8066 bytes between the actual allocation and the end of the node form silent red zones that ASAN doesn’t know about. A later memcpy of 5000 bytes into that region overflowes the node’s boundaries but passes undetected, potentially laying the groundwork for a vulnerability like CVE-2020-9273 in ProFTPD, which has a nearly identical pool implementation.
Building Compiler-RT with Custom Hooks
ASAN interceptors live in the compiler-rt runtime, not in your target. To add your own, you need to build compiler-rt yourself and link the patched runtime into the Apache build. For this example, we use compiler-rt 9.0.0:
cd compiler-rt-9.0.0.src
mkdir build-compiler-rt
cd build-compiler-rt
cmake ../
make
After the build finishes, point Apache’s build process at the custom runtime with the relevant environment variables, and set LD_LIBRARY_PATH similarly so the patched libasan is loaded:
LD_LIBRARY_PATH= /Downloads/compiler-rt-9.0.0.src/build-compiler-rt/lib/linux
CFLAGS="-I/Downloads/compiler-rt-9.0.0.src/lib -shared-libasan”
LD_LIBRARY_PATH=/Downloads/compiler-rt-9.0.0.src/build-compiler-rt/lib/linux
How ASAN Interceptors Actually Work
At program startup, __asan_init calls AsanActivate and AsanInternal. The latter performs most of the initialization, including InitializeAsanInterceptors. That function contains an ASAN_INTERCEPT_FUNC entry for every function ASAN hooks by default.
The macro expands to INTERCEPT_FUNCTION_LINUX_OR_FREEBSD, which calls InterceptFunction. That routine walks the symbol table with GetFuncAddr, which invokes dlsym() to find the address of the real function at load time. That address is stored into the ptr_to_real pointer, giving the interceptor a way to call the original implementation.
Adding a new interceptor involves two steps:
- Define an
INTERCEPTORfor the target function, e.g.,INTERCEPTOR(void*, foo, const char *bar, double baz) { ... }. - Call
ASAN_INTERCEPT_FUNC(foo)fromInitializeAsanInterceptors, before the target function is first used.
A Practical Interceptor: apr_palloc
Let’s apply this pattern to apr_palloc. The key is to prevent the inner malloc from being ASAN-instrumented—otherwise ASAN marks the entire node as writable. Instead, we replace that call with __libc_malloc, bypassing ASAN’s interception entirely. After the real apr_palloc returns, we align the requested size the same way APR does and allocate a separate ASAN-managed block of the actual usable size with asan_malloc. That second block becomes the only region ASAN treats as accessible.
The interceptor starts with an ENSURE_ASAN_INITED() check, then captures the current stack trace early via GET_STACK_TRACE_MALLOC so ASAN reports appear with a useful call chain. It then calls the original pool allocation through REAL(apr_palloc).
To reclaim this memory correctly, we also intercept the node-destruction paths: allocator_free and apr_allocator_destroy. Those functions need to release every ASAN-managed block we created for the node. The simplest implementation keeps an array of addresses per node, traverses it to call free on each entry, and then releases the node itself via __libc_free(node) to avoid recursive interception.
That approach is straightforward but slow: it requires scanning the whole address list per node. A better structure would map each node to a set of ASAN-tracked regions, with a std::set or custom collection of unique pointers, so reclaim is O(1) per allocation rather than a full traversal.
Extending Beyond Memory Pools
The same technique generalizes beyond custom allocators. If a target performs file-system operations in ways you want to validate for logic errors—such as path traversal or directory-restriction bypass—you can intercept the relevant syscalls with the same workflow. Define the INTERCEPTOR, register it via ASAN_INTERCEPT_FUNC, and use the captured stack trace to pinpoint exactly which caller triggered the faulty behavior. Fuzzing then surfaces not just memory corruption, but semantic violations in how the application handles its I/O boundaries.
Watching the syscall layer
Memory corruption bugs respond well to classic coverage-guided fuzzing, but logic flaws such as access-control bypasses often slip through. When a fuzzer drives a file server, HTTP methods are turned into filesystem syscalls on the remote host: open(), write(), read(). Bugs in this path may not crash the process; instead, they cause improper file interactions that go unnoticed by a standard fuzzer.
A different observation technique is needed. One basic approach is to log filesystem activity during the fuzz run for later inspection. The analysis goal is to compare the low-level syscalls with their high-level counterparts — the intended file operations based on the HTTP request — to verify that the syscall sequence and its arguments match expectations.
Instrumenting the handler layer
To demonstrate, consider three WebDAV methods: PUT, MOVE, and DELETE. Each triggers a distinct high-level Apache function, typically named for the operation it carries out:
static int dav_method_put(request_rec *r)
static int dav_method_copymove(request_rec *r, int is_move)
static int dav_method_delete(request_rec *r)
Instrumentation begins by inserting a call to a logging function, log_high, at the top of each of these handlers. A corresponding ENABLE_LOG = 0; line disables logging at the end of the function to mark completion of the operation. The log_high routine records the method and relevant arguments for later correlation:
The implementation of the logging helper itself:
Intercepting syscalls
On the low-level side, AddressSanitizer's interception mechanism captures the filesystem syscalls issued by Apache. For this use case three syscalls are sufficient to observe the relevant behavior for the WebDAV methods under test:
openrenameunlink
Each intercepted syscall is appended to an output log alongside its arguments and return value. A sample of the resulting log file looks like this:
The two logs — high-level handler calls and low-level syscalls — can be joined chronologically to examine whether the handlers performed the expected operations on the file system.
Analyzing the traces
The volume of trace data from a fuzzing session calls for a systematic analysis workflow. For a-posteriori inspection, indexing the output with Elasticsearch is one way to run queries over the combined logs. Automating a real-time analysis pipeline directly inside the AFL++ loop is something we will cover in a later write-up.
