Approaching HTTP fuzzing differently

Fuzzing text-based protocols like HTTP requires a different toolkit than binary formats. Standard AFL mutations—bit flips, arithmetic tweaks, block splicing—work well for structured binary data but often produce garbage when applied to request lines and headers. For this Apache HTTP Server research, I built custom mutators tailored to HTTP's grammar and tested them systematically against coverage metrics.

Custom mutators for HTTP

The custom mutation strategies fall into two broad categories: piece-swapping operations and charset brute-force attempts.

Piece swapping takes two different requests from the input corpus and combines them at either the line or word level. Line swapping exchanges complete lines between requests, while word swapping mixes individual tokens. Both preserve enough structure that the resulting requests remain plausibly parseable by HTTP servers.

Charset brute-force mutators systematically substitute values from specific character sets:

  • 1-byte brute force covering 0x00 – 0xFF
  • 2-byte brute force covering 0x0000 – 0xFFFF
  • 3-letter combinations from [a-z]{3}
  • 4-digit combinations from [0-9]{4}
  • Mixed alphanumeric patterns
  • 3-byte and 4-byte string brute force using all possible strings of those lengths present in the input file

Example of Line swapping custom mutator

Example of word swapping custom mutator

Measuring mutator effectiveness

Before committing to a long fuzzing campaign, I ran 24-hour coverage tests with different mutator combinations. The baseline coverage using only the initial corpus was 30.5% line coverage and 40.7% function coverage. All tests ran with AFL_DISABLE_TRIM=1 and -s 123.

Table comparing different mutation strategies

The combination of line mixing plus AFL HAVOC consistently outperformed other configurations. This held both in the default module configuration and in a second test with additional Apache modules enabled—the same combination won again.

Winner: Line mixing + HAVOC

Test 2 winner: Line mixing + HAVOC

Coverage efficiency is one metric, but the goal is finding bugs. In the full fuzzing campaign I used all custom mutators available, accepting lower efficiency in exchange for broader exploration of the input space.

Grammar-based mutation with AFL++

A complementary approach uses Grammar-Mutator, a tool integrated into AFL++. With a grammar specification, the mutator produces structurally valid HTTP requests rather than relying purely on random changes:

make GRAMMAR_FILE=grammars/http.json
./grammar_generator-http 100 100 ./seeds ./trees


export AFL_CUSTOM_MUTATOR_LIBRARY=./libgrammarmutator-http.so
export AFL_CUSTOM_MUTATOR_ONLY=1
afl-fuzz …

I built a simplified HTTP grammar covering common verbs (GET, HEAD, PUT, and others), using short 1-byte strings as leaves. Later fuzzing stages use Radamsa—another general-purpose fuzzer available as an AFL++ custom mutator library—to lengthen those strings. Most additional vocabulary went into dictionaries rather than the grammar itself.

a simplified HTTP grammar specification

Configuring Apache for fuzzing

Apache HTTP Server's main configuration file is httpd.conf, containing one directive per line in [install_path]/conf. Additional configuration files can be pulled in via the Include directive, with wildcards supported. A trailing backslash continues a directive onto the next line, with no whitespace allowed after it.

Module strategy

Apache's modular architecture means each enabled module extends the attack surface. I followed an incremental path: start with few modules enabled using --enable-mods-static=few, establish a stable fuzzing workflow, then enable additional modules one at a time. Static linking via --enable-[mod]=static and --enable-static-support noticeably improved fuzzing speed.

./configure --enable-[mod]

Each module gets bound to a unique Location (directory or file) in httpd.conf, giving the fuzzer distinct server paths that exercise different modules. To simplify input generation, files in htdocs mostly have 1- or 2-byte filenames so AFL++ can easily guess valid requests like:

  • GET /a HTTP 1.0
  • POST /b HTTP 1.1
  • HEAD /c HTTP 1.1

Httpd.conf configuration example

Our 1-byte htdocs directory

Expanding dictionary limits

AFL's deterministic dictionary handling caps out at 200 entries. With multiple modules and their locations in httpd.conf, plus HTTP verbs from modules like webdav (PROPFIND, PROPPATCH, etc.), that limit becomes a bottleneck quickly.

I submitted a pull request to AFL++ adding an AFL_MAX_DET_EXTRAS environment variable to raise the deterministic dictionary size. This is useful beyond Apache fuzzing whenever richer dictionaries are needed.

Code modifications for fuzzability

MPM considerations

Apache HTTP Server 2.0 extends the modular design to core server functions. Multi-Processing Modules (MPMs) handle network port binding, request acceptance, and child process dispatch. On Unix systems the default is MPM event, but the --with-mpm=[choice] flag selects alternatives. I tested two configurations: the threaded event MPM and the single-process prefork MPM.

Rather than substituting file descriptors for sockets—the usual fuzzing approach—this exercise creates a real local network connection and sends input through it.

a new local network connection

sending the fuzzing input through the new local network connection

Reducing entropy and delays

General fuzzing changes follow patterns established in earlier socket-fuzzing work with FreeRDP. The key areas:

  • Entropy reduction: Replace random() and rand() calls with constant seeds; replace time(), localtime(), and gettimeofday() with constants; replace getpid() with a fixed value.
  • Delay removal: Strip out selected sleep() and select() calls to keep executions fast.
  • Crypto determinism: Disable checksum validation and set static nonces so cryptographic paths don't introduce input-dependent nondeterminism.

Disabling apr_sleep

Both patches containing all changes are available for review. The next part of this series covers filesystem syscall handling and the concept of "file monitors" for more efficient Apache fuzzing.

When the Fuzzer Becomes the Bug

A crash that only reproduces under AFL++ but never when launching the target directly is a classic heisenbug scenario. That’s exactly the situation I hit while fuzzing Apache HTTP. The initial assumption was a non-deterministic bug, so I wrote a script to launch the server 10,000 times, then 100,000 times. No crash.

cript that launched the application 10,000 times and redirect its stdout output to a file

Yet the crash was perfectly consistent under AFL++. I spent considerable time investigating environmental factors and AddressSanitizer (ASAN) influences. Nothing explained the reliable reproduction under the fuzzer. Eventually I began to suspect the tooling itself and turned to GDB for a closer look.

investigating the bug candidate using GDB

The crash surfaced inside sanitizer_stackdepotbase.h, specifically in the find function. This ASAN library code runs every time a new item is pushed onto the program stack. The s linked list was corrupted, and the segmentation fault came from dereferencing an invalid memory address in the s->link expression. A bug in ASAN itself seemed far-fetched, but the more I dug, the more plausible it became. At least I gained a solid understanding of ASAN internals along the way.

Still, pinpointing the source of the linked-list corruption was difficult. I needed to determine whether Apache or AFL++ was responsible. That’s when I brought in the rr debugger, a Linux reverse-execution tool that records and replays program flow. With rr I could step backward through execution and finally trace the root cause.

rr debugger

The culprit was not Apache or ASAN. AFL++ injects code at branch points to track coverage in a shared memory bitmap. The injected logic is essentially equivalent to:

cur_location = <COMPILE_TIME_RANDOM>;
shared_mem[cur_location ^ prev_location]++;
prev_location = cur_location >> 1;

The bitmap defaults to 64 kb, but the guard variable in my case held a value of 65576. That value overflowed the __afl_area_ptr array, overwriting program memory. Normally AFL++ warns when the map size is too small for the instrumented code, but in this instance it did not. The reason remains unclear. Setting the environment variable MAP_SIZE=256000 resolved the issue cleanly.

The takeaway: your fuzzing infrastructure can mislead you. When a crash won’t reproduce outside the fuzzer, consider the fuzzer itself as a suspect before chasing ghosts in the target.

Fuzzing Apache HTTP: Quick Start

If you want to get straight to fuzzing, here are the essential pieces.

First, apply the source patches:

patch -p2 < /Patches/Patch1.patch
patch -p2 < /Patches/Patch2.patch

Then configure and build Apache HTTP:

CC=afl-clang-fast CXX=afl-clang-fast++ CFLAGS="-g -fsanitize=address,undefined -fno-sanitize-recover=all" CXXFLAGS="-g -fsanitize=address,undefined -fno-sanitize-recover=all" LDFLAGS="-fsanitize=address,undefined -fno-sanitize-recover=all -lm" ./configure --prefix='/home/user/httpd-trunk/install' --with-included-apr --enable-static-support --enable-mods-static=few --disable-pie --enable-debugger-mode --with-mpm=prefork --enable-negotiation=static --enable-auth-form=static --enable-session=static --enable-request=static --enable-rewrite=static --enable-auth_digest=static --enable-deflate=static --enable-brotli=static --enable-crypto=static --with-crypto --with-openssl --enable-proxy_html=static --enable-xml2enc=static --enable-cache=static --enable-cache-disk=static --enable-data=static --enable-substitute=static --enable-ratelimit=static --enable-dav=static
make -j8
make install

Finally, launch the fuzzer:

AFL_MAP_SIZE=256000 SHOW_HOOKS=1 ASAN_OPTIONS=detect_leaks=0,abort_on_error=1,symbolize=0,debug=true,check_initialization_order=true,detect_stack_use_after_return=true,strict_string_checks=true,detect_invalid_pointer_pairs=2 AFL_DISABLE_TRIM=1 ./afl-fuzz -t 2000 -m none -i '/home/antonio/Downloads/httpd-trunk/AFL/afl_in/' -o '/home/antonio/Downloads/httpd-trunk/AFL/afl_out_40' -- '/home/antonio/Downloads/httpd-trunk/install/bin/httpd' -X @@

Useful resources from the original project:

Next Steps

The second part of this series will cover advanced fuzzing techniques, including custom interceptors and file monitors. I’ll also explain how to fuzz specific modules like mod_dav and mod_cache.

References