Memory pressure on Mess With DNS

Mess With DNS has periodically run out of memory for years, and I mostly ignored it — the service would restart in a few minutes and it rarely happened more than once a day. But when it started interfering with my new database backup process, I finally dug in. The VM has 465MB of RAM: about 100MB for PowerDNS, 200MB for Mess With DNS, 40MB for hallpass, leaving roughly 110MB free. I had set GOMEMLIMIT to 250MB to keep Go's garbage collector in check, but that wasn't enough.

The actual trigger was restic: when it ran, it sometimes needed more memory than was available and got OOM-killed. Since restic holds a lock while running, a killed backup meant manual intervention to unlock before the next backup could start — the kind of maintenance I want to avoid entirely. Rather than add memory, I decided to reduce what Mess With DNS itself was consuming.

The memory hog: IP address lookups

I already knew from past profiling exactly where the memory went: Mess With DNS loads a full IP-to-ASN database into memory at startup so it can answer queries like "which ASN owns 74.125.16.248?" and respond with GOOGLE. The original text files were only 37MB, but the in-memory representation was using about 117MB.

$ du -sh *.tsv
26M	ip2asn-v4.tsv
11M	ip2asn-v6.tsv

The naive implementation stored each range as a struct and did a binary search over an array:

type IPRange struct {
	StartIP net.IP
	EndIP   net.IP
	Num     int
	Name    string
	Country string
}

That approach is extremely fast — about 9 million lookups per second — and I was hesitant to give up the performance.

First attempt: move the data to SQLite

Since I'd been using SQLite frequently, my first idea was to put the table on disk and query it with SQL. I wrote a quick Python script using sqlite-utils to import the TSV data, then changed my Go code to run SELECT queries instead of doing binary search. Memory usage dropped to nearly nothing after a garbage collection — but there were complications.

IPv6 addresses are 128-bit values and SQLite doesn't support big integers, so I stored them as text. I made sure to expand the addresses with Python's ipaddress.ip_address(s).exploded so that string comparisons sort correctly. The schema looked like:

CREATE TABLE ipv4_ranges (
   start_ip INTEGER NOT NULL,
   end_ip INTEGER NOT NULL,
   asn INTEGER NOT NULL,
   country TEXT NOT NULL,
   name TEXT NOT NULL
);
CREATE TABLE ipv6_ranges (
   start_ip TEXT NOT NULL,
   end_ip TEXT NOT NULL,
   asn INTEGER,
   country TEXT,
   name TEXT
);
CREATE INDEX idx_ipv4_ranges_start_ip ON ipv4_ranges (start_ip);
CREATE INDEX idx_ipv6_ranges_start_ip ON ipv6_ranges (start_ip);
CREATE INDEX idx_ipv4_ranges_end_ip ON ipv4_ranges (end_ip);
CREATE INDEX idx_ipv6_ranges_end_ip ON ipv6_ranges (end_ip);

Performance, however, was poor: a microbenchmark showed only 17,000 lookups per second — roughly 500x slower. Mess With DNS is low-traffic enough that even this might have been workable, but the drop bothered me. Running EXPLAIN QUERY PLAN revealed the problem:

sqlite> explain query plan select * from ipv6_ranges where '2607:f8b0:4006:0824:0000:0000:0000:200e' BETWEEN start_ip and end_ip;
QUERY PLAN
`--SEARCH ipv6_ranges USING INDEX idx_ipv6_ranges_end_ip (end_ip>?)

SQLite only used the end_ip index and never consulted the start_ip index. I tried a compound index, running ANALYZE, and using INTERSECT to force use of both indexes — the last one worked but made the query a thousand times slower because it had to materialize both subquery results. I abandoned SQLite; the complexity wasn't worth the memory savings when a simpler fix might exist.

Second attempt: a trie

I then tried the ipaddress-go library, which offers trie-based IP lookup. The result was disappointing — using 800MB to store only the IPv4 addresses, and managing only 100K lookups per second. I likely misused the library, but I decided not to investigate further and instead focus on shrinking my original struct.

Third attempt: shrink the structs

My original Go struct stored both net.IP objects and repeated strings:

type IPRange struct {
	StartIP net.IP
	EndIP   net.IP
	Num     int
	Name    string
	Country string
}

I had three ideas:

  1. Deduplicate the ASN metadata (name, country) since many ranges share the same ASN.
  2. Replace net.IP with something more compact than a byte slice.
  3. If ranges were mostly consecutive, store only the start IP and derive the end implicitly.

I implemented the first idea quickly: store ASN info in a separate pool and keep only an index in each range entry:

type IPRange struct {
	StartIP netip.Addr
	EndIP   netip.Addr
	ASN     uint32
	Idx     uint32
}

type ASNInfo struct {
	Country string
	Name    string
}

type ASNPool struct {
	asns   []ASNInfo
	lookup map[ASNInfo]uint32
}

That saved 50MB, bringing total memory usage to 65MB. The ASN numbers themselves fit in a uint32 — the largest real value in the dataset was 401307, though a few malformed-looking entries reached 4294901931, still inside the range.

59.101.179.0	59.101.179.255	4294901931	Unknown	AS4294901931

The second idea was also straightforward. The Go standard library now has netip.Addr (which grew out of Tailscale's netaddr), a compact IP address type designed exactly to avoid the pointer indirection of net.IP and its underlying []byte. Swapping it in saved another 20MB, bringing usage to 46MB. I left the third idea for another day; at that point the improvements were satisfying enough.

Final result

Deploying this reduced Mess With DNS's memory footprint by about 70MB, bringing peak usage from roughly 200MB to around 130MB — and it freed up headroom for restic. Lookups did slow slightly in microbenchmarks, from 9 million to 6 million per second, mostly due to the added indirection. That tradeoff seemed acceptable. The structure still uses a bit more memory than the raw data files (46MB versus 37MB) — pointers and fixed-size types have overhead.

A few ideas from the community I haven't tried yet:

  • Go's unique package for deduplication (though it reportedly uses more memory due to 64-bit pointers).
  • Compiling with GOARCH=386 to shrink pointers to 32 bits.
  • Storing IPv6 addresses in 64 bits since only the first half of the address is publicly routable.
  • Using mmdbwriter or mmdbctl to serve from the MaxMind database format.

I used a small Go snippet with the runtime package to measure live allocations whenever I needed a number:

func memusage() {
	runtime.GC()
	var m runtime.MemStats
	runtime.ReadMemStats(&m)
	fmt.Printf("Alloc = %v MiB\n", m.Alloc/1024/1024)
	// write mem.prof
	f, err := os.Create("mem.prof")
	if err != nil {
		log.Fatal(err)
	}
	pprof.WriteHeapProfile(f)
	f.Close()
}

Memory profiling with go tool pprof also taught me the difference between --alloc-space (everything allocated since program start) and --inuse-space (what's live right now) — an easy thing to overlook when interpreting heap profiles.