Hardening a Go Service Against a Flaky Network
GitHub’s monolith is still Rails, but we’ve been steadily extracting critical paths into Go services, focusing on the pieces that need higher performance and reliability than Ruby can provide. One of those is authzd, the service that backs our "fine-grained authorizations" feature, deployed in production last year after the Satellite 2019 announcement. It marks a shift for us: authzd is the first Go service at GitHub that reads from production MySQL databases directly in the request path. Our earlier Go services touching MySQL, like the manticore search cluster manager or the gitbackups batch job, were either internal management tools or asynchronous tasks. authzd is different—it is invoked multiple times during a typical request to the Rails monolith, so its performance and reliability requirements are far stricter.
The network presents a particular operational hazard. authzd runs on our Kubernetes clusters, where opening new TCP connections has historically shown high latency. That behavior directly stresses the connection pooling in the Go MySQL driver. The uncomfortable truth we keep rediscovering is that most of the time the network is reliable—and that is exactly when we stop accounting for the fact that it sometimes isn't. When the network slows or becomes patchy, the weaknesses in libraries we depend on become visible.
Getting authzd to serve all production traffic within our availability SLOs required confronting these issues from a MySQL driver perspective. Here is what we found.
The MySQL Query Timeout Blind Spot
Go’s database/sql package, which wraps the MySQL driver, does not fully support query timeouts in the way you might expect. The standard pattern is to use context.WithTimeout to cancel a query after a deadline. When that context expires, the driver cancels the query server-side—so the MySQL server stops executing it—and the client returns the error context deadline exceeded. This behavior is what most developers rely on to protect their services from slow queries. However, there is a gap in the abort procedure.
The cancellation works by closing the underlying connection to the server, because the MySQL protocol uses the connection itself as the signal for interruption. The problem occurs when a future query reuses that connection. The driver must also ensure that the now-closed socket is not handed back to the pool. If the driver mistakenly considers the connection healthy after a context cancellation, the next query on that connection will be stuck writing to a dead socket, waiting for a response on a connection that no longer exists. This can manifest as a query that hangs indefinitely, rather than returning a clean error.
Dodgy Ping Checks
A common workaround for stale connections is to validate them before use. Calling db.Ping() or using the SetConnMaxLifetime function are both popular tactics. Configuring a maximum connection lifetime ensures that the driver periodically discards connections and opens fresh ones. Yet, with these mechanisms in place, a dead connection can still be returned from the pool if the disk write of the previous query happened to succeed just before the network failed. The driver assumes the connection is in a good state because the last operation completed successfully.
To combat this, many teams turn to SetConnMaxLifetime with aggressive values, but this approach merely reduces the window of exposure. It does not detect the problem until it happens again. The next query on the dead connection will then fail, and only after that failure will the pool discard the connection and open a new one. For a low-latency service, the cost of that first failure is often acceptable. But if the issue is systemic—say, network partitions—then the first query on every pooled connection will fail in turn, causing a wave of errors that can cascade into an outage.
The Next Generation: Driver Internals and Rotten Connections
The core of the issue lies in the interplay between two competing behaviors in the MySQL driver: its handling of context cancellation and its connection validation. The database/sql design gives the driver the responsibility of deciding whether a connection is usable after an error. In the case of a timeout, the correct decision is to mark the connection as broken and close it, as the underlying TCP connection is compromised.
We discovered that in certain versions of the driver, a race condition could occur between the connection’s read and write goroutines when a timeout struck at the same time a query was being sent. This race could lead the driver to behave incorrectly, reusing the connection that had, in fact, been closed. The result was that occasional timeouts turned into hard stalls, with queries never returning. During our rollout, this happened to authzd just as we were routing production traffic to it, and the service began accumulating `context deadline exceeded` errors that were, in fact, from future queries on poisoned connections.
Resolving these cases required two changes. First, we audited every place in our code where we used query cancellation to ensure that the error handling logic explicitly closed the connection on any error that could possibly be network-related. Second, we adjusted the driver configuration to set an absolute maximum time for any single query to be in flight, separate from the total request timeout that the caller provides. These steps, along with careful pool configuration, allowed authzd to handle the latency spikes and synthetic network failures we injected during chaos testing.
Why MySQL “Invalid connection” errors kept crashing Go services
The Go MySQL driver historically struggled with a persistent failure mode: "Invalid connection (unexpected EOF)". Reports of this bug date back years, surfacing in production across many services. In one deployment, it generated hundreds of errors per minute, sharply degrading application error rates.
The root cause is a mismatch between the TCP protocol and the MySQL wire protocol. MySQL connections are client-managed once established: the server only sends packets in response to client requests. If the server closes a connection that has been idle in Go’s database/sql pool, the TCP [FIN, ACK] is not noticed by the client immediately. Because TCP is full-duplex, the client’s write path stays open. When the client later sends a request, the server responds with a [RST], and the client receives an EOF during the read.
That explains the invalid connection—but not why database/sql does not transparently recover. The driver is supposed to signal invalid connections by returning driver.ErrBadConn, which prompts the pool to discard the connection and retry. However, in this scenario, returning ErrBadConn is not safe. The client wrote the request before discovering the connection was closed. The query may have reached the server before the reset, and retrying it on a new connection could execute it twice—especially dangerous for UPDATE statements. With that uncertainty, the driver could not automatically retry.
The first workaround and its flaws
A common band-aid was calling (*DB).SetConnMaxLifetime with a duration shorter than the server’s idle timeout. This forces database/sql to recycle connections before the server kills them. The approach has operational shortcomings. Most notably, the setting controls maximum total lifetime, not idle time. Active connections are churned unnecessarily, creating overhead. It also fails to handle cases where the server actively prunes connections at will.
The better approach is to check a connection’s health before writing to it. This was not feasible until Go 1.10 introduced the SessionResetter interface, which gives a driver visibility into when a connection is returned to the pool. With that hook, the driver can inspect a connection as it is being checked out, rather than waiting for a failed request.
A low-cost TCP-level health check
The fix avoids expensive MySQL-level pings in favor of a non-blocking read at the TCP layer. If the server has closed its side of the connection, the read returns EOF instantly. If the connection is still healthy, the read returns an EWOULDBLOCK—also instantly—since there is no pending data. In Go, all sockets are already non-blocking under the hood, but the standard net.Conn.Read would still put the goroutine to sleep via the scheduler. The required low-level access comes from (*TCPConn).SyscallConn, introduced in Go 1.9, which exposes the underlying file descriptor for direct read system calls.
With this mechanism, the driver can detect stale connections in under five microseconds of overhead and safely return driver.ErrBadConn before any application-level operation runs. In production deployments, this eliminated the invalid-connection errors immediately.
Production guidance
- If your MySQL server uses idle timeouts or actively prunes connections, avoid
(*DB).SetConnMaxLifetime. The driver now handles stale connections gracefully, so enforcing a lifetime only causes unnecessary churn. - For high-throughput workloads, configure
(*DB).SetMaxIdleConnsand(*DB).SetMaxOpenConnsto support peak traffic. Let the MySQL server prune idle connections during off-peak hours—the driver will detect and replace them as needed.
Context Lost in the Connection Pool
When a service like authzd sits in the request path of a monolith, its latency directly affects the whole application. Every authorization call adds its own response time to the overall request, so the Go backend must enforce strict timeouts. The standard mechanism is context.Context, introduced in the standard library with Go 1.7. Each incoming HTTP request carries a cancelable context, and deadlines can be layered on top. The engineering team wired this properly through the service, ensuring that every MySQL query received the request context so slow queries could be canceled early.
The metrics told a different story. Despite the context propagation, responses regularly blew past the configured 1,000ms timeout by up to five times. Stack traces from production revealed the culprit: the context stopped at the connection pool.
Figure 3: In-service response times for an authzd deployment. The resolver’s timeout was set to 1000ms—you can tell from the helpful red line I’ve drawn in the graph, but definitely not from the many random spikes that climb all the way to 5000ms.
0 0x00000000007704cb in net.(*sysDialer).doDialTCP
at /usr/local/go/src/net/tcpsock_posix.go:64
1 0x000000000077041a in net.(*sysDialer).dialTCP
at /usr/local/go/src/net/tcpsock_posix.go:61
2 0x00000000007374de in net.(*sysDialer).dialSingle
at /usr/local/go/src/net/dial.go:571
3 0x0000000000736d03 in net.(*sysDialer).dialSerial
at /usr/local/go/src/net/dial.go:539
4 0x00000000007355ad in net.(*Dialer).DialContext
at /usr/local/go/src/net/dial.go:417
5 0x000000000073472c in net.(*Dialer).Dial
at /usr/local/go/src/net/dial.go:340
6 0x00000000008fe651 in github.com/github/authzd/vendor/github.com/go-sql-driver/mysql.MySQLDriver.Open
at /home/vmg/src/gopath/src/github.com/github/authzd/vendor/github.com/go-sql-driver/mysql/driver.go:77
7 0x000000000091f0ff in github.com/github/authzd/vendor/github.com/go-sql-driver/mysql.(*MySQLDriver).Open
at <autogenerated>:1
8 0x0000000000645c60 in database/sql.dsnConnector.Connect
at /usr/local/go/src/database/sql/sql.go:636
9 0x000000000065b10d in database/sql.(*dsnConnector).Connect
at <autogenerated>:1
10 0x000000000064968f in database/sql.(*DB).conn
at /usr/local/go/src/database/sql/sql.go:1176
11 0x000000000065313e in database/sql.(*Stmt).connStmt
at /usr/local/go/src/database/sql/sql.go:2409
12 0x0000000000653a44 in database/sql.(*Stmt).QueryContext
at /usr/local/go/src/database/sql/sql.go:2461
[...]
When QueryContext is called but no connection is available in the pool, database/sql must open a new one. That path calls driver.Driver.Open(), an interface method that does not accept a context.Context. Opening a MySQL connection is expensive: TCP handshake, SSL negotiation, MySQL authentication, and default option setup—at least six network round trips that completely ignored the request deadline.
A DSN-level timeout parameter only bounds the TCP dial, not the remaining connection steps, so requests still exceeded the global timeout. The deeper problem traced back to Go 1.8, when QueryContext/ExecContext were added. Those APIs made query cancellation possible, but driver.Open was left without a context parameter. Go 1.10 introduced the Connector interface as a separate implementation from the driver itself, requiring a substantial refactor to support both. The Go MySQL driver project had not yet adopted it.
That refactor shipped in PR #941. Once the driver implemented Connector, stack traces confirmed the context now flowed through connection creation:
0 0x000000000076facb in net.(*sysDialer).doDialTCP
at /usr/local/go/src/net/tcpsock_posix.go:64
1 0x000000000076fa1a in net.(*sysDialer).dialTCP
at /usr/local/go/src/net/tcpsock_posix.go:61
2 0x0000000000736ade in net.(*sysDialer).dialSingle
at /usr/local/go/src/net/dial.go:571
3 0x0000000000736303 in net.(*sysDialer).dialSerial
at /usr/local/go/src/net/dial.go:539
4 0x0000000000734bad in net.(*Dialer).DialContext
at /usr/local/go/src/net/dial.go:417
5 0x00000000008fdf3e in github.com/github/authzd/vendor/github.com/go-sql-driver/mysql.(*connector).Connect
at /home/vmg/src/gopath/src/github.com/github/authzd/vendor/github.com/go-sql-driver/mysql/connector.go:43
6 0x00000000006491ef in database/sql.(*DB).conn
at /usr/local/go/src/database/sql/sql.go:1176
7 0x0000000000652c9e in database/sql.(*Stmt).connStmt
at /usr/local/go/src/database/sql/sql.go:2409
8 0x00000000006535a4 in database/sql.(*Stmt).QueryContext
at /usr/local/go/src/database/sql/sql.go:2461
[...]
Response times dropped to match the configured budget. With a 90ms timeout, the 99th percentile looked less like a scatter plot and more like a clean cut-off:
Figure 4: Resolver response times with a timeout set at 90ms. You can tell that the Context is being respected because the 99th percentile looks more like a barcode than a graph
Applying the Fix
You don’t have to change how you initialize your database handle. Calling sql.Open in Go 1.10 or later still detects a Connector-aware driver and creates connections with context propagation. Your existing sql.(*DB) setup continues to work.
That said, migrating to sql.OpenDB with a mysql.NewConnector brings a practical advantage: you can configure connection settings directly from a mysql.Config struct instead of composing and parsing a DSN string.
Avoid hardcoding a ?timeout= (or the equivalent mysql.(Config).Timeout) value. A static dial timeout knows nothing about how much of your request budget has already been consumed. The correct pattern is to funnel every SQL operation through the QueryContext/ExecContext APIs so cancellation applies uniformly—whether the problem is a slow dial or a slow query. The request context is the single source of truth for how long any operation is allowed to take.
A Data Race Born From Context
The most insidious of the three bugs is a security issue hiding inside a long-standing optimization. At this point, the discussion of pooled connections matters because of a subtle behavior: calling (*DB).Query or (*DB).QueryContext does not just borrow a connection—it steals control of it. Unlike (*DB).Exec, a query can return rows, and those rows are streamed across a single, stateful connection. That is why Query returns a sql.(*Rows) handle, which wraps the connection; you have to read from that specific socket to gather the results. This is also why (*Rows).Close is mandatory: it is the handshake that returns the stolen connection to the pool.
rows, err := db.Query("SELECT a, b FROM some_table")
if err != nil {
return err
}
defer rows.Close()
for rows.Next() {
var a, b string
if err := rows.Scan(&a, &b); err != nil {
return err
}
...
}
Internally, the standard library wraps the driver’s own Rows interface with sql.(*Rows). A crucial optimization in the database/sql/driver contract is that the driver’s Rows.Next(dest []Value) error iterator can return pointers straight into its memory—often a connection buffer where bytes arrive directly from MySQL. User-side conversion into int, string, and so on happens later in sql.(*Rows).Scan. Essentially, the driver promises not to touch that buffer until the next call to Next, making the borrowed pointers safe for the brief window between iterations.
That implicit contract breaks with Go 1.8’s context-aware APIs. A context timeout generates a cancellation in the client; the MySQL server does not know the query was aborted and will keep streaming result packets. To safely return the connection to the pool, those lingering packets have to be drained by reading them into the connection buffer. If that draining is triggered just as the user’s code is inside a Scan call, the driver overwrites the memory behind the Values being converted—corrupting results with no immediate error. The scan itself only checks once for cancellation upon entry, and the subsequent Next call returns false; even strict error checking in the loop may never notice.
The pernicious part is that the only reliable signal is the return value of (*Rows).Close, which idiomatically few Go programs ever inspect. This was harmless had this bug not existed, but it is currently impossible to know how many production queries returned subtly altered data.
Double-Buffer Fix
Fixing this by cloning memory in driver.Rows.Next defeats the purpose of the entire interface, forcing double allocation per row. Another proposal to truncate the buffer at Close was also rejected, as it would allocate on every scan. The realistic solutions all carried an unacceptable performance penalty, which put the fix on hold for months before a workable pattern emerged.
The merged solution takes a page from pre-compositing graphics pipelines: double buffering. When driver.Rows is closed because a query is interrupted, the driver swaps its active connection buffer for an allocated back buffer. Draining of the rogue MySQL packets then happens into the back buffer, while the user scans data from the original front buffer. A subsequent query on the same connection keeps reading into background memory until its own rows are closed, at which point the buffers flip back. No extra memory is reserved on a normal connection unless a query is actually canceled early; the back buffer is allocated lazily on that event and reused thereafter. The change is both minimal at driver.Rows.Next and prevents any performance regression in the conventional, non-canceled path.
Figure 5: Double buffering in the Nintendo 64 graphics stack
As with the older race conditions in the library, detection without Close returns is near impossible. Fixing the driver does not undo the reader’s underlying logic, and the current release still requires application-level adjustment.
Production Advice
- Check the error from
(*Rows).Close. No other code path can reveal that the SQL query was aborted mid-scan after the final successfulNext. But keep thedeferstatement;Closeis idempotent, and it is the only guarantee of pool hygiene. - Avoid scanning into
sql.RawBytesclose to the wire. The tiny allocation bypass is not worth undefined reads if a context expires under a less tolerant driver. Inseparable[]bytetargets are the safe choice.
Shipping the fixes
Moving a new language into production service is rarely a smooth transition, especially at GitHub’s scale. MySQL traffic patterns are demanding enough that the Ruby client has been refined over many years, and the same level of attention is now being directed at the Go driver.
The patches described above have been merged upstream and are available in the v1.5.0 release of the go-sql-driver/mysql package. These fixes address the specific failure modes encountered during GitHub’s initial Go rollout, covering serialization edge cases, connection handling, and query cancellation behavior.
Work on the driver will continue as Go’s production footprint grows within GitHub’s infrastructure. The maintainers, @methane and @julienschmidt, collaborated on the review and integration of these contributions.
