We’re All Just Looking for Connection

When TCP Half-Closes Bit Us in the Backend

Slack’s canvas feature and the stand-alone Quip product are both powered by the same Python backend. In July, that backend started throwing a strange cluster of EOFError exceptions during SQL queries. The errors were spread across multiple services and database hosts, with no obvious common factor.

Chasing a Phantom Connection Close

The stack trace showed an asyncio.IncompleteReadError (which we translate to EOFError) when reading a response from the database. Our connection-handling code hadn’t changed recently, so we started with the usual suspects.

File "core/mysql.py", line 299, in __read_result_set
    header = await self.__read_packet(timeout=timeout)
File "core/mysql.py", line 532, in __read_packet
    header = await self.conn.read_exactly(4, timeout=timeout)
File "core/runtime_asyncio.py", line 1125, in read_exactly
    raise EOFError() from None

The relevant code path looked something like this:

async def _perform_query_locked(...) -> core.sql.Result:
    ...
    if not self.conn.is_connected():
        await self.__connect(...)
        await self.__send_command(...)
        result_set = await self.__read_result_set(...) # <-- EOFError

We have a few explicit places where we close the database connection, such as after certain errors. One early theory was that another Python coroutine was closing the connection after we issued a query but before reading the response. We ruled that out quickly, because connection access is protected by an in-memory lock acquired at a higher level. If that were impossible, the other side of the connection must have closed on us.

That theory gained weight when we found that our database proxies had closed a large batch of connections at the exact time of the incident. The close events were distributed independently of database host, matching the errors’ spread. Our metric for client-initiated closes showed no increase from us at that time. The timing was suspicious, but it still didn’t explain the high error rate, since we had just checked the connection state before issuing queries.

Two Broken State Checks

Something was off, so we looked closely at the guard clause if not self.conn.is_connected() and its supporting code in our AsyncioConnection implementation.

class AsyncioConnection(core.runtime.Connection[memoryview]):
    def closed(self) -> bool:
        return not self.writer or self.writer.is_closing()

    def is_connected(self) -> bool:
        return self.connected

Both functions turned out to be wrong. First, closed() only checked the state of the connection’s StreamWriter, which is controlled by the application server. It ignored the StreamReader, which is the part that knows when the other side has closed the connection. A unit test confirmed this:

async def test_reader_at_eof_vs_writer_is_closing(self):
    conn = await self.create_connection()

    # Ask the unit test's server to close
    await conn.write(self.encode_command("/quit"))

    # Don't read. Still open since we haven't seen the
    # response yet
    self.assertFalse(conn.writer.is_closing())

    # Read, then sleep(0) since it requires another run
    # through the scheduler loop so the stream can detect
    # the zero read/eof
    response = await conn.read_until(self.io_suffix)
    await core.runtime.sleep(0)

    self.assertTrue(conn.reader.at_eof()) # passes 
    self.assertTrue(conn.writer.is_closing()) # fails

Second, is_connected() had two problems. It wasn’t derived from closed(), so the two could drift apart. And it could return a false positive: the instance variable self.connected was only set to false when the application server initiated the close, so it too was blind to the reader being in the EOF state.

To measure how badly these checks drifted, we logged a metric comparing self.connected against not self.closed(). That investigation turned up six additional bugs:

1. is_connected() could also return a false negative. These false negatives only occurred on services that maintain websocket connections to clients, because self.connected is only set to true when the application server initiates a connection — never when a client initiates one.
2. A check we run when releasing the connection lock — to detect unread data on the connection — was wrong for websocket connections, where clients are expected to send data at any time.
3. Our HTTP client pool could contain closed clients.
4. Exceptions during reconnect weren’t handled correctly.
5. There was another spot where we should have been removing connections from the pool but weren’t.
6. Redis-specific cleanup that should run on every connection close was skipped whenever the close wasn’t initiated by the application server.

Putting the Pieces Together

With all the bugs on the table, the original incident made sense:

1. The database proxy closed connections on us. AWS had introduced behavior where the proxy closes connections after 24 hours, even if they are not idle. The timing matched our daily release, which restarts all servers.
2. Because the closes weren’t initiated by us, we failed to notice, and we didn’t reconnect before issuing SQL queries.
3. We would issue a query, try to read the response, and find the reader already in the EOF state, raising EOFError.

After deploying the fixes, the EOFError spike during SQL queries almost entirely disappeared:

And once the fix for client-initiated connections rolled out, the false negatives vanished as well:

The Real Payoff: Unblocking asyncio

The fixes for these connection-state bugs had an impact far beyond the original EOFError issue. They unblocked a long-stalled migration to asyncio.

That migration started in 2020. We had been running on a custom runtime from our early days adopting Python 3. The move to asyncio — the standard library IO framework introduced in Python 3.4 — went smoothly everywhere except one service on our largest single-tenant cluster. That customer sporadically hit timeouts when exporting spreadsheets to PDF, always at peak load, and only rarely. Reproducing it was hard, so the migration stalled.

The PDF export handler sends an RPC request over a pooled connection to a separate PDF-generation service. If that service is overloaded, it may close the connection — and our code wouldn’t notice. The handler would send the request into a closed connection, the write would hang because the buffer never drained, and the request would time out. (For small exports, the write would likely raise EOFError immediately instead, since the buffer wouldn’t fill completely.)

Once our connection-state fixes were live, we re-enabled asyncio on that last service and confirmed the errors did not return. The asyncio project resumed and finished shortly after, capping it off with a satisfying round of deleting custom runtime scheduler code.

TCP Half-Closes Are Easy to Miss

The root of all this trouble is that a TCP socket can be in a half-shutdown state. When the other side sends FIN, you enter CLOSE WAIT — you can still attempt reads indefinitely, and the operating system won’t move you out of that state until you close or shut down the socket yourself.

We had expected StreamWriter.is_closing() to cover that half-shutdown state, but its documentation is more subtle: “Return True if the stream is closed or in the process of being closed.” In practice, both the default asyncio event loop and uvloop behave exactly the way our code did — the closing state is only set for client-initiated closes, not for closes from the remote side. That’s why our unit test failed with and without uvloop, and why trusting StreamWriter.is_closing() alone was not enough.

For us, the takeaway is to keep digging when something doesn’t quite add up. Patching this by adding EOFError to the list of exceptions that trigger a failover query would have papered over the symptom. But the real bugs — six of them, lurking under that first rock — were affecting connection handling far beyond the original incident, and fixing them is what finally let us complete the asyncio migration.