Why GitHub started deferring telemetry

Flamegraph analysis across GitHub’s most critical workflows revealed a recurring bottleneck: a substantial chunk of request time was spent sending statsd metrics. In some cases, telemetry accounted for as much as 65ms per request. While that data is essential for improving the platform, it doesn’t directly benefit the user — and when it lands in the critical path, it actively hurts page load performance.

The obvious fix was to stop sending telemetry immediately and instead batch it. Background jobs were ruled out due to the sheer volume of data, which led the team to experiment with Rack::Events. That approach failed for two reasons: callbacks in Rack::Events block the response from closing, so users would see the browser’s loading indicator until deferred work finished, and response-time metrics still included the deferred code’s execution time.

The rack.after_reply detour

Further digging turned up rack.after_reply, a Rack extension implemented by Puma. As described in the Puma source, applications can write callables to this array and the server invokes them once the request is done. That matched the requirement precisely: collect telemetry during the request, flush it in one batch after the response reaches the browser, and keep network calls out of the response-time equation.

There was one obstacle — GitHub runs Unicorn, not Puma. The team contributed rack.after_reply back to Unicorn, then built a thin wrapper to make it easier to consume in the application:


class AfterResponse

  def initialize(env)

    env[“rack.after_reply”] ||= []

    env[“rack.after_reply”] << -> do

      self.call

    end

    @to_perform = []

  end

  # Calls each callable defined via #perform

  def call

    @to_perform.each do |block|

      begin

        block.call(self)

      rescue Object => e

        Rails.logger.error(e)

      end

    end

  end

  # Adds given block to the array of callables that will

  # be called after the user has received the response.

  def perform(name, &block)

    @to_perform << block

  end

end

With the server-side support in place, a wrapper around the telemetry class buffers all stats data, and a middleware orchestrates the flush:


GitHub.after_response.perform do

  GitHub.statsd.flush!

end

Trade-offs and results

The main drawback is that a Unicorn worker executing rack.after_reply callables cannot serve another HTTP request until they finish. If that deferred work grows unchecked, HTTP queueing — time spent waiting for a worker — could increase. GitHub mitigated the risk by adding timeout behavior to cap how long any callable may run.

The payoff is significant: p50 response times dropped by 30ms across all pages, and p99 times improved by more than 50ms site-wide. Beyond the immediate win, the change adds a capability to Unicorn that has uses well outside telemetry, and there is early movement toward making rack.after_reply an official part of the Rack spec. A recently released gem called maybe_later builds additional functionality on top of this feature worth exploring.