Why We Moved Pod-to-Pod Communication to gRPC

Cloudflare's migration from bare metal and Mesos Marathon to Kubernetes let us break a monolithic DNS application into a set of microservices with precise control over how they talk. Kubernetes supplies the plumbing — ReplicaSets keep the right number of pods alive, Services give a stable load-balanced endpoint, Ingress exposes them outward, and NetworkPolicies restrict traffic at L3/L4. What Kubernetes doesn't answer is which application-level protocol those pods should use when they talk to each other.

The DNS team's first Kubernetes services communicated over REST APIs, with Kafka in the mix to absorb traffic spikes. A typical flow looked like this:

When the DNS team first moved to Kubernetes, all of our pod-to-pod communication was done through REST APIs and in many cases also included Kafka.

That setup, e.g., for Secondary DNS zone transfers, worked, but scaling exposed real limitations. Large DNS zones sent over HTTP hit sizing and compression constraints. And because we controlled both ends of the connection, we could pick something better than HTTP/1.1 with JSON.

gRPC Improves on REST in Everyday Use

HTTP client libraries force you to define paths, handle parameters, and parse responses from bytes. gRPC hides that: a network call looks like an ordinary function call on a struct. The contract lives in a protobuf schema, and the protoc command generates code for many languages.

A schema defines strongly typed messages and a service with RPC functions. A simple example declares a Hello message, a HelloResponse, and a service HelloWorldHandler with a SayHello RPC:

message Hello{
   string Name = 1;
}

message HelloResponse{}

service HelloWorldHandler {
   rpc SayHello(Hello) returns (HelloResponse){}
}

Server-side, you implement that schema. A struct must define every RPC in the service — here, a function SayHello that receives a context and a *pb.Hello and returns a *pb.HelloResponse. In main, you create a TCP listener, instantiate a gRPC server, and register your implementation:

type Server struct{}

func (s *Server) SayHello(ctx context.Context, in *pb.Hello) (*pb.HelloResponse, error) {
    fmt.Println("%s says hello\n", in.Name)
    return &pb.HelloResponse{}, nil
}

func main() {
    lis, err := net.Listen("tcp", ":8080")
    if err != nil {
        panic(err)
    }
    gRPCServer := gRPC.NewServer()
    handler := Server{}
    pb.RegisterHelloWorldHandlerServer(gRPCServer, &handler)
    if err := gRPCServer.Serve(lis); err != nil {
        panic(err)
    }
}

The client is equally simple: open a TCP connection, create a pb.HandlerClient, and call SayHello directly with a *pb.Hello object:

conn, err := gRPC.Dial("127.0.0.1:8080", gRPC.WithInsecure())
if err != nil {
    panic(err)
}
client := pb.NewHelloWorldHandlerClient(conn)
client.SayHello(context.Background(), &pb.Hello{Name: "alex"})

That type agreement makes cross-language calls feel like local function calls. And gRPC supports more than simple unary requests. Four service method types are available:

  • Unary: one request, one response — like a normal function call.
  • Server Streaming: the server replies with a stream of messages.
  • Client Streaming: the client streams messages, then the server replies once.
  • Bi-directional Streaming: both sides send streams asynchronously.

Protobuf and HTTP/2 Deliver the Performance Gains

Our old setup used HTTP/1.1, where pipelining queues requests on a connection. HTTP/2, the transport gRPC is built on, uses multiplexing: many requests share a connection and proceed concurrently. gRPC also replaces JSON with protobuf, which has a schema agreed upon at registration and serializes measurably faster.

Protobuf performs better in small, medium, and large data sizes.

Benchmarks on a laptop show protobuf winning at small, medium, and large payload sizes — faster per operation and smaller after marshalling. The gap widens with scale: unmarshalling a large data set takes protobuf 96.4ns/op versus JSON's 22647ns/op, roughly a 235X reduction. For our large DNS zones, that difference directly shortens the time from a record change to serving it at the edge.

From the application's point of view, HTTP/2 plus protobuf produced no dramatic shift in throughput — our pods sit close together, so connection times were already low, and most gRPC calls involve small payloads. But we did see a notable drop in latency spikes when writing new, edited, and deleted records to the edge, likely a side effect of HTTP/2's multiplexing.

Our latency spikes dropped in both amplitude and frequency.

Layering Security Beyond NetworkPolicies

Kubernetes NetworkPolicies protect communication at the network layer. For instance, a policy like this:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-network-policy
  namespace: default
spec:
  podSelector:
    matchLabels:
      role: db
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - ipBlock:
        cidr: 172.17.0.0/16
        except:
        - 172.17.1.0/24
    - namespaceSelector:
        matchLabels:
          project: myproject
    - podSelector:
        matchLabels:
          role: frontend
    ports:
    - protocol: TCP
      port: 6379
  egress:
  - to:
    - ipBlock:
        cidr: 10.0.0.0/24
    ports:
    - protocol: TCP
      port: 5978

restricts ingress and egress to pods labeled role=db, allowing only frontend pods, pods with project=myproject, and specific source IP ranges in, plus one destination subnet out. NetworkPolicies do not discriminate between endpoints within a pod, so they can't enforce rules per API function.

gRPC's per RPC credentials fill that gap. Adding interceptors to stream and unary handlers lets you authenticate each call before it reaches its function:

func (s *Server) UnaryAuthInterceptor(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
    // Get the targeted function
    functionInfo := strings.Split(info.FullMethod, "/")
    function := functionInfo[len(functionInfo)-1]
    md, _ := metadata.FromIncomingContext(ctx)

    // Authenticate
    err := authenticateClient(md.Get("username")[0], md.Get("password")[0], function)
    // Blocked
    if err != nil {
        return nil, err
    }
    // Verified
    return handler(ctx, req)
}

Here, the username, password, and requested function are extracted from the interceptor's info object and checked for authorization — one implementation guards every RPC. The client side verifies the server's certificate first, then attaches its credentials when dialing:

transportCreds, err := credentials.NewClientTLSFromFile(certFile, "")
if err != nil {
    return nil, err
}
perRPCCreds := Creds{Password: grpcPassword, User: user}
conn, err := grpc.Dial(endpoint, grpc.WithTransportCredentials(transportCreds), grpc.WithPerRPCCredentials(perRPCCreds))
if err != nil {
    return nil, err
}
client:= pb.NewRecordHandlerClient(conn)
// Can now start using the client

Where We're Going

The next phase is consolidating all DNS-related code into a single API exposed through one gRPC interface, so applications no longer need individual database access. That removes duplication, simplifies schema evolution, and gives control at the function level rather than the table level.

The migration has gone well, but REST isn't gone entirely yet. We're also watching for HTTP/3 support in gRPC to take advantage of QUIC.