Pipelining against the network round trip
Performance testing Riemann has a bottleneck: the Java client. The existing RiemannTcpClient and RiemannRetryingTcpClient were thread-safe but effectively serialized all traffic behind a mutex. In write-heavy tests, that single lock turned the client into a bottleneck no matter how much headroom the server had.
Riemann's protocol is simple and synchronous—one request message on a TCP connection gets exactly one response message. The old clients forced every caller to wait the full network round trip plus server processing time before issuing the next request. With enough threads, that serialization dominates throughput.
Two ways to get more writes through
One option is bundling: the API allows multiple events in a single Message, but that only helps if your calling code can naturally batch events. Where higher latency is acceptable, the client can instead hold a buffer of outbound events and flush it periodically.
The more general fix is request pipelining. A new RiemannThreadedClient sends requests before earlier responses arrive, then matches each response back to its originating request. All calling threads push messages into a lock-free queue and receive Promise objects that are fulfilled when the response lands. The synchronous API remains available; multiple threads can pipeline their requests concurrently underneath it.
Internally, a writer thread drains the write queue and sends messages over the socket, pushing each onto an in-flight queue. A separate reader thread pulls responses off the socket and resolves the corresponding promises. Both queues are bounded, providing backpressure that caps how many requests can be outstanding. That same bound limits how many events can be lost if the connection fails.
What changes on loopback
The charts below show the same write test against a Riemann instance on loopback, first with the naive round-trip client:
Then with the pipelined RiemannThreadedClient:
No tuning or optimization has been applied to the algorithm, and error handling is still rough. Even so, the pipelined client shows roughly double the throughput in exchange for roughly double the per-event latency. The gains should be larger on real networks where round-trip time dominates.
