A broader, faster Riemann

Riemann 0.2.0 is now available, and it represents one of the most significant leaps forward for the event-stream processor since its inception. The release is the cumulative result of six months of full-time development, funded through a mix of individual donations and commercial contracts with Blue Mountain Capital, SevenScale, and Iovation. Beyond the financial support, feedback from production users directly shaped the roadmap for this version.

Development has also been a collaborative effort. Contributions from a broad cohort of open-source developers are acknowledged in the release thanks, with particular mentions for Pierre-Yves Ritschard, lwf, Ben Black, Thomas Omans, and Dave Cottlehuber. The surrounding ecosystem—dashboards, clients, and integration packages—also received attention during this cycle.

Now that the release is out, the core maintainer is starting a new position at Factual, which will slow the pace of development. Still, work on the next release will continue.

What’s new

This version introduces several capabilities that change how events can be modeled and processed:

  • Arbitrary key-value string pairs on events, a building block for richer metadata.
  • Hot configuration reloading so core services such as pubsub and the index survive config changes without restart.
  • An integrated nREPL server for live introspection and manipulation.
  • streams/sdo, streams/split, and streams/splitp for composite and conditional stream dispatch.
  • streams/top for streaming top-k and streams/tag for attaching tags to events.
  • config/delete-from-index to explicitly evict events from the index.
  • RPM packaging with init scripts, proper log directories, and dedicated users—meaning /etc/init.d/riemann reload now works for Debian and RPM installs alike.
  • New folds: difference, product, and quotient, each available in sloppy and strict variants designed to do the right thing in most contexts.
  • Executor services for asynchronous queued event processing.
  • streams/exception-stream to capture exceptions and translate them into events.

Tuning and cleanup

The throughput story improved substantially. TCP and UDP server performance received major speedups, with netty updated to 3.6.1 and a streamlined execution-handler layout behind the gain. The server now uses fewer threadpools, and a threadpool leak from netty was fixed. The graphite client is also faster and more complete.

A range of ergonomic improvements accompanied the performance work:

  • Configuration syntax errors are detected early, with messages that include file names and line numbers.
  • Config files can include other files by relative path, and ship with emacs modelines.
  • Server logging has been cleaned up; closed-channel exceptions are silenced.
  • streams/rate now operates in real time, eliminating the need for fill-ins.
  • streams/where reports whether it matched, making it reusable as a predicate, as do tagged-any and tagged-all.
  • streams/counter can be reset to a specific metric and supports expiry.
  • core/reaper takes a :keep-keys option for field preservation.
  • riemann.email/mailer accepts custom :subject and :body functions.
  • riemann.config now bundles common time and scheduling helpers.

The dedicated website at riemann.io launched alongside expanded documentation and examples, and the "hyperspace core online" message is back.

Bugfixes of note

Several fixes target correctness rather than just efficiency:

  • The part-time-fast undercounting bug under contention is resolved.
  • Exceptions during expired-event processing are caught.
  • Metric name escaping for librato was corrected, and the mailer can reach SMTP relays again.
  • graphite-path-percentiles now converts decimals of three or more places to percentile strings.
  • streams/rollup is much more efficient, doesn’t leak tasks, and now aggregates and forwards expired events instead of stalling.
  • A streams/coalesce bug involving lazy transient persistence and an issue with streams/ddt edge cases were both fixed.

Internal restructuring

Under the hood, 0.2.0 cleans up a number of structural details. Cores are now immutable, and a service protocol provides lifecycle management for components. Netty server transports are pluggable, and composing pipeline factories is far simpler. The old riemann.periodic has been replaced by riemann.time, and riemann.common/deprecated powers migration notices for the few streams affected by API changes. Most functions continue to work as before but log a deprecation warning. The project also moved to Clojure 1.5, and many anonymous functions got names to make profiling easier.

Known limitations

A handful of issues remain known entering this release:

  • Passing :host to websocket-server has no effect—it binds to * regardless.
  • folds/mean throws on empty lists.
  • graphite-server lacks test coverage.
  • Riemann can happily overload browsers via websockets.
  • streams/rate doesn’t stop its internal poller cleanly when self-expiring.
  • When netty runs out of filehandles, it will hang new connections.