The Shape of a Program
A useful test for a language or API design: write down the simplest syntactically valid expression of what you want to do. That expression should be a program.
When the language can’t express the idea directly, you end up repeating the same ten-to-twenty-line combination of symbols every time. This isn’t a critique of the underlying concept—it’s about how unwieldy its expression becomes in a particular domain. Things like Builders and Factories fall into this category.
Netty and the Factories That Bind
Consider Netty, the Java network server library. Each connection is a channel; bytes flow through a pipeline of handlers, each transforming messages. Some handlers, like ProtobufDecoder, are stateless and can be shared across channels. Others, like LengthFieldBasedFrameDecoder, track the state of a single read—they need a fresh instance per connection.
In a language with first-class functions, you’d write a function that returns a new pipeline with the right handlers. In Clojure, that looks like this:
(fn []
(doto (Channels/pipeline)
(.addLast "integer-header-decoder"
(LengthFieldBasedFrameDecoder. Integer/MAX_VALUE 0 4 0 4))
(.addLast "protobuf-decoder"
(ProtobufDecoder. (Proto$Msg/getDefaultInstance)))))
The doto macro rewrites the code at compile time, turning (doto obj (fn1 arg1) (fn2)) into a let binding with a unique symbol, calling each function on that symbol, and returning it. It removes the need to name the object repeatedly.
Java doesn’t have first-class functions. It has Callable, but only for zero-argument functions, which means you have to write a new class and explicitly capture any variables. Netty’s workaround is a factory interface for generating pipelines, so you write an implementation:
public class RiemannTcpChannelPipelineFactory implements ChannelPipelineFactory
public ChannelPipeline getPipeline() throws Exception {
ChannelPipeline p = Channels.Pipeline();
p.addLast("integer-header-decoder",
new LengthBasedFieldFrameDecoder(Integer/MAX_VALUE, 0, 4, 0, 4);
p.addLast("protobuf-decoder",
new ProtobufDecoder(Proto.Msg.getDefaultInstance()));
return p;
}
}
new RiemannTcpChannelPipelineFactory()
Even with anonymous classes, you’re still locked into the layout of ChannelPipelineFactory: its method name, its constructor, and the interface itself.
new ChannelPipelineFactory() {
public ChannelPipeline getPipeline throws Exception {
In Clojure, reify creates a dynamic class instance that can implement any interface and close over local variables. If you want to share a single protobuf decoder across pipelines, the expression becomes:
(let [pb (ProtobufDecoder. (Proto$Msg/getDefaultInstance))]
(reify ChannelPipelineFactory
(getPipeline [this]
(doto (Channels/pipeline)
(.addLast "integer-header-decoder"
(LengthFieldBasedFrameDecoder. Integer/MAX_VALUE 0 4 0 4))
(.addLast "protobuf-decoder" pb)))))
The Java equivalent requires a field and pipes for mutation:
public class RiemannTcpChannelPipelineFactory {
final ProtobufDecoder pb = new ProtobufDecoder(Proto.Msg.getDefaultInstance());
...
Note what’s absent in the Clojure version: no name for the factory, no file, no source-control entry. The factory is a meaningless object—its only role is to be a partially applied function. It disappears into the library. Factories are just awkward ways to express partial functions.
Making the Pipeline Itself Extensible
Riemann, Aphyr’s monitoring system, needs more than one pipeline. With distinct channels sharing some handlers and not others, you’d have to write nearly identical classes for each combination. Instead, the macro channel-pipeline-factory turns a list of handler names and constructor forms into a factory function:
(defmacro channel-pipeline-factory
"Constructs an instance of a Netty ChannelPipelineFactory from a list of
names and expressions which evaluate to handlers. Names with metadata
:shared are evaluated once and re-used in every invocation of
getPipeline(), other handlers will be evaluated each time.
(channel-pipeline-factory
frame-decoder (make-an-int32-frame-decoder)
^:shared protobuf-decoder (ProtobufDecoder. (Proto$Msg/getDefaultInstance))
^:shared msg-decoder msg-decoder)"
[& names-and-exprs]
(assert (even? (count names-and-exprs)))
(let [handlers (partition 2 names-and-exprs)
shared (filter (comp :shared meta first) handlers)
forms (map (fn [[h-name h-expr] ]
`(.addLast ~(str h-name)
~(if (:shared (meta h-name))
h-name
h-expr)))
handlers)]
`(let [~@(apply concat shared)]
(reify ChannelPipelineFactory
(getPipeline [this]
(doto (org.jboss.netty.channel.Channels/pipeline)
~@forms))))))
This is a macro, meaning it runs at compile time on the source code itself. Because Clojure is homoiconic, manipulating code is just data manipulation—macros let you define new syntax with the same tools you use for data.
The macro begins with documentation for the REPL. Metadata flags handle sharing: ^:shared marks a handler that can be reused across channels. Handlers without that flag get instantiated fresh with each pipeline.
[& names-and-exprs]
The macro takes arguments as a flat list of name/handler pairs:
(assert (even? (count names-and-exprs)))
A compile-time check validates that the list has an even number of elements:
(let [handlers (partition 2 names-and-exprs)
shared (filter (comp :shared meta first) handlers)
Partitioning the list into pairs makes processing easier. The macro then finds which handlers are marked :shared by composing three functions: take the name, extract its metadata, and look up the flag:
(let [handlers (partition 2 names-and-exprs)
shared (filter (comp :shared meta first) handlers)
forms (map (fn [[h-name h-expr] ]
`(.addLast ~(str h-name)
~(if (:shared (meta h-name))
h-name
h-expr)))
handlers)]
Each pair is turned into code: shared handlers become variable references, others are expanded inline. The shared handlers are bound to local variables at the top of the generated pipeline factory:
`(let [~@(apply concat shared)]
Backtick and ~@ are the macro author’s tools: backtick constructs the code template without evaluating, and ~@ splices the evaluated forms into place.
(reify ChannelPipelineFactory
(getPipeline [this]
(doto (org.jboss.netty.channel.Channels/pipeline)
~@forms))))))
What was bulky boilerplate—channel class names, method calls, variable naming, manual construction of non-shared decoders—collapses into a compact, symmetric form:
(channel-pipeline-factory
integer-header-decoder (LengthFieldBasedFrameDecoder. Integer/MAX_VALUE 0 4 0 4)
^:shared protobuf-decoder (ProtobufDecoder. (Proto$Msg/getDefaultInstance))
The .addLast is gone, as are the intermediate symbols. The protobuf handler is reused globally; the length decoder is built fresh per connection. The syntax is simple because the macro took the complexity to compile time.
When to Reach for a Macro
Riemann’s pipeline code didn’t start with a macro. Earlier versions used plain functions across three namespaces, with pipelines split into multiple layers of indirect calls as variants proliferated. The structure became difficult to parse when tracking down performance issues, and clarifying the pipelines themselves required a syntax that showed their shape directly.
(channel-pipeline-factory
int32-frame-decoder (int32-frame-decoder)
^:shared int32-frame-encoder (int32-frame-encoder)
^:shared executor shared-execution-handler
^:shared protobuf-decoder (protobuf-decoder)
^:shared protobuf-encoder (protobuf-encoder)
^:shared msg-decoder (msg-decoder)
^:shared msg-encoder (msg-encoder)
^:shared handler (gen-tcp-handler
core
channel-group
tcp-handler))
This macro-based pipeline form makes the relationship between handlers obvious and simplifies edits. But the same semantics can be expressed without a macro, using an ordinary function that accepts other functions — for instance, a constructor over a list where handler names are given as keywords:
(channel-pipeline-factory
:unshared :int32-frame-decoder #(int32-frame-decoder)
:shared :int32-frame-encoder (int32-frame-encoder)
:shared :executor shared-execution-handler
:shared :protobuf-decoder (protobuf-decoder)
:shared :protobuf-encoder (protobuf-encoder)
:shared :msg-decoder (msg-decoder)
:shared :msg-encoder (msg-encoder)
:shared :handler (gen-tcp-handler
core
channel-group
tcp-handler))
The function-based variant is arguably just as clear, but it has a different set of costs. A macro expands at compile time, so expensive work happens once during compilation and generates runtime code that is quick to execute. In the function version, each invocation must iterate over the handler forms, recognize shared versus unshared entries, and fire off helper functions for unshared handlers. If that path is performance-critical, the runtime iteration and calls may not be shaped in a way the JIT can optimize.
Macros also shine for repeated expression patterns, which is why many libraries use them to offer tiny domain-specific languages. Riemann’s own query language relies on macros to cut filtering boilerplate down to a terse syntax. That expressiveness costs something, though: macros break the usual rule that a variable can be replaced with its value, making it harder to reason about when code actually evaluates. They tend to work best for end-users rather than as a backend implementation detail, and function equivalents should be offered whenever possible.
Because macros violate substitution and bend evaluation order, they also defeat runtime composition. They operate on unevaluated expressions, not runtime values—so you can’t bind a macro form to a variable or pass it as data. (map future ['(+ 1 2) (+ 3 4)]) throws a CompilerException because the compiler can’t take the value of a macro. This leads to macro contagion: any enclosing code that invokes a macro with a non-literal argument must itself be a macro. Dynamic evaluation freedom is the price of that power.
For Riemann specifically, the performance gains from (channel-pipeline-factory) justified losing flexibility, but the tradeoff is not one to make casually. When in doubt, prefer functions.
Control Flow as Syntax
Many things built into other compilers become ordinary user-level code in Clojure. JavaScript expresses futures with explicit callbacks:
var a = 1;
var f = future(function() { return a + 2; });
f.await(); // returns 3
A Clojure equivalent might be written with an anonymous function itself:
(let [a 1
f (future (fn [] (+ a 2)))] ; Or alternatively, #(+ a 2)
(deref f)) ; returns 3
But the macro eliminates the anonymous function entirely, as in Clojure’s built-in future:
(let [a 1
f (future (+ a 2))]
(deref f)) ; returns 3
The standard library relies heavily on macros for control flow: short-circuiting and/or, conditionals like cond and condp, and the concurrency primitives dosync, future, delay, and lazy-seq. Java’s synchronize { … } block has an analogue in the locking macro. Try/catch semantics can be re-implemented by a library macro, as the Slingshot library does. In Clojure, language features that other languages hard-code into the compiler are accessible to anyone.
Guidelines
Macros condense complex ideas, but just as language designers must balance new syntax against interaction complexity, the same judgment applies at the library level. The general rules:
- Keep macros as simple and as predictable as possible.
- Use them for purely syntactic transformations, such as control flow.
- Reach for a macro when writing efficient—but awkward—code that the runtime will not optimize.
- Prefer plain functions in other cases.
