Finding the failure domains
HelloWorks, a HelloSign product, converts PDF forms into an intelligent, mobile-friendly format. Because the product handles data collection that is critical to both requesters and filers, end-to-end reliability is a prime requirement. After months of instability traced to the application's core, the engineering team reorganized code into three tiers: an error kernel, core features, and extended features.
The error kernel is the smallest possible part of the system that should never fail. Keeping it minimal reduces surface area and makes catastrophic failures easier to reason about and triage. The principle, described in Reactive Design Patterns, is to keep important state and functionality near the root of a supervision hierarchy and delegate risky operations outward. Originating in Erlang's actor model, the idea still applies to monolithic applications. By identifying failure domains—the ways a method, module, service, or function can fail—an error kernel clarifies which errors are recoverable and which contain the logic for what happens next. It also prioritizes hardening efforts: failures inside critical infrastructure need isolation and attention first.
HelloWorks runs two independent error kernels for its two core services: one in the portal, another in the end-user application. Both handle initialization and error handling, since without those nothing else can proceed. Architectural choices vary—the actor model suits servers, while a monolithic design fits clients.
Tiering beyond the kernel
Regular development rarely touches the error kernel because it is intended to stay small. The bulk of work sits in core features: areas where the system can recover from issues, albeit with significant effort. Data corruption and inconsistencies are painful but usually recoverable. Core business logic lives here, not in the error kernel. The rationale is stability—infrastructure must remain available regardless of whether business logic is correct. Building and running workflows are examples of core business logic in HelloWorks. Everything else falls into extended features: user-facing conveniences that may irritate when broken but are fixable with minimal risk.
Tiering features gives engineering, product, and design a common language. It specifies a feature's complexity and its fundamental role. The answer to whether a feature can be hacked together quickly or requires deep deliberation comes almost immediately from its tier. Extended features can ship fast without lasting impact, while extending core mechanisms such as launch methods demands higher-level testing before release.
Testing with properties
Rigorous testing of the error kernel and core features is needed to build new features reliably. Formal verification is too rigid for most systems, but property-based testing offers a middle ground that validates behavior without tying implementation down tightly. The approach turns test writing into automating the generation of test cases, shifting the mental model from passing individual tests to holding true properties. It defines and checks invariants, and shrinks complex failures down to their simplest form for easier diagnosis.
A property test comprises two parts: a generator, which describes the input data, and a property, which is a generic outcome that always holds for generated inputs. For example, demonstrates this simple logic. In HelloWorks, every successfully saved database object is retrievable by UUID. The properties tested include:
- Only one object is retrieved on a successful save.
- The object remains unchanged after being saved on retrievals.
- Every object of type X can be saved.
- No database row exists when retrieving a UUID that does not exist, and a UUID should not exist when saving fails.
Generators differ from fuzzing because they define a problem-space, constraining randomly generated inputs to a meaningful range rather than supplying arbitrary data.
Property test patterns that catch real bugs
Property testing’s main advantage is surfacing defects before users do. Many of those defects are non-trivial edge cases, though trivial ones show up too. A handful of easy-to-implement property tests on code paths centered around regular expressions can deliver outsized value. Build a generator that produces the expected string input, then verify that the regex either behaves as intended or rejects everything that should fail validation. When a regex filters input destined for another function, a single property test can exercise both the validator and the downstream function at once:
fn send_email(msg, addr) ->
if valid_email_addr(addr) {
really_send_email(msg, addr)
}
end
property status_code(200) == send_email(msg, generated_valid_email_addr)Assuming stable networking in this example, two error classes can arise from the addr variable: the validator fails silently so no email is sent, or really_send_email receives a bad address that slipped through. One property test catches both, making three separate unit tests largely redundant.
Modeling
The modeling pattern builds a property test that re-implements the system under test in a simplified way that makes outcomes obvious. Stripping out optimizations, logging, and other incidental code leaves a model that is easy to reason about. Once the model exists, it becomes a true equivalency baseline for any future production changes, allowing engineers to refactor internals for extension or performance without worrying about behavioral drift.
Symmetric properties
Models are not always practical. Symmetric properties cover systems that translate data between representations: serialization, marshaling, encoder/decoder pairs. These amount to identity proofs—serialize then deserialize, or encode then decode, and the result must equal the input. Property testing matters here because generated data explores the full problem space, not just a handful of hand-picked examples known to work. The property reduces to:
fn encode(x) -> ...
fn decode(y) -> ...
property input == encode(decode(input))
property input2 == decode(encode(input2))Stateful properties
The techniques above are stateless, but they extend to stateful systems, albeit with more effort. Many systems transition between states, and stateful properties can traverse every allowed transition, not just the fixed scenarios example-based tests repeat. Because intermediary transitions in some systems can loop infinitely, a model that exercises the transition graph helps engineers understand complexity and catch errors before the test suite even finishes running. Each run traverses new paths, exposing inconsistent behavior that emerges from complex interactions between data manipulations across state changes.
Pitfalls of bundling properties
An early mistake is bundling many properties under a single generator. The reasoning seems sound—one master generator can feed all properties for an entity, reducing boilerplate, and in some cases tests might run faster because generated inputs are reused across properties. But in practice the downsides dominate:
- Generator code becomes unnecessarily complicated
- A failing property is hard to diagnose because the exact input that triggered the failure is unclear
- Tests run slower, since more bundled generators means larger generated inputs
- Bundling reduces diversity in the generated values
In the email example, adding a property for the msg variable alongside addr would make it nearly impossible to know whether the failure came from an invalid address or invalid message. Isolating individual properties and generators keeps tests readable, writable, and debuggable.
msg = generate_msg()
addr = generate_addr()
# if this fails then who is responsible?
property status_code(200) == send_email(msg, addr)
# better to separate properties like this so failures are clear
property status_code(200) == send_email("valid message", addr)
property status_code(200) == send_email(msg, "[email protected]")
# targeted unit tests are clear and quick but writting many becomes burdensome
test status_code(200) == send_email("\"; drop table students;", "[email protected]")
test status_code(200) == send_email("message", "[email protected]")There is another cost to heavy unit testing: when requirements shift—say a status code changes from 200 to 300—updating a large set of unit test cases is tedious, while changing a couple of property tests that generate thousands of cases is far less painful. Clear generator targets and well-scoped properties pay off over time.
Where HelloSign applies property testing
Adoption is a continuous process. The forms in HelloWorks embed logic that is not obvious when building them. Both the building and publishing phases currently rely on modeling to surface inconsistencies and detect errors. As the system shifts from validation-based logic toward a parsed-based language, property tests verify that new features extending what forms can do, and how they run, do not regress existing behavior. DSLs that translate into protocols consumed by other services are the primary target for symmetric properties. Work is also underway on a job executor using stateful property models, which is already yielding results despite being early stage.
PDF mapping logic is another area of focus. Basic property tests already exist there, but the goal is to make mappings robust enough to catch issues before a customer publishes a form, rather than when end users fill out documents. Property testing can solve this problem, though occasionally another approach may be equally robust and faster to implement.
Measured impact so far
Property-based testing has reshaped how the team builds reliable software. Notable outcomes include:
- Database adapter bugs surfaced when swapping out adapters
- Unintentional behavior changes were caught when adding form editor features
- Five JSON Schema inconsistencies were found while writing tests—before any were run
- Storage issues with specific string types were caught at the database level
The main challenge is onboarding. Property testing has a steep learning curve, and without intuition about where it adds value, engineers struggle to know where to start. But the net result is strongly positive: even team members who cannot write property tests can read the failures and understand the cause. Property tests offer true equivalency testing that keeps paying dividends—the more complex tests shipped roughly eight months ago and have not needed modification or correction in six. Unit tests tend to churn as implementation details change, even when written against behavior. Property tests have a higher initial cost, but that cost fades with familiarity, letting engineers build more interesting systems quickly and stably.
A tiered architecture makes testing tractable
Pairing property testing with a mental model of the tiered application structure eliminates frantic debugging sessions that once required entire teams crammed into a makeshift war-room. Application tiers define sub-component priority, so the areas that need hardening are clear. With clear boundaries, each sub-component becomes more isolated from external dependencies, making behavior easier to test. Defects surface early and predictably, instead of emerging during end-of-day emergencies.
