Testing a fault-tolerant cron

Chronos is a distributed cron for Mesos, and we want to know if it can survive basic network partitions. Our test environment places Zookeeper on all five nodes, Chronos across all five nodes, and separates Mesos control and worker duties: masters run on n1-n3, slaves on n4-n5. Versions under test: Zookeeper 3.4.5+dfsg-2, Mesos 0.23.0-1.0.debian81, Chronos 2.3.4-1.0.81.debian77.

Chronos adds another layer of leader election on top of Mesos, which itself has a leader among its masters. Slaves feed resource offers to the leading master, which negotiates with frameworks like Chronos. Both Chronos and Zookeeper store job state and discovery information. That's four distinct coordination domains stacked together — plenty of room for a partition to break something.

Constructing workloads

Jepsen generates randomized jobs, each with a unique integer :name, a :start time, a repetition :count, a :duration for each run, an :epsilon window for launch slack, and an :interval between window starts. Simpler generators looked attractive, but real Chronos behavior forced constraints on us:

  • Task spin-up takes seconds, so runs can spill past epsilon. We add an epsilon-forgiveness buffer to account for it.
  • Jobs can't start immediately after submission, so we delay the first target with a head-start.
  • Chronos avoids concurrent runs of the same job, bounding the minimum interval between targets. Our intervals must be wide enough that a run at the end of one epsilon-plus-forgiveness window still finishes before the next begins.

Our generator submits jobs through an HTTP POST to Chronos, and only acknowledged submissions count toward correctness. Each job's task writes its job ID and current time to a file when it starts, sleeps for the duration, then writes the current time again to mark completion. Collecting those files from all nodes reconstructs the full run history. We parse runs from each host and union them without clock correction, since node clocks are perfectly synchronized in this test.

The client issues add-job operations with a 30-second randomized stagger, while the nemesis creates and resolves failures every 200 seconds. After a stabilization period, a read operation pulls the current run set and we check the results.

What correctness means for cron

A scheduler is correct when every target time in every acknowledged job has a corresponding run. Since this isn't a realtime system, a run must simply begin within [t, t + epsilon] — no requirement on completion time. Failure tolerance means duplicate runs are acceptable; the scheduler can always re-execute a task that appeared to die, which is preferable to missing a target entirely.

For a job that was acknowledged by Chronos, we expand its parameters into a set of target times. Then we try to assign each target to a distinct run. If the scheduler meets every target for the job, that job is valid.

To determine if a job's targets were all met by that job's runs, we solve a constraint satisfaction problem over assignments of targets to runs, with freedom to assign any target to any run as long as launches fall in the target's window and each run maps to at most one target. We can also check the result in linear time when target windows don't overlap by sorting and pairing both series. That fast path also yields useful partial solutions when the full set of jobs fails — showing which jobs broke and which stayed valid.

Applying this worker across all jobs — grouping runs by job ID and requiring satisfaction per job — yields the total verdict.

Failure modes under partition

Chronos does not make diagnosing failures easy. Invalid requests—say, a job with a malformed date—come back as HTTP 400 with an empty body.

{:orig-content-encoding nil,
 :request-time 121
 :status 400
 :headers {"Server" "Jetty(8.y.z-SNAPSHOT"
           "Connection" "close"
           "Content-Length" "0"
           "Content-Type" "text/html;charset=ISO-8859-1"
           "Cache-Control" "must-revalidate,no-cache,no-store"}
 :body ""}

Proxying requests to the leader can crash Chronos outright, producing invalid responses:

org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 1290; received: 0)

Or the more terse:

org.apache.http.NoHttpResponseException: n3:4400 failed to respond

Occasionally the error is wonderfully opaque:

{:orig-content-encoding nil,
 :trace-redirects ["http://n4:4400/scheduler/iso8601"]
 :request-time 19476
 :status 500
 :headers {"Server" "Jetty(8.y.z-SNAPSHOT"
           "Connection" "close"
           "Content-Length" "1290"
           "Content-Type" "text/html;charset=ISO-8859-1"
           "Cache-Control" "must-revalidate,no-cache,no-store"}
 :body "<html>\n
        <head>\n
        <meta http-equiv=\"Content-Type\" content=\"text/html;charset=ISO-8859-1\"/>\n
        <title>Error 500 Server Error</title>\n
        </head>\n>
        <body>\n
        <h2>HTTP ERROR: 500</h2>\n
        <p>Problem accessing /scheduler/iso8601. Reason:\n
        <pre>    Server Error</pre></p>\n
        <hr /><i><small>Powered by Jetty://</small></i>\n  \n  \n ... \n</html>\n"}

And sometimes you get no response because there is no process left to answer. When Chronos loses its Zookeeper connection, rather than retrying, it crashes the entire JVM and waits for a supervision daemon to bring it back. The official Mesosphere Debian packages ship no such supervisor, and service chronos start is not idempotent; a scripted restart can easily leave you with zero or dozens of conflicting Chronos processes.

Chronos is the first system Jepsen has tested that hard-crashes under network partition. The maintainers defend this as intentional: keeping the process alive through a ZK outage would permit split brain behavior, so crashing is expected, if undocumented. The philosophy is applied unevenly: a network partition can also crash the Mesos master, and Mesos maintainers say that's not how Mesos should behave.

Even without a partition, scheduling has a floor. Jobs with intervals finer than --schedule_horizon (60 seconds by default) miss their target times because the scheduler loop can't keep up. Lowering the horizon to 1 second lets Chronos hit every execution for intervals around 30 seconds—as long as the network holds.

Quorum stacking and recovery failure

A partition cleanly separating two nodes from three halts job execution entirely; worse, jobs stay halted after the network heals. The plot below charts targets (thick bars) against runs (narrow, darker bars). Green targets are satisfied by a run inside the time window; red targets show jobs that should have run but didn't.

Chronos fails to run any jobs for the first two minutes, then fails to run any jobs after the partition initiates, and never recovers.

The gray region marks a partition isolating [n2 n3] from [n1 n4 n5]. Chronos stops accepting new jobs for about a minute after the partition begins, then recovers its quorum. But Mesos, to preserve a majority of its nodes [n1 n2 n3], can only elect a leading master in [n2 n3]—which isolates the Mesos leader from Chronos. The stack depends on three distinct quorums (ZK, Mesos, Chronos) all staying connected to one another; any partition that crosses those boundaries means no jobs run.

What stands out is that recovery never happens. After the partition heals, Chronos keeps accepting new jobs but never runs them. The team's recommended workaround, --offer_timeout (set to 30secs in testing), along with restarting Chronos and Mesos processes, lets Mesos reclaim resources. The success is partial: some jobs run outside their windows, some run twice in too short a span, and some, like Job 9, never recover.

Chronos only recovers some jobs but not others, and runs jobs at the wrong times

The underlying issue is bug #520: when a new Chronos leader takes over, it registers with Mesos as an entirely new framework rather than re-registering as the old one. Mesos still thinks the original framework owns the cluster's resources, so it offers nothing to the new leader. Why the original leader consumed all resources when it needed a fraction is unclear.

In exceptional cases, jobs can even run during a partition, followed by a cluster of failures after resolution and lingering scheduling errors.

There are just SO MANY EXCITING WAYS FOR CHRONOS TO FAIL

The timeline across these tests runs roughly:

  • 0 seconds: Mesos on n3 becomes leading master
  • 15 seconds: Chronos on n1 becomes leader
  • 224 seconds: a partition isolates [n1 n4] from [n2 n3 n5]
  • 239 seconds: Chronos on n1 detects ZK connection loss and does not crash
  • 240 seconds: several elections; n2 becomes Mesos leading master
  • 270 seconds: Chronos on n3 becomes leader and detects n2 as Mesos leading master
  • 375 seconds: partition heals
  • 421 seconds: Chronos on n1 recovers its ZK connection and recognizes n3 as new leader

Notably, in this test neither leaders nor followers crashed on connection loss; they slept and reconnected. That contradicts the stated design in issue #513, where a crash was described as necessary. Fail-fast is apparently not a consistent rule.

Recommendations for operators

Mesos and Chronos documentation targets developers, not operations. Nothing in the docs prepares you for the fact that Chronos nodes may need a supervision daemon, and the official packages don't provide one. You'll need to write and test your own.

Similar conditions produce wildly different outcomes: both Mesos and Chronos can sleep through some partitions and crash on others. Error messages give you little to work with, and internal visibility is poor. As Camille Fournier's talk on consensus systems puts it, "Zookeeper Owns Your Availability." Chronos runs three consensus systems—ZK, Mesos, Chronos itself—and any one being down stops all work.

In Jepsen's testing, no network failure has ever been fully recovered by Chronos. Transient resource failures can wedge the scheduler permanently. Production users report that node failures are handled well, but ZK unavailability wedges the system. Where possible, prefer redundant nodes running cronfiles and coordinating via a consensus system if you can tolerate the operational complexity; several engineers suggest Aurora as a more robust (and harder to set up) alternative.

If Chronos stays in your stack:

  • Wrap Chronos and Mesos in automatic restart supervisors
  • Monitor uptime to catch restart loops
  • Keep schedule_horizon shorter than your shortest job interval
  • Set Mesos --offer_timeout to a value you've tested
  • Instrument jobs to track whether they actually ran
  • Design jobs that tolerate running late, or not at all
  • Avoid network failures

The correct restart pattern for a fully wedged cluster is still unknown; some combination of total restarts and ZK cleanup may fix it. Even when the underlying bugs are resolved, expect no job execution during a partition. If you can live with zero runs during a network disruption, Chronos may still work; if you need jobs to keep running through one, it won't.