Long-horizon agents pursue outcomes over hours, days or weeks, often pausing to exchange information with people and external systems along the way. Since the running process, the network and the outside world all remain in play throughout that period, failures are not exceptional — they are expected. Durable execution is the pattern that lets an agent absorb them and carry on.

The economics and shape of long work

Operations have long been organized around scarce human attention: anything requiring weeks of someone's time was attempted only when the payoff justified it. Long-horizon agents shift that calculation, making work viable that previously was not. Customers who churn for lack of onboarding help can be guided through it one at a time; deals can be shepherded across sales, legal and procurement dependencies; patients can be tracked through referrals, appointments, rescheduling and paperwork. Conventional software automates predictable processes, but these tasks are not predictable, and the agent can pursue the outcome even when the route there is not known in advance.

Note that "long horizon" here is about elapsed time and many exchanges with the world, not about how many turns a model can complete without interruption — that latter figure is the one frontier-model benchmarks tend to report.

The agentic loop

Agents such as Claude Code or OpenClaw share a single structure, the agentic loop, with four parts: a goal (a program written, a part kept in stock); tools through which the agent acts (read_file, edit_file, bash; or get_quotes, place_order); state, holding everything known about progress so far; and a model that inspects state each turn and picks the next tool and its arguments. The code surrounding the model — the harness, as in the harness around Claude — runs the chosen tool and folds the result back into state. The loop ends at the goal or at abandonment, and whenever outside input is required the agent asks and waits.

Inside Stock Replenisher

A stock-ordering agent keeps a manufacturer's parts above safety stock: it gathers quotes, selects a supplier, seeks the procurement manager's approval, places the order and follows it until delivery. Most of the elapsed time is spent waiting on the approval, the supplier and the shipment, during which prices may change, questions may arrive, or silence may persist. In code, decider.decide calls the model — here the classification model Jev — with the GOAL, the replenishment's state and a prefiltered allowed_tools list; it returns a tool for that turn.

GOAL = ("Bring stock back above safety stock: order from a suitable "
        "supplier, get every price approved by the manager before money "
        "is committed, and see the order through to delivery.")
MAX_TURNS = 30

TOOLS = [
    GetQuotes(), DraftPurchaseOrder(), RequestApproval(),
    WaitForApproval(), PlaceOrder(), WaitForSupplierReply(),
    ReadSupplierReply(), WaitForGoods(), ClosePurchaseOrder(), Done(),
    ...
]

class Replenish:
    async def run(self, stock: StockLevel) -> ReplenishmentResult:
        state = State(stock.sku, stock.on_hand, stock.safety_stock)
        for turn in range(MAX_TURNS):
            allowed_tools = [t for t in TOOLS if t.allowed(state)]
            tool = await self.decider.decide(GOAL, state, allowed_tools)
            if isinstance(tool, Done):
                break
            if isinstance(tool, Wait):
                try:
                    payload = await asyncio.wait_for(
                        self.inbox[tool.event].get(),
                        timeout=timedelta(days=tool.days).total_seconds())
                    tool.apply(state, payload)
                except TimeoutError:
                    tool.apply(state, None)
            else:
                result = await tool.execute(state, self.context)
                tool.apply(state, result)
        return outcome(state)

The loop terminates when the model selects Done, or after MAX_TURNS; outcome records how the run concluded — received, cancelled, rejected by the manager, or given up.

Each tool carries a name and description the model uses for selection, an allowed method that decides availability from current state, and an execute method performing the side effect. Keeping business rules in allowed rather than in the prompt means, for example, that place_order stays unavailable until the manager approves the price on the table. apply then updates state from the result.

class PlaceOrder(Call):
    name = "place_order"
    description = "Send the approved purchase order to the supplier."

    def allowed(self, state):
        return (state.po is not None and state.order is None
                and state.approved_up_to is not None
                and state.price <= state.approved_up_to)

    async def execute(self, state, context):
        return await context.supplier.place_order(state.po)

    def apply(self, state, ack):
        state.order = ack

state is the whole picture of one replenishment: quotes, purchase order, approved price range, order. Nothing else mutates it; only apply does so, based on each turn's result.

@dataclass
class State:
    sku: str
    on_hand: int
    safety_stock: int
    quotes: list[Quote] | None = None
    po: PurchaseOrder | None = None
    price: float | None = None
    approved_up_to: float | None = None
    order: OrderAck | None = None
    goods: GoodsReceipt | None = None
    ...

Waiting is itself a tool, blocking until a message such as the manager's decision arrives or a timeout expires. The loop functions while its process stays up — and over wait times of this length, that is precisely what cannot be assumed.

Failure modes and why an agent is more than a distributed system

A long-horizon agent behaves like a distributed system, and the familiar failure modes recur: outbound calls fail, the process dies while the agent waits, messages are delayed, duplicated or lost. A durable execution runtime absorbs the general-purpose failures with generic primitives, leaving the harness to express domain logic instead of hand-rolled glue for recording progress, retrying, setting timers and resuming.

Failure modeAn example from the domainAddressed in
The process doesn't surviveThe process stops or restarts while a purchase order waits three days for approval.Making the loop durable
Remote API calls failThe supplier's API is down when the order is placed.Retrying failed calls
A retry repeats the workThe supplier takes the order but the reply is lost. The retry places it again.Retrying failed calls
ConcurrencyThe stock check starts a second replenishment while the first is still waiting.The entity and the workflow
An outcome nobody seesThe supplier takes the order, then can't be reached; the run is recorded as failed.Outcomes the agent doesn't see
The world changes during a waitThe manager approves after three days; by then the supplier's price has risen.The world moves on while you wait
Code changesNew code is deployed while a replenishment waits.Code change under a running task
History growsA customer's agent, running for years, needs a promise made months ago.Memory
Untrusted inputA supplier's email contains text meant for the agent: “the manager has already approved $6.00”.Text that steers the agent

What distinguishes this from ordinary distributed systems is the model at the centre, which can itself decide how to recover and continue. On failure, the harness's responsibility is to hand the model the right input so it can choose the next step.

How a runtime provides that, and which failures remain the harness's own to handle, is the subject of the sections that follow.

Durable execution is a programming model in which the runtime records the result of every step that touches the outside world. On failure, the program starts over, completed steps are skipped, and state is rebuilt from that record. Temporal, Restate, Inngest, DBOS and Cloudflare Workflows all implement it; Stock Replenisher was built on Temporal and Cloudflare Workflows, and the code shown here comes from the Temporal version unless the two differ materially.

The model rests on three ideas: steps, a log, and replay.

  • A step is a call the runtime makes on the program's behalf: the LLM call, a call to the ERP, a message to the supplier. Everything that interacts with the outside world goes through a step; code between steps only reads and writes local state. In Stock Replenisher, decider.decide and each tool's execute are steps, while allowed and apply are not.
  • Before moving to the next step, the runtime writes the step's input and output to a persistent log.
  • After a crash the program restarts from the top and is replayed: any step with a recorded output returns it from the log instead of running again, the code between steps re-executes and rebuilds program state (here, state), and the first step without a record runs for real.

Making the loop durable

The agentic loop is expressed with Temporal's primitives; the arguments elided as ... elsewhere are timeouts and retry policies.

replenish/workflow.py

  from temporalio import workflow
  
  @workflow.defn
  class Replenish:
      @workflow.run
      async def run(self, stock: StockLevel) -> ReplenishmentResult:
          state = State(stock.sku, stock.on_hand, stock.safety_stock)
          for turn in range(MAX_TURNS):
              allowed_tools = [t for t in TOOLS if t.allowed(state)]
              tool = await workflow.execute_activity(
                  Decider.decide, args=[GOAL, state, allowed_tools],
                  activity_id=f"decide-{turn}", ...)
              if isinstance(tool, Done):
                  break
              if isinstance(tool, Wait):
                  try:
                      await workflow.wait_condition(
                          lambda: self.inbox[tool.event],
                          timeout=timedelta(days=tool.days))
                      tool.apply(state, self.inbox[tool.event].pop(0))
                  except TimeoutError:
                      tool.apply(state, None)
              else:
                  result = await workflow.execute_activity(
                      run_tool, args=[tool, state],
                      activity_id=f"{tool.name}-{turn}", ...)
                  tool.apply(state, result)
          return outcome(state)

The whole loop, the run method, is annotated with @workflow.run, which makes it a workflow — Temporal's term for a durable program that is re-run after a failure with its state rebuilt from the log.

Every step that reaches outside goes through workflow.execute_activity, so Temporal logs the step's input and output under the name given by activity_id. (Temporal calls such a step an activity.) Two calls in the loop are activities: the model's choice of the next tool (Decider.decide) and the tool itself (get_quotes, place_order, close_purchase_order and the rest, via run_tool).

The third primitive, workflow.wait_condition, suspends the workflow until a condition holds — here, until a message arrives in the inbox — or the timeout passes. The wait is recorded too, so the process need not stay up while waiting. Everything else, namely allowed and apply, is unchanged and runs again on each replay. Cloudflare writes the same loop with step.do in place of execute_activity and step.waitForEvent in place of wait_condition; Restate, Inngest and DBOS have equivalents.

Consider one replenishment as Temporal recorded it. The agent fetched quotes, drafted a purchase order and requested manager approval. During the wait the process crashed and stayed down for a day. When it came back, nothing before the crash ran again: no new quotes, no second purchase order, no second approval request. When the approval arrived, the order was placed.

Entity workflows versus process workflows

Stock replenishment involves two loops. One follows a single replenishment for a SKU through to receipt of the goods. The other operates at SKU level: it checks stock against the replenishment policy, triggers replenishment one at a time, and serves as the SKU's point of contact for messages from the manager, suppliers and warehouse. Over its life a SKU is replenished many times. This maps to the distinction between process and entity workflows.

  • A process workflow is a durable workflow that performs one piece of work, such as a replenishment, and then finishes.
  • An entity workflow is a durable workflow that represents one business entity, such as a SKU, for that entity's lifetime; it maintains the entity's state and initiates process workflows.

The same pattern fits a subscription business, where a customer entity workflow holds the customer context and starts process workflows for onboarding, renewal, support and so on, each of which completes while the entity workflow continues.

Watching stock

the entity workflow: one per SKU

  @workflow.defn
  class SkuAgent:
      @workflow.run
      async def run(self, sku: str, outcomes: list[ReplenishmentResult]):
          stock = await workflow.execute_activity(get_stock, sku, ...)
          if stock.on_hand < stock.safety_stock:
              self.active = f"replenish-{sku}-{len(outcomes) + 1}"
              try:
                  result = await workflow.execute_child_workflow(
                      Replenish.run, stock, id=self.active)
              except ChildWorkflowError as e:
                  result = ReplenishmentResult(sku, "failed", str(e.cause))
              outcomes.append(result)
              self.active = None
          else:
              await workflow.sleep(CHECK_EVERY)
          workflow.continue_as_new(args=[sku, outcomes])

Each run of the entity workflow checks stock once. If the SKU is below safety stock, it starts a replenishment and waits for it to finish; otherwise it sleeps until the next check. continue_as_new then starts a fresh run under the same ID with an empty history, receiving only what it is given — here the SKU and the outcomes so far.

Receiving messages

The entity workflow also receives the SKU's messages from outside. It has a stable, unique ID, agent-SKU-123, for the SKU's life. Each outgoing request, whether an approval request or a purchase order, carries the SKU or the PO number so the answer can be routed back to agent-SKU-123. Neither the approving manager nor the replying supplier needs to know which replenishment is running; the entity forwards the message to the current one.

the manager's approval, sent to SKU-123

  await client.get_workflow_handle("agent-SKU-123").signal(
      "deliver", args=["approval", {"approved": True}])

the entity's address: messages from outside

  class SkuAgent:
      ...
      @workflow.signal
      async def deliver(self, event: str, payload: dict):
          if self.active:
              await workflow.get_external_workflow_handle(self.active).signal(
                  Replenish.deliver, args=[event, payload])

the replenishment's inbox: messages from the entity

  @workflow.defn
  class Replenish:
      @workflow.run
      async def run(self, stock: StockLevel) -> ReplenishmentResult:
          state = State(stock.sku, stock.on_hand, stock.safety_stock)
          for turn in range(MAX_TURNS):
              ...
          return outcome(state)
  
      @workflow.signal
      def deliver(self, event: str, payload: dict):
          self.inbox[event].append(payload)

Every message for SKU-123 passes through its entity workflow to the running replenishment, which applies them one at a time. In code, Stock Replenisher is now two classes: SkuAgent, the entity workflow, and Replenish, the process workflow running the loop above.

Waiting cheaply, and waking up fast

Long-horizon agents spend much of their time waiting on humans or external systems, and the runtime offers primitives for waiting without holding a process in memory or consuming compute. This matters under the entity-workflow pattern: every SKU has its own workflow, running as long as the SKU is stocked, so a manufacturer with thousands of parts has thousands of agents, almost all idle at any moment. If a wait pinned a process or memory, cost would scale with the number of SKUs and the length of waits rather than with work done.

workflow.wait_condition suspends until a message of the right type is in the inbox or the timeout expires. Both the wait's timer and the message that ends it are logged, so a wait survives a crash just as a step does.

Waking has a cost as well, since rebuilding a workflow means replaying the log, and that cost grows with log length; slow or expensive wake-up would make thousands of waiting agents as infeasible as thousands of open processes. In our runs, waking took under three seconds and tens of milliseconds of CPU. We measured waits on Cloudflare (deployed) and on a local Temporal server: a waiting workflow used no CPU on either platform; a workflow still in memory woke in milliseconds (14 ms on Temporal, about 0.1 s on Cloudflare); one rebuilt from the log took about two seconds on Cloudflare and under a tenth of a second on Temporal, growing as the log grew.

Where replay becomes too slow, the next option is a snapshot, so only the log after it must be replayed. This matters most for the entity, whose log would grow for the SKU's whole life. On Temporal, continue_as_new achieves the same effect by carrying the entity's state into a fresh run with an empty history. On Cloudflare the entity is a Durable Object, which reads current state from storage and has no log to replay. A replenishment workflow needs no snapshot, since its run ends and its log stays short.

Timeouts as a path through the loop

A wait ends either when a message arrives or when the timeout expires. If the supplier has not replied within seven days, wait_for_supplier_reply marks the order as overdue:

class WaitForSupplierReply(Wait):
    event, days = "supplier-reply", 7
    ...
    def apply(self, state, payload):
        if payload is None:
            state.supplier_overdue = True
        else:
            state.supplier_reply = SupplierReply(**payload).message

class ChaseSupplier(Call):
    ...
    def allowed(self, state):
        return (state.supplier_overdue and state.chases < MAX_CHASES
                and is_open(state))

That makes chase_supplier available, so the model sees it as an option on its next turn. In the recorded run, the supplier goes quiet after the order is placed; the wait ends on the seven-day timeout, the chase follows, and the reply arrives a day later.

Retries and at-least-once semantics

Calls to external systems fail for network reasons or because the receiving system is down. Runtimes provide retries with configurable retry policies so a failing dependency is not overloaded.

RETRY = RetryPolicy(maximum_attempts=6,
                    initial_interval=timedelta(seconds=2),
                    backoff_coefficient=2)

@workflow.run
async def run(self, stock: StockLevel) -> ReplenishmentResult:
    ...
    result = await workflow.execute_activity(
        run_tool, args=[tool, state],
        activity_id=f"{tool.name}-{turn}", retry_policy=RETRY)
    ...

What the runtime cannot do is make a retry safe. If the supplier takes the order but the response is lost, no result is recorded and the runtime calls again. This is at-least-once semantics, not exactly-once, and unhandled it can produce duplicate orders. To get exactly-once effect for a write, the receiver must recognise repeats via an idempotency key, which the harness must hold constant across every attempt and replay. In the place_order tool the PO number serves as that key.

class PlaceOrder(Call):
    ...
    async def execute(self, state, context):
        ...
        key = state.po.po_number
        ack = await context.supplier.place_order(po, idempotency_key=key)
        return PlaceOrderResult(placed=True, price=price, ack=ack)

In the recorded run the supplier's system is down on the first attempt and the runtime retries with the same PO number. The trace shows two attempts on place_order and a single order at the supplier.

The primitives used so far, on both platforms:

PurposeTemporalCloudflare
The agent: an entity workflow, one per SKU (SkuAgent)workflow (long-lived)Agent (a Durable Object)
Start a process workflow: one replenishmentchild workflowrunWorkflow
A step: run a call, record input and outputexecute_activitystep.do
Retry a failed stepRetryPolicyretries on step.do
Wait for a messagesignal + wait_conditionstep.waitForEvent
Send a message to a runsignalsendWorkflowEvent
Check on a scheduleworkflow.sleepscheduleEvery
Start a fresh run with an empty historycontinue_as_newnot used: the entity isn't a workflow

Together they let the loop survive crashes, wait without holding a process, and retry failed calls. Idempotency is the first thing the harness must handle explicitly, and other failure modes of the same kind follow in the next section.

Failure modes durable execution cannot fix

Durable execution guarantees that a workflow survives restarts. It does not guarantee that the workflow's decisions were right. Some residual failure modes come from the essential difficulty of a long-running task that spans people and systems; others are side effects of the tools themselves — the runtime and the model. Handling them takes pragmatic strategies in the harness, informed by close knowledge of the business domain.

The five failure modes covered here correspond to the ones from the earlier section on what can go wrong:

Failure modeKindWhyAddressed in
An outcome nobody seesEssentialA supplier takes the order, then can't be reached.Outcomes the agent doesn't see
The world changes during a waitEssentialPrices change while the manager decides.The world moves on while you wait
Code changesAccidental: durable executionReplay runs the new code against a log the old code wrote.Code change under a running task
History growsAccidental: the modelThe model can't see everything on every turn.Memory
Untrusted inputAccidental: the modelThe model can't tell the text it reads from its instructions.Text that steers the agent

Silent success and duplicate orders

Retries with an idempotency key handle only half the problem. A call the agent treats as failed may in fact have completed, with only the response lost. Because the agent cannot distinguish the two cases, "we don't know" becomes "it failed," and the agent proceeds as if nothing happened.

Mitigation means giving the agent tools for repair. cancel_order, for instance, can undo an accidental duplicate order.

class CancelOrder(Call):
    name = "cancel_order"
    description = ("Cancel an order with its supplier and tell them why, "
                   "for example a duplicate placed by an earlier run.")
    po_number = None  # the model's choice, from a list the harness builds

    async def execute(self, state, context):
        po = self.po_number
        # a generation model (here Claude) writes the email
        text = await context.writer.write_cancellation(po, state)
        sent = await context.supplier.send_message(
            po, text, idempotency_key=context.key)
        await context.supplier.cancel_order(po, idempotency_key=context.key)
        return Cancellation(po, sent)

    def apply(self, state, result):
        ...

A realistic trace: supplier A accepts an order and then goes dark, and the retries fail along with it, so place_order gives up after six attempts. Still short of stock, the agent starts another replenishment and orders from supplier C. Days later, supplier A confirms the first order by email.

The agent now checks the PO number and, if there is still time, calls cancel_order against supplier A with an explanation. It spotted the duplication only because the supplier wrote in. A persistent log records what the agent observed, not what actually occurred: the decision to order from supplier C was wrong when it was made, on incomplete information.

Decisions that outlive their inputs

A long-horizon agent decides at one moment and acts days later, and the facts underpinning the decision may have changed by then. A manager might approve a price from a three-day-old quote, while the supplier's price may have risen before the order goes out.

Three patterns let an agent cope autonomously:

  • Expire the decision. Attach a validity date to an approval, after which the agent must decide afresh. Quotes usually carry a validity period, and an approval can as well.
  • Re-check before committing. Re-read the facts behind any hard-to-reverse action, such as placing an order, immediately before taking it; if they have moved, re-decide instead of acting.
  • Approve a range, not a value. Let the decision tolerate variance. A manager who approves a price range frees the agent to act anywhere inside it without asking again.

In Stock Replenisher the manager approves a price plus up to 5%. The place_order tool then re-fetches the quote and compares it to that ceiling: within the ceiling, it orders at the quoted price; above it, no order is placed and the agent returns to the manager. The check lives in the tool, so it is deterministic and runs regardless of what the model decides.

class PlaceOrder(Call):
    ...
    async def execute(self, state, context):
        quotes = await context.supplier.get_quotes(state.sku)
        price = ...  # this supplier's price today, from quotes
        if price > state.approved_up_to:  # above the range: don't order
            return PlaceOrderResult(placed=False, price=price)
        ...  # within the range: order at today's price

So when the manager approves $4.20 with a $4.41 ceiling and the supplier confirms at $4.35, the agent lets the order stand; a confirmation at $4.54 would have gone back for re-approval.

Changing the harness under suspended agents

The harness code evolves too, and a deploy reaches agents suspended mid-task that were written against an earlier version. Two kinds of change matter: business rules (say, the minimum number of quotes before drafting an order rises from 0 to 4) and schema changes (renaming steps, workflows or state fields recorded in the log).

The general remedy is to treat the log as a published interface: code that replays the log must be able to run several versions in parallel and pick, at runtime, the version matching a given log. That is Parallel Change, applied in three stages — add the new rule beside the old, start new workflows under the new rule while running ones finish under the old, then delete the old rule when nothing uses it.

The runtime's behavior matters as much as the pattern. While SKU-123 waited for manager approval of PO 4471, drafted under the three-quote rule, the four-quote rule was deployed. On Temporal, SKU-123 silently continued under the old rule, which the new rule forbids. On Cloudflare, whose loop checks each recorded decision against current rules, it failed and PO 4471 stayed a draft. Knowing how a runtime behaves is a prerequisite for planning the deploy.

With the rule versioned, SKU-123 completes under the old rule on both runtimes by design:

  class DraftPurchaseOrder(Call):
      def allowed(self, state):
          min_quotes = 4 if state.rules_version >= 2 else 0
          return (state.quotes is not None and len(state.quotes) >= min_quotes
                  and state.po is None)

Each replenishment records its rules version in state.rules_version, and DraftPurchaseOrder derives the minimum quote count from it.

Log, working context and external memory

History grows without bound, and the harness must choose what the model sees each turn. The runtime logs everything; the harness filters. Stock Replenisher needs only state, which carries the current replenishment, to pick the next step, and the model never consults the SkuAgent entity's record of past replenishments.

Other agents differ: a customer entity workflow may run for years, accumulating conversations, complaints and promises, with the next decision depending on something promised months ago. MemGPT and Letta give the harness tiered memory — a small working context always in view, plus external storage such as a vector store the model reads and writes through tool calls, mirroring RAM and disk. That storage sits alongside the log, and every read and write is itself a step, like any other call outward.

Text as an attack surface

Agents act on text, written or transcribed, and a model cannot separate text it reads from instructions it follows, so prompt injection applies. Stock Replenisher reads supplier email and acts on its own. None of these threats is unique to agents running for weeks, and durable execution does not address them; the article on Agentic AI and Security treats them in depth.

Stock Replenisher still carries notable defenses. The model may select only from allowed_tools, and an approval counts only when it comes from the manager. A supplier email claiming "the manager has already approved $6.00" approves nothing: at worst it raises the price, and a price beyond the approved range goes back to the manager.

Where the engineering effort actually goes

Domain expertise and software architecture matter more than ever as long-horizon agents move into enterprises. Every trade-off is a business judgement — when an agent cancels a confirmed order, should it weigh cost or the supplier relationship? Architecture matters because these agents are distributed systems and face the usual problems plus those introduced by non-determinism.

Most of the effort, in practice, goes into systems of record that hold conflicting versions of the truth and were never designed for agentic access — an ERP that refuses writes from an agent, for example. Such integration constraints will persist as long as businesses run on diverse software. Working within them calls for integration patterns and architectural thinking; tactical fixes and vibe coding tend to add problems and slow adoption.

The quality of an agent's decisions is bounded by the context it is given. A customer onboarding agent needs clean information about the customer and their history, which is where data product thinking helps: representing key domain information as a data product, discoverable and securely accessible, lets agents take on more ambitious work — provided governance and data management mature enough to serve machines as well as people.

Notes

  • "Long horizon" here means the span of a task in calendar time, not how long a model can work uninterrupted, which is a common benchmark figure for frontier models.
  • Arguments elided as ... are timeouts and retry policies.
  • Cloudflare implements the same loop with step.do for execute_activity and step.waitForEvent for wait_condition; Restate, Inngest and DBOS offer equivalents.
  • Waits were measured on a deployed Cloudflare setup and a local Temporal server. A waiting workflow used no CPU on either. A workflow still in memory woke in milliseconds (14 ms on Temporal, about 0.1 s on Cloudflare); one rebuilt from the log took about two seconds on Cloudflare and under a tenth of a second on Temporal, longer as the log grew. A wake consumed tens of milliseconds of CPU.