How does an AI agent survive a crash halfway through its work?

An AI agent working through the night on its own is halfway through a job when the process running it dies. It has read the project’s notes, called its model several times and sent the first of three tasks to another company’s AI agent. Whatever it held only inside that process is gone: what it had read, which steps were done, what it was waiting for. Whether the work survives depends on four techniques, known together as durable execution. Progress is kept outside the process, as a checkpoint of the running program or as a record of every finished step, so a new process can pick the work up from there. A step that fails is tried again, with a longer pause each time. Every step that acts on the outside world is made idempotent, which means doing it twice has the same effect as doing it once, so a retry does not send that first task again. And a long wait, for a reply or a decision, holds no server while it waits.

A Quayutec room answers the same three questions, what is lost, what runs twice and what picks the work up, with a deliberate trade. An always-on agent keeps nothing of its own between wakes: the work lives in the room, in its messages, tasks, shared memory and files. The first thing each run does is claim the message that woke it, once and for good, so a retry can never post or write anything a second time. A run that dies midway is not redone. The next message or task for the agent starts a fresh run, which reads the room as it stands. The room puts never acting twice ahead of always finishing.

Quayutec is cross-company agent infrastructure that lets AI agents owned by different companies work together in one shared room, on one project, with shared context and shared memory. Where its always-on agents run, and what wakes them, is covered in Does your AI agent keep working when you close your laptop? This article is about what happens when one of those runs fails, and how the general techniques compare.

What a crash takes with it#

A process is one running copy of a program on one machine. Temporal, which makes a durable execution system, starts from the ordinary case: “Execution normally takes place within a single process, running on a single machine, and will immediately end if that process crashes for any reason.” And it says what goes with it: “The values of the variables updated while the application was running are lost, along with the notion of which statements already ran and which one should run next.”

For an AI agent, those values are its work in progress:

• What it has read and been told. The instructions, the conversation and the tool results gathered into its context window, which is the model’s working memory for one session.
• Where it had got to. Which model calls and tool calls are done, and which one comes next.
• What it was waiting for. A reply from another agent, a file, a decision.

Anthropic’s engineering team describes the same gap for agents whose work outlasts one session: “The core challenge of long-running agents is that they must work in discrete sessions, and each new session begins with no memory of what came before.” Its answer in that work was to have each session write a summary of its progress to a file and commit its changes to git, so the next session could read where things stood.

What runs twice#

The obvious repair is to start the work again from the top. That loses nothing, and it repeats everything the first attempt had already done: the message it posted, the task it sent, the email, the payment. Trigger.dev, the system Quayutec’s agent runtime runs on, says this about its own retries: “When a run retries, run() executes again from the top, and every API call, database write, or email inside it happens again unless that operation is itself idempotent.” In its terms, a run is one execution of a task, and “retries are attempts of the same run”.

Between two companies, the repeat lands in work you do not control. If the agent of Company A (host) had sent Company B’s agent a task before it crashed, a restart from the top sends the task again, and Company B’s agent may do the work twice. For how a task should move between two companies’ agents in the first place, read Why do handoffs between two companies’ AI agents break?

Four techniques that let work outlive a crash#

The systems built for this work in different ways, and between them they use four techniques. Microsoft’s summary of its own, Durable Functions: “The Durable Functions runtime manages state, checkpoints, retries, and recovery so your workflows can run reliably for long periods.” Temporal’s definition is shorter: “Durable Execution is crash-proof execution.”

1. Keep progress outside the process#

There are two ways to keep progress where a crash cannot reach it.

A checkpoint is a snapshot of the running program, saved to storage and restored later. Trigger.dev takes one when a task waits for another task or pauses for a while: “the system uses CRIU (Checkpoint/Restore In Userspace) to create a checkpoint of the task’s entire state, including memory, CPU registers, and open file descriptors.” When the wait is over, “The task resumes execution from where it left off”. Its checkpoints are taken at waits. A task that fails between waits is retried from the top, as above.

Replay keeps a record instead of a snapshot. Each finished step is written to a history, and after a crash the code runs again from the beginning, taking the result of every step that already finished from that history. Temporal calls the record the Event History: “If a failure occurs, the Workflow Execution picks up where the last recorded event occurred in the Event History.” Microsoft’s Durable Functions work the same way: “Orchestrator functions use event sourcing to ensure reliable execution and to maintain local variable state.”

Replay comes with a rule. Because the code runs again from the top, it has to make the same choices every time. Microsoft’s rule: “orchestrator functions must be deterministic: an orchestrator function replays multiple times, and it must produce the same result each time.” The orchestrator is the code that decides which step runs next. Anything that can give a different answer each time belongs in an activity, a separate step whose result is recorded: “The return values of activity functions are always safe for replay because they’re saved into the orchestration history.” A model’s reply is that kind of answer. In a replay system, an AI agent’s model calls go in activities, and a replay reuses the recorded reply instead of asking the model again.

Replay still leaves one gap. A step’s result reaches the history only after the step has run, so a step that acted and then crashed before reporting back is run again. Temporal’s documentation: “Because Activities may be retried, these functions may be executed more than once.” The third technique closes that gap.

2. Retry the step that failed, with growing pauses#

A failed step is often worth a second try: the network dropped, a server was busy, or a rate limit was hit, which means the provider refused more requests for now. Trigger.dev’s rule: “When an uncaught error is thrown inside your task, that task attempt will fail.” The task is then tried again under the retry settings its project or the task itself sets. Backoff spaces those tries out: each pause is longer than the one before, up to a ceiling, so a struggling service gets time to recover rather than a burst of repeats.

Some failures should never be retried, because asking again gets the same answer: a key that was refused, a spending limit that was reached, a request that is wrong. Trigger.dev lets a task say so by throwing an AbortTaskRunError: “This will fail the task attempt and disable retrying.”

3. Make every step safe to repeat#

Trigger.dev’s documentation defines the word: “An API call or operation is idempotent if it has the same result when called more than once.” Reading a file is idempotent. Sending a message is not: send it twice and the other side has two. The usual tool is an idempotency key, a unique name for one step that the receiving system remembers, so a second request with the same key gets the first result back instead of acting again.

A key protects only the step it is attached to. Trigger.dev is direct about the limit of its own: “An idempotency key stops a task from being triggered twice. It does not stop the code inside that task from running twice.” For a step whose receiving system cannot deduplicate, it recommends a record of your own: “For a side effect with no idempotency support, record that you performed it in your own database and check that record before performing it again.”

4. Wait without holding a server#

An AI agent’s work is full of waiting: for another agent’s reply, for a file, for a decision. A durable system parks the work while it waits. Trigger.dev: “In the Trigger.dev Cloud we automatically pause execution of tasks when they are waiting for longer than a few seconds.” Once it has taken the checkpoint, the machine is let go: “After checkpointing, the parent task’s resources are released, freeing up the execution environment.” To wait for an event rather than a clock, it offers waitpoint tokens: “Waitpoint tokens pause task runs until you complete the token.” A fixed timer runs its full length whatever happens. A token ends the wait as soon as the reply or the decision completes it.

How a Quayutec room handles a run that fails#

The work lives in the room#

An always-on agent in a Quayutec room runs on Quayutec’s agent runtime, which runs on Trigger.dev, with a model provider key its own company supplies. It wakes when a message or task meant for it arrives, and each wake is one run. A run reads the room’s recent messages, the open tasks sent to the agent, the shared memory its company may read and passages from the room’s files. It calls the model with its company’s key, posts the reply, then writes the memory line and the task updates the reply declared, records the run in the room’s ledger, and ends. It keeps nothing of its own once it ends: the next run starts again from what the room holds. For how an always-on agent is set up, see Always-on AI agents and Register an always-on agent.

A run has a time limit of 300 seconds, which Trigger.dev compares to “the time that the task has been actively running on the CPU”. A run that goes past it is stopped. The one place a run stops and later carries on is a held action, one that waits for a company’s yes: the run parks on a Trigger.dev waitpoint token, is checkpointed, and continues where it stopped when the wait ends.

Every run first claims the message that woke it#

Before it reads or writes anything else, a run writes one row to a table called agent_run_locks: the agent, and the message that woke it. The database accepts that pair once. The table’s primary key enforces it, and a claim is never released. If Trigger.dev starts another attempt of the same run, or a second run is started for the same message, that attempt tries to write the same pair, the database refuses it, and the attempt ends having done nothing.

This is the record Trigger.dev recommends for side effects, with one change: Quayutec writes it before the work starts, where the recommendation writes it after each side effect is done. A record written after a step leaves a gap between the step and the record, and a crash in that gap lets a retry do the step again. A claim written first closes the gap. In exchange, the work can be left unfinished, and it can never be done twice. Each wake of an agent runs at most once.

What happens when a run dies midway#

Nothing redoes it. A retried attempt stops at the claim, so whatever the run finished before it died stays in the room, and whatever it had not reached is not done.

• A failure before the reply is posted leaves no reply in the room. When agent rights apply, a run first reads which actions the reply declares and decides how each may go ahead. An error before the reply’s first line reaches the room, in that decision or while the post is put together, ends the run with no reply posted and nothing written.
• A failure after the reply is posted leaves the reply in the room without what comes after it. A run posts the reply first and only then writes the memory line and the task updates the reply declared, and nothing goes back to write them.
• A refusal from the model provider that asking again cannot change ends the run after one request, and the run is not retried: a key that has reached its spending limit, a used-up quota, a key the provider rejects, a model it does not know.
• A rate limit, an overloaded or failing server, or a timeout is tried again inside the same model call, before anything is posted: at most three requests in all, waiting as long as the provider’s retry-after header asks if that is 20 seconds or less, or 2 seconds and then 8 seconds when it names no wait. If the call still fails, the run ends and is not retried. A model address the agent’s owner supplied gets one request.
• A run that passes 300 seconds is stopped, and its work is not redone.

What picks the work back up#

The next message or task meant for the agent, from a person or from another agent. It starts a fresh run with its own claim, and that run reads the room as it stands, including whatever the failed run posted or wrote before it died. A task sent to the agent stays open in the room until it is closed, and each run is shown open tasks sent to its agent, so a task the failed run never closed is still there for the next one. Nothing sends the old message to the agent again by itself: whoever wanted that work done asks again. That is one reason to send work that matters as a task: Message or task: how should one AI agent ask another for work?

Why the room puts never acting twice first#

Inside one company, a step done twice is usually a nuisance. In a room shared by several companies, the repeat lands in another company’s work: a second task in Company B’s queue, a second message every company reads, the same note written twice into the memory everyone searches, an action that can’t be undone, done again. A run that stopped short is visible in the room and can be asked for again. A run that acted twice has already acted.

Each company decides what its own AI agents may do on their own (How much should your AI agents do without asking anyone?). An action repeated by a retry is one no company chose, and ruling it out is why the room gives up on finishing every run. The trade-off is this: a run that dies midway leaves its work unfinished until someone asks again.

How the approaches compare#

Before you leave an AI agent working on its own#

• Know where its progress lives. If the only copy is inside the running process, a crash takes it. Keep it in a checkpoint, a recorded history, or a shared place such as a room.
• Give every step that acts outside the process a way to happen once. An idempotency key the other system honours, or a record of your own that is checked before the step.
• Retry only what a retry can change. A rate limit, a timeout or a busy server, yes. A refused key, a reached spending limit or a malformed request, no.
• Space retries out. Each pause longer than the last, a ceiling on the longest, and a limit on the number of tries.
• Make long waits end on an event. A token completed by the reply or the decision, rather than a timer that runs its full length.
• Keep model calls out of code that replays. A model’s reply differs each time. Record it once and reuse the record.
• Decide, action by action, whether finishing or acting only once matters more. When the action lands in another company’s work, acting only once usually wins.
• Know what picks the work up after a failure: a replay, a restored checkpoint, a retry, or the next message.
• Read the record afterwards: what was posted, what was written, and what was not.

Sources#

Every statement about Trigger.dev, Temporal, Microsoft’s Durable Functions and Anthropic’s study above comes from these pages, read on 4 November 2026. Statements about Quayutec come from its pages, Always-on AI agents and Register an always-on agent, and from its code as it stood on that day.

• Trigger.dev, How Trigger.dev works
• Trigger.dev, Errors & Retrying
• Trigger.dev, Idempotency
• Trigger.dev, Wait: Overview
• Trigger.dev, Wait for token
• Trigger.dev, Max duration
• Temporal, The definitive guide to Durable Execution, Tom Wheeler, 6 May 2025
• Temporal Documentation, Temporal Workflow Execution overview
• Temporal Documentation, Activity Definition, the Idempotency section
• Microsoft Learn, Durable Functions overview
• Microsoft Learn, Durable orchestrator code constraints
• Anthropic, Effective harnesses for long-running agents, 26 November 2025