Crash recovery

A durable runtime records each model call and tool call of a run as a step. When the process crashes, the next process continues the run from its last step: finished steps are not run or billed again, and a tool call that was in flight is either run again or reported to the model as interrupted.

The terminal and desktop apps, the daemon and the server do this on their sessions.db file. The Crash recovery setting controls it. In your own code, pass a runtime to createAgent.

What it covers#

Use it in code#

import { createAgent } from 'zidane'
import { createSqliteRuntime } from 'zidane/durable/sqlite'
import { anthropic } from 'zidane/providers'

const runtime = createSqliteRuntime({ path: './sessions.db' })
const agent = createAgent({ provider: anthropic(), durable: { runtime, onInterrupted: 'resume' }, sessionId: 'job-42' })

// After a crash, the next process's run() first continues the interrupted run, then runs this prompt.
await agent.run({ prompt: 'Migrate the config files.' })

sessionId loads the session from the runtime's store, or creates it there. onInterrupted says what run() does with a run a dead process left: 'resume' continues it first, 'heal' (the default, and what a bare durable: runtime means) closes its open tool calls with the heal text (see Off under In the apps) and starts the new run. 'resume' only continues a run whose process is gone and that is not waiting on an unanswered question. Any other run is healed. With 'heal', call await agent.resume() yourself to continue it. It resolves null when there is nothing to continue. It also resolves null when the run's checkpoints no longer match its saved turns (the history was rewritten): the run is healed, and the resume:mismatch hook fires.

To stop a run so the next process can resume it (before a clean exit), call await agent.suspend({ graceMs }): the model stream stops at once, running tool calls get graceMs to finish, no new tool call starts, and the run keeps its checkpoint. run() resolves with suspended: true. agent.abort() ends a run for good.

To pass a session you already hold (session instead of sessionId), its session.store must be durable.stores.session: createAgent throws otherwise. A session with no store passes. A run from a build without durability is never resumed: resume() resolves null, and the next run() closes its open tool calls with the heal text.

Without a runtime, the agent records nothing, as before.

Safe and unsafe tools#

A tool call that was running at the crash follows the tool's replay policy:

const lookup: ToolDef = {
  spec: { name: 'lookup', description: 'Read a record', inputSchema: { type: 'object', properties: { id: { type: 'string' } }, required: ['id'] } },
  replay: 'safe',
  execute: async ({ id }) => readRecord(id as string),
}

Mark a tool safe only when running it twice does the same as running it once. The built-in read_file, grep, glob, list_files, todoread, skills_read and wait_task are safe. Every other tool, MCP tools, web_search and fetch_url included, is unsafe.

In a resumed run, an interrupted unsafe call gets this result:

[Tool use interrupted] The process running this call stopped before it finished. Its effect may or may not have happened: check before you run it again.

A codemode script runs again, and the calls it already made answer from their records. If one of its unsafe calls was interrupted, the whole codemode call is. A sub-agent is its own run: the resumed parent continues the same child run.

A call approved before the crash is not asked again, codemode and spawn calls included. An approval still waiting when the process died is asked again, and the call runs once. A call that runs again is checked against the deny rules in force when it runs.

A resumed run keeps the limits, model and effort its run() had, and its time budget (maxWallMs) carries over instead of restarting. On a checkpoint run, ask_user and present_plan wait on an awakeable, so an answer given while no process ran the run is kept. If you pass your own requestInteraction to createInteractionTools, answer with runtime.resolveAwakeable(options.awakeable.id, response) so the answer survives a crash.

agent.decide outside a run is not recorded, on any runtime. With Restate, restateRuntime sets no maxConcurrentTools, so the core default, 10, applies (localReliabilityBehavior() uses 3). Set behavior.maxConcurrentTools: 1 to run calls one at a time. Without promises: RestatePromise, Restate tool steps run one at a time anyway.

Performance#

Durability adds about 0.36 ms per turn. That figure comes from a fake model that answers at once, the worst case, since the turn then does nothing but the loop's own work. With a model turn of 20 ms it is about 1 ms, or 4.8 %. With a real model, whose turns take seconds, the cost does not show.

• A 5 MB tool output costs about 3 ms more. Outputs over 256 KiB go to the call's persisted-output file, not into the database, when the tool runs on the local machine and behavior.persistDir is set. The apps set it (<cache or data dir>/sessions/<id>/tool-results/<callId>.txt). The example above does not, so its outputs stay in the database.
• Opening a session on a file with 10,000 runs takes under 0.4 ms.
• Every commit syncs the WAL (synchronous = FULL). Under WAL a process crash keeps commits even at NORMAL. FULL adds a guard against an OS crash on Linux.

The method and all the numbers are in DURABILITY.md.

In the apps#

Crash recovery is in Settings, under Sessions: on the General page in the desktop app, on the Agent tab in the TUI.

• A change applies to new runs. A project can override it with durability in its projects.json entry. There is no UI for the override.
• On a host whose session store is not SQLite, the row shows Unavailable and can't be changed. zidane-server and the daemon always use SQLite.
• The daemon and the server look for interrupted runs when they start. Every app also looks when you open a session.
• A run that stopped on an open question or plan shows its question again. Answering it, in any window or in Slack, continues the run from its checkpoint. An answer given while no process ran the run is kept, and the run continues without asking. A question left open by a build without durability gets its answer as a tool result, and the run does not continue.
• Messages you steer into a running run are saved when they arrive. After a crash or a takeover, they are sent once, when the run ends.
• In Slack, a resumed run keeps editing its reply in the thread, and questions the run was waiting on come back to the thread when the channel connects.
• Calls of a resumed batch that had not finished show a resuming badge.
• When another process took a run over, the run ends with "this run continued in another process".
• A resumed run's usage counts the turns the crashed process saved once.

On the command line, --durability on|resume-ask|off applies with --session-db. bun start asks on a terminal. --json, --stream-json and bun run headless never ask, so under resume-ask they do not resume. The session setup behind these is tested. The flag parsing and the terminal prompt are not.

Reference#

• zidane/durable
• zidane/durable/sqlite
• zidane/restate
• DURABILITY.md for the full contract: step keys, fencing, compatibility and writing your own runtime