# zidane/eval

> Eval runner primitives that score headless agent runs and write artifacts.

Import: `import { … } from 'zidane/eval'`. Source: [src/eval.ts](https://github.com/Tahul/zidane/blob/main/src/eval.ts).

Use this entry point to test agent behavior against a real provider. `runEvalCase` prepares a workspace, runs the agent [headless](https://zidane.si/docs/cli-headless.md), applies scorers, computes metrics and the trajectory, and writes artifacts. Built-in scorers check the run status and the files in the workspace. `llmJudge` grades a run with a model. `defineEval` registers a case factory, and `buildRegisteredEvals` builds the cases with the provider and judge that the harness passes.

```ts
import { fileExists, runEvalCase, statusCompleted } from 'zidane/eval'
import { anthropic } from 'zidane/providers/anthropic'

const result = await runEvalCase({
  id: 'creates-readme',
  suite: 'files',
  prompt: 'Create README.md with a one-line project description.',
  provider: anthropic(),
  maxTurns: 8,
  workspace: { capture: ['.'] },
  scorers: [statusCompleted(), fileExists('README.md')],
})
console.log(result.passed, result.score)
```

## Functions

### `artifactPath`

Function.

Resolves the artifact directory of a case: `<root>/<suite>/<id>`. Characters that are unsafe in a path segment are replaced.

```ts
function artifactPath(root: string, result: EvalCaseResult): string
```

**Parameters**

- `root` (`string`): Root artifact directory.
- `result` (`EvalCaseResult`): Case result that names the suite and id.

**Returns** `string`

Source: [src/eval-artifacts.ts:54](https://github.com/Tahul/zidane/blob/main/src/eval-artifacts.ts#L54)

### `buildEvalRunSummary`

Function.

Builds the summary of an eval run: case counts, mean score, duration and usage totals, one row per case, metric and tag aggregates, and per-variant summaries when cases have variants.

```ts
function buildEvalRunSummary(input: readonly EvalCaseResult[]): EvalRunSummary
```

**Parameters**

- `input` (`readonly EvalCaseResult[]`): Case results in any order. They are sorted by variant, then suite and id.

**Returns** `EvalRunSummary`

Source: [src/eval-summary.ts:28](https://github.com/Tahul/zidane/blob/main/src/eval-summary.ts#L28)

### `buildRegisteredEvals`

Function.

Snapshot of every registered eval, materialized for one provider/judge pair.

```ts
function buildRegisteredEvals(ctx: EvalDefinitionContext): EvalCaseOptions[]
```

**Parameters**

- `ctx` (`EvalDefinitionContext`): Provider and judge to build the eval cases with.

**Returns** `EvalCaseOptions[]`

Source: [src/eval-registry.ts:64](https://github.com/Tahul/zidane/blob/main/src/eval-registry.ts#L64)

### `buildTrajectory`

Function.

Build an ordered, grouped trajectory from a transcript. Walks assistant
turns in order (an aborted one, output a process death cut off and sent
again, is skipped), emitting `think` / `text` / `tool:<name>` steps, grouping
only consecutive identical kinds. Per-step duration is approximated from the
owning turn's `createdAt` delta to the next turn.

```ts
function buildTrajectory(transcript: SessionTurn[]): Trajectory
```

**Parameters**

- `transcript` (`SessionTurn[]`): Session turns of the run. Only assistant turns contribute steps.

**Returns** `Trajectory`

Source: [src/eval-trajectory.ts:46](https://github.com/Tahul/zidane/blob/main/src/eval-trajectory.ts#L46)

### `clearRegisteredEvals`

Function.

Drop every registered eval — test isolation helper.

```ts
function clearRegisteredEvals(): void
```

Source: [src/eval-registry.ts:69](https://github.com/Tahul/zidane/blob/main/src/eval-registry.ts#L69)

### `computeEvalTagScores`

Function.

Computes the mean normalized score for each tag across the given metrics.

```ts
function computeEvalTagScores(metrics: EvalMetric[]): Record<string, number>
```

**Parameters**

- `metrics` (`EvalMetric[]`): Metrics of one case.

**Returns** `Record<string, number>`

Source: [src/eval-metrics.ts:168](https://github.com/Tahul/zidane/blob/main/src/eval-metrics.ts#L168)

### `createEvalAgent`

Function.

Multi-turn eval agent over the low-level headless runner. It keeps session
and execution lifetime outside any Playwright-specific fixture layer,
so downstream platforms can wrap it with their own `agent.run(...)` shape.

```ts
function createEvalAgent(options: CreateEvalAgentOptions): EvalAgent
```

**Parameters**

- `options` (`CreateEvalAgentOptions`): Provider plus [`HeadlessOptions`](https://zidane.si/docs/reference/headless.md#HeadlessOptions) defaults for every run. `cwd`, `execution`, `session` and `store` set up the shared environment.

**Returns** `EvalAgent`

**Example**

```ts
~~~ts
import { createEvalAgent } from 'zidane/eval'
import { anthropic } from 'zidane/providers/anthropic'

const agent = createEvalAgent({
  provider: anthropic({ apiKey: process.env.ANTHROPIC_API_KEY }),
  maxTurns: 8,
})

await agent.run({ prompt: 'Create hello.txt containing "hi".' })
const second = await agent.run({ prompt: 'Append " there" to hello.txt.' })

console.log(second.stats.turns, agent.stats.runs)
await agent.dispose()
~~~
```

Source: [src/eval-agent.ts:116](https://github.com/Tahul/zidane/blob/main/src/eval-agent.ts#L116)

### `createEvalRunReporter`

Function.

Creates a reporter that collects case results, builds the run summary and formats it as a terminal table.

```ts
function createEvalRunReporter(options?: EvalRunReporterOptions): EvalRunReporter
```

**Parameters**

- `options?` (`EvalRunReporterOptions`, default `{}`): Output directory and color setting. All fields are optional.

**Returns** `EvalRunReporter`

Source: [src/eval.ts:319](https://github.com/Tahul/zidane/blob/main/src/eval.ts#L319)

### `createReusableExecutionContext`

Function.

Wrap an execution context so repeated agent runs share one handle until the
caller disposes it. This is useful for eval fixtures that want per-test
Docker/process lifetime while still using `runHeadless()` per turn.

```ts
function createReusableExecutionContext(
  base: ExecutionContext,
): ReusableExecutionContext
```

**Parameters**

- `base` (`ExecutionContext`): Execution context to wrap.

**Returns** `ReusableExecutionContext`

Source: [src/eval-execution.ts:24](https://github.com/Tahul/zidane/blob/main/src/eval-execution.ts#L24)

### `defineEval`

Function.

Register an eval definition. Eval files call this at module load; the harness
(`buildRegisteredEvals`) materializes them with a provider at run time. The
factory keeps definitions lazy so a single registry serves any provider/judge
pairing without re-importing eval files.

Returns the same factory so a file can also `export default` it for direct,
registry-free use (e.g. hermetic unit tests).

```ts
function defineEval(define: EvalDefinition): EvalDefinition
```

**Parameters**

- `define` (`EvalDefinition`): Function that builds the eval case from the provider and judge.

**Returns** `EvalDefinition`

**Example**

```ts
~~~ts
import { defineEval, fileExists, statusCompleted } from 'zidane/eval'

export default defineEval(({ provider, model }) => ({
  id: 'creates-readme',
  suite: 'files',
  prompt: 'Create README.md with a one-line project description.',
  provider,
  model,
  workspace: { capture: ['.'] },
  scorers: [statusCompleted(), fileExists('README.md')],
}))
~~~
```

Source: [src/eval-registry.ts:49](https://github.com/Tahul/zidane/blob/main/src/eval-registry.ts#L49)

### `defineMetrics`

Function.

Declare an eval's metric set. Returns the same map, typed; pass it to
`EvalCaseOptions.metrics`. Declared metrics that are not emitted are recorded
as missing with a normalized score of 0; emitting an undeclared metric is an
authoring error and throws.

```ts
function defineMetrics<T extends MetricSpecMap>(spec: T): T
```

**Type parameters**

- `T` (`extends MetricSpecMap`)

**Parameters**

- `spec` (`T`): Metric specs keyed by metric id.

**Returns** `T`

Source: [src/eval-metrics.ts:60](https://github.com/Tahul/zidane/blob/main/src/eval-metrics.ts#L60)

### `efficiencyMetricValues`

Function.

Computes the raw values of the [`EFFICIENCY_METRICS`](https://zidane.si/docs/reference/eval.md#EFFICIENCY_METRICS) for a run: wall-clock time, input plus output tokens, and the share of input read from cache.

```ts
function efficiencyMetricValues(
  result: HeadlessResult,
): Record<keyof typeof EFFICIENCY_METRICS, number>
```

**Parameters**

- `result` (`HeadlessResult`): Result of the run.

**Returns** `Record<keyof typeof EFFICIENCY_METRICS, number>`

Source: [src/eval-metrics.ts:109](https://github.com/Tahul/zidane/blob/main/src/eval-metrics.ts#L109)

### `emitEfficiencyMetrics`

Function.

Emits every efficiency metric for a run through `emit`.

```ts
function emitEfficiencyMetrics(emit: MetricEmitter, result: HeadlessResult): void
```

**Parameters**

- `emit` (`MetricEmitter`): Callback that receives each metric id and raw value.
- `result` (`HeadlessResult`): Result of the run.

Source: [src/eval-metrics.ts:127](https://github.com/Tahul/zidane/blob/main/src/eval-metrics.ts#L127)

### `fileContains`

Function.

Creates a scorer that passes when the captured content of `path` contains `expected`. A missing file fails.

```ts
function fileContains(
  path: string,
  expected: string | RegExp,
  name?: string,
): EvalScorer
```

**Parameters**

- `path` (`string`): Path relative to the workspace root.
- `expected` (`string | RegExp`): Substring, or a pattern the content must match.
- `name?` (`string`): Scorer name. Defaults to `file.contains:<path>`.

**Returns** `EvalScorer`

Source: [src/eval-scorers.ts:61](https://github.com/Tahul/zidane/blob/main/src/eval-scorers.ts#L61)

### `fileContentQuality`

Function.

Creates a scorer that grades how closely a file matches `expected`: 1 for an exact match after trimming, 0.75 when the trimmed content contains `expected`, 0.5 when the raw content does, otherwise 0. It passes at 0.75 or more.

```ts
function fileContentQuality(path: string, expected: string, name?: string): EvalScorer
```

**Parameters**

- `path` (`string`): Path relative to the workspace root.
- `expected` (`string`): Text the file should match.
- `name?` (`string`): Scorer name. Defaults to `file.quality:<path>`.

**Returns** `EvalScorer`

Source: [src/eval-scorers.ts:86](https://github.com/Tahul/zidane/blob/main/src/eval-scorers.ts#L86)

### `fileExists`

Function.

Creates a scorer that passes when the workspace snapshot contains `path`.

```ts
function fileExists(path: string, name?: string): EvalScorer
```

**Parameters**

- `path` (`string`): Path relative to the workspace root.
- `name?` (`string`): Scorer name. Defaults to `file.exists:<path>`.

**Returns** `EvalScorer`

Source: [src/eval-scorers.ts:24](https://github.com/Tahul/zidane/blob/main/src/eval-scorers.ts#L24)

### `fileExistsOneOf`

Function.

Creates a scorer that passes when the workspace snapshot contains at least one of `paths`.

```ts
function fileExistsOneOf(paths: string[], name?: string): EvalScorer
```

**Parameters**

- `paths` (`string[]`): Paths relative to the workspace root.
- `name?` (`string`): Scorer name. Defaults to `file.existsOneOf:<paths joined by |>`.

**Returns** `EvalScorer`

Source: [src/eval-scorers.ts:42](https://github.com/Tahul/zidane/blob/main/src/eval-scorers.ts#L42)

### `finalizeEvalMetrics`

Function.

Builds the normalized metrics of a case. A declared metric that was never emitted is recorded as missing with a score of `0`.

```ts
function finalizeEvalMetrics(
  specs: MetricSpecMap,
  raw: Map<string, number>,
): EvalMetric[]
```

**Parameters**

- `specs` (`MetricSpecMap`): Declared metric specs.
- `raw` (`Map<string, number>`): Raw values emitted during the run, keyed by metric id.

**Returns** `EvalMetric[]`

Source: [src/eval-metrics.ts:139](https://github.com/Tahul/zidane/blob/main/src/eval-metrics.ts#L139)

### `formatEvalCaseSummary`

Function.

Formats one case result as plain text: status, score, duration, usage, workspace and artifact info, and per-scorer results.

```ts
function formatEvalCaseSummary(result: EvalCaseResult): string
```

**Parameters**

- `result` (`EvalCaseResult`): Case result to format.

**Returns** `string`

Source: [src/eval-report.ts:14](https://github.com/Tahul/zidane/blob/main/src/eval-report.ts#L14)

### `formatEvalRunSummary`

Function.

Formats an eval run as a terminal table, with a comparison matrix when the cases have more than one variant.

```ts
function formatEvalRunSummary(results: readonly EvalCaseResult[]): string
```

**Parameters**

- `results` (`readonly EvalCaseResult[]`): Case results in any order.

**Returns** `string`

Source: [src/eval-report.ts:57](https://github.com/Tahul/zidane/blob/main/src/eval-report.ts#L57)

### `formatTrajectoryLine`

Function.

Render a trajectory as a one-line timeline string (no color).

```ts
function formatTrajectoryLine(trajectory: Trajectory): string
```

**Parameters**

- `trajectory` (`Trajectory`): Trajectory to render.

**Returns** `string`

Source: [src/eval-trajectory.ts:93](https://github.com/Tahul/zidane/blob/main/src/eval-trajectory.ts#L93)

### `functionalityMetric`

Function.

Aggregate a group of boolean scorers into a single `0..1` functionality
metric (fraction passed) and emit it. Useful for "N of M files present".

```ts
function functionalityMetric(
  metricId: string,
  scorers: EvalScorer[],
  name?: string,
): EvalScorer
```

**Parameters**

- `metricId` (`string`): Declared metric id that receives the fraction.
- `scorers` (`EvalScorer[]`): Boolean scorers to run and count.
- `name?` (`string`, default `metricId`): Scorer name. Defaults to `metricId`.

**Returns** `EvalScorer`

Source: [src/eval-scorers.ts:130](https://github.com/Tahul/zidane/blob/main/src/eval-scorers.ts#L130)

### `llmJudge`

Function.

Creates a scorer that asks a model to grade the run against a rubric. The judge answers through a forced `submit_grade` tool call with a score from 0 to 1 and a short reasoning. The scorer passes at a score of 0.7 or more.

```ts
function llmJudge(options: LlmJudgeOptions): EvalScorer
```

**Parameters**

- `options` (`LlmJudgeOptions`): Judge provider, rubric and optional model, prompt and limits.

**Returns** `EvalScorer`

Source: [src/eval-judge.ts:67](https://github.com/Tahul/zidane/blob/main/src/eval-judge.ts#L67)

### `normalizeMetric`

Function.

Maps a raw value to a score from `0` to `1`, clamped to the spec's range and flipped for `'lower-is-better'`. Returns `1` when `min` equals `max`.

```ts
function normalizeMetric(raw: number, spec: MetricSpec): number
```

**Parameters**

- `raw` (`number`): Raw metric value.
- `spec` (`MetricSpec`): Range and direction of the metric.

**Returns** `number`

Source: [src/eval-metrics.ts:77](https://github.com/Tahul/zidane/blob/main/src/eval-metrics.ts#L77)

### `registerEvalTests`

Function.

Register one test per eval case against a test runner, funnel every result
into a shared reporter, and print the aggregated summary once after all cases.

A failing test means the agent run itself broke (provider/tool/timeout). Low
scores are reported, not asserted — see `RegisterEvalTestsOptions.failOnIncomplete`.

```ts
function registerEvalTests(options: RegisterEvalTestsOptions): EvalRunReporter
```

**Parameters**

- `options` (`RegisterEvalTestsOptions`): Cases, test runner and optional reporter, artifact directory, repeat count and concurrency.

**Returns** `EvalRunReporter`: The reporter that receives every case result.

**Example**

```ts
~~~ts
import { afterAll, it } from 'bun:test'
import { fileExists, registerEvalTests, statusCompleted } from 'zidane/eval'
import { anthropic } from 'zidane/providers/anthropic'

registerEvalTests({
  runner: { it, afterAll },
  artifactDir: 'eval-artifacts',
  cases: [{
    id: 'creates-readme',
    suite: 'files',
    prompt: 'Create README.md with a one-line project description.',
    provider: anthropic({ apiKey: process.env.ANTHROPIC_API_KEY }),
    workspace: { capture: ['.'] },
    scorers: [statusCompleted(), fileExists('README.md')],
  }],
})
~~~
```

Source: [src/eval.ts:229](https://github.com/Tahul/zidane/blob/main/src/eval.ts#L229)

### `relativeArtifactPath`

Function.

Returns `path` relative to `root`.

```ts
function relativeArtifactPath(root: string, path: string): string
```

**Parameters**

- `root` (`string`): Root artifact directory.
- `path` (`string`): Path inside the artifact tree.

**Returns** `string`

Source: [src/eval-artifacts.ts:64](https://github.com/Tahul/zidane/blob/main/src/eval-artifacts.ts#L64)

### `runEvalCase`

Function.

Runs one eval case: prepares the workspace, runs the agent headless, runs the scorers, computes metrics and the trajectory, and writes artifacts when `artifactDir` is set. The workspace is cleaned up when the run ends.

Throws [`EvalMetricError`](https://zidane.si/docs/reference/eval.md#EvalMetricError) when a scorer emits a metric that was not declared or a non-finite value. Other scorer failures are recorded as failed scores.

```ts
function runEvalCase(options: EvalCaseOptions): Promise<EvalCaseResult>
```

**Parameters**

- `options` (`EvalCaseOptions`): Case definition: headless options plus `id`, `scorers`, `metrics` and `workspace`.

**Returns** `Promise<EvalCaseResult>`: The case result. A run that did not complete resolves with `passed` set to `false`.

**Example**

```ts
~~~ts
import { fileExists, runEvalCase, statusCompleted } from 'zidane/eval'
import { anthropic } from 'zidane/providers/anthropic'

const result = await runEvalCase({
  id: 'creates-readme',
  suite: 'files',
  prompt: 'Create README.md with a one-line project description.',
  provider: anthropic({ apiKey: process.env.ANTHROPIC_API_KEY }),
  maxTurns: 8,
  workspace: { capture: ['.'] },
  scorers: [statusCompleted(), fileExists('README.md')],
})

console.log(result.passed, result.score)
~~~
```

Source: [src/eval.ts:58](https://github.com/Tahul/zidane/blob/main/src/eval.ts#L58)

### `statusCompleted`

Function.

Creates a scorer that passes when the run status is `'completed'`.

```ts
function statusCompleted(name?: string): EvalScorer
```

**Parameters**

- `name?` (`string`, default `'status.completed'`): Scorer name. Defaults to `'status.completed'`.

**Returns** `EvalScorer`

Source: [src/eval-scorers.ts:9](https://github.com/Tahul/zidane/blob/main/src/eval-scorers.ts#L9)

## Classes

### `EvalMetricError`

Class.

Thrown by `runEvalCase` when a scorer emits a metric id that was not declared
in `defineMetrics`. This is an authoring error (a typo'd or stale metric id),
not a low score — it fails the eval test rather than being recorded as a
failed scorer. Exported so downstream harnesses can type-narrow on it.

```ts
class EvalMetricError extends Error
```

**Constructor**

```ts
new EvalMetricError(message: string)
```

Creates the error.

**Parameters**

- `message` (`string`): Error message.

**Returns** `EvalMetricError`

Source: [src/eval-metrics.ts:92](https://github.com/Tahul/zidane/blob/main/src/eval-metrics.ts#L92)

## Constants

### `EFFICIENCY_METRICS`

Constant.

Always-available efficiency metrics derived from `HeadlessResult`.

```ts
const EFFICIENCY_METRICS: { 'cache-read-rate': { description: string; direction: 'higher-is-better'; max: number; min: number; tags: string[] }; 'execution-time': { description: string; direction: 'lower-is-better'; max: number; min: number; tags: string[] }; 'provider-tokens': { description: string; direction: 'lower-is-better'; max: number; min: number; tags: string[] } }
```

Source: [src/eval-metrics.ts:65](https://github.com/Tahul/zidane/blob/main/src/eval-metrics.ts#L65)

## Interfaces

### `CreateEvalAgentOptions`

Interface.

Options for [`createEvalAgent`](https://zidane.si/docs/reference/eval.md#createEvalAgent). The [`HeadlessOptions`](https://zidane.si/docs/reference/headless.md#HeadlessOptions) fields set here are the defaults for every run.

```ts
interface CreateEvalAgentOptions extends Omit<HeadlessOptions, 'prompt' | 'provider' | 'execution' | 'cwd' | 'session' | 'store' | 'mcpServers' | 'onEvent' | 'signal'>
```

**Properties**

- `behavior?` (`AgentBehavior`, from `Omit`): Agent behavior defaults for this headless run. Top-level shortcuts below override matching fields.
- `cwd?` (`string`): Working directory for the default process context. Ignored when
  `execution` is supplied.
- `durable?` (`DurableRuntime | DurableOptions`, from `Omit`): Durable runtime the session is stored on (`createAgent({ durable })`), or it with `onInterrupted`: runs keep checkpoints.
- `execution?` (`ExecutionContext`): Execution context for agent tools. Wrapped with
  `createReusableExecutionContext` so runs share one handle until `dispose`.
- `extensions?` (`ExtensionRegistry`, from `Omit`): Pre-built extension registry folded into the agent via `createAgent`
  (build it with `buildExtensionRegistry(defs.map(inlineLoadedExtension))`).
  Contributes the registry's tools / hooks / mcpServers / skills /
  system-fragments / subagents, with the explicit options above winning on
  collision. Because hooks fold through `createAgent`, a headless run with
  `extensions` also gets extension hooks (e.g. a `tool:transform` that
  rewrites results), not just tools.
- `includeThinking?` (`boolean`, from `Omit`): Keep `thinking` blocks in `result.transcript` (default true).
- `maxTokens?` (`number`, from `Omit`): Maximum output tokens per turn. Sets `behavior.maxTokens`.
- `maxTurns?` (`number`, from `Omit`): Maximum number of loop turns. Sets `behavior.maxTurns`.
- `mcpServers?` (`EvalAgentMcpServers`): Static or lazily resolved MCP servers. Resolved fresh before each run.
- `model?` (`string`, from `Omit`): Model id. Defaults to the provider's default model.
- `onEvent?` (`(event: HeadlessEvent) => void`): Live event callback for every run.
- `preset?` (`Preset`, from `Omit`): Base [`Preset`](https://zidane.si/docs/reference/presets.md#Preset) for the agent — spread in place of the default `basic`
  preset (use it to swap the system prompt / tool set wholesale). The
  top-level shortcuts above (`system`, `behavior`, `tools`, `mcpServers`,
  `skills`) still override matching preset fields.
- `provider` (`Provider`): Provider that drives the agent.
- `resumeRunId?` (`string`, from `Omit`): Continue this checkpoint run of `session` (`agent.resume`) before running `prompt`. Needs `durable`.
- `schema?` (`Record<string, unknown>`, from `Omit`): JSON Schema for structured-output enforcement → populates `result.output`.
- `session?` (`Session`): Session to reuse across runs. When omitted, one in-memory session is created.
- `skills?` (`SkillsConfig`, from `Omit`): Skills configuration.
- `store?` (`SessionStore`): Store used for the auto-created session. Defaults to an in-memory store.
- `subagents?` (`SubagentRegistry`, from `Omit`): Subagents folded into the preset's spawn tool, filling gaps only (an
  agent the preset or an extension already defines wins). The CLI passes
  builtin + markdown agents from `loadMarkdownAgents`.
- `system?` (`string`, from `Omit`): Override the preset system prompt for this run.
- `thinking?` (`'off' | 'minimal' | 'low' | 'medium' | 'high' | 'xhigh' | 'max' | 'adaptive'`, from `Omit`): Thinking effort for the run. Defaults to `'off'`.
- `timeoutMs?` (`number`, from `Omit`): Wall-clock cap; on expiry the run is aborted and `status` becomes `'timeout'`.
- `tools?` (`Record<string, ToolDef>`, from `Omit`): Tool overrides. Omit to use the basic preset's tools.

Source: [src/eval-agent.ts:50](https://github.com/Tahul/zidane/blob/main/src/eval-agent.ts#L50)

### `EvalAgent`

Interface.

Multi-turn agent for evals. Every run shares one session and one execution handle.

```ts
interface EvalAgent
```

**Properties**

- `[asyncDispose]` (`() => Promise<void>`): Same as `dispose`, for `await using`.
- `dispose` (`() => Promise<void>`): Destroys the shared execution handle. Safe to call more than once.
- `run` (`(options: EvalAgentRunOptions) => Promise<EvalAgentRunResult>`): Runs one prompt in the shared session. Throws when a run is already in progress or the agent is disposed.
- `session` (`() => Promise<Session>`): Returns the shared session, creating it on the first call. Throws after `dispose`.
- `stats` (`EvalAgentStats`, readonly): Totals across the runs so far.

Source: [src/eval-agent.ts:79](https://github.com/Tahul/zidane/blob/main/src/eval-agent.ts#L79)

### `EvalAgentRunOptions`

Interface.

Options for one `EvalAgent.run` call. Overrides the defaults set in [`createEvalAgent`](https://zidane.si/docs/reference/eval.md#createEvalAgent); `mcpServers` are added to the agent's own.

```ts
interface EvalAgentRunOptions extends Omit<HeadlessOptions, 'provider' | 'execution' | 'cwd' | 'session' | 'store'>
```

**Properties**

- `behavior?` (`AgentBehavior`, from `Omit`): Agent behavior defaults for this headless run. Top-level shortcuts below override matching fields.
- `durable?` (`DurableRuntime | DurableOptions`, from `Omit`): Durable runtime the session is stored on (`createAgent({ durable })`), or it with `onInterrupted`: runs keep checkpoints.
- `extensions?` (`ExtensionRegistry`, from `Omit`): Pre-built extension registry folded into the agent via `createAgent`
  (build it with `buildExtensionRegistry(defs.map(inlineLoadedExtension))`).
  Contributes the registry's tools / hooks / mcpServers / skills /
  system-fragments / subagents, with the explicit options above winning on
  collision. Because hooks fold through `createAgent`, a headless run with
  `extensions` also gets extension hooks (e.g. a `tool:transform` that
  rewrites results), not just tools.
- `includeThinking?` (`boolean`, from `Omit`): Keep `thinking` blocks in `result.transcript` (default true).
- `maxTokens?` (`number`, from `Omit`): Maximum output tokens per turn. Sets `behavior.maxTokens`.
- `maxTurns?` (`number`, from `Omit`): Maximum number of loop turns. Sets `behavior.maxTurns`.
- `mcpServers?` (`McpServerConfig[]`, from `Omit`): MCP servers to connect and expose as tools.
- `model?` (`string`, from `Omit`): Model id. Defaults to the provider's default model.
- `onEvent?` (`(event: HeadlessEvent) => void`, from `Omit`): Live event callback — the in-process stream-json equivalent.
- `preset?` (`Preset`, from `Omit`): Base [`Preset`](https://zidane.si/docs/reference/presets.md#Preset) for the agent — spread in place of the default `basic`
  preset (use it to swap the system prompt / tool set wholesale). The
  top-level shortcuts above (`system`, `behavior`, `tools`, `mcpServers`,
  `skills`) still override matching preset fields.
- `prompt` (`string | PromptPart[]`): User prompt — plain string or multimodal `PromptPart[]`.
- `resumeRunId?` (`string`, from `Omit`): Continue this checkpoint run of `session` (`agent.resume`) before running `prompt`. Needs `durable`.
- `schema?` (`Record<string, unknown>`, from `Omit`): JSON Schema for structured-output enforcement → populates `result.output`.
- `signal?` (`AbortSignal`, from `Omit`): External abort signal — chained with the internal timeout controller.
- `skills?` (`SkillsConfig`, from `Omit`): Skills configuration.
- `subagents?` (`SubagentRegistry`, from `Omit`): Subagents folded into the preset's spawn tool, filling gaps only (an
  agent the preset or an extension already defines wins). The CLI passes
  builtin + markdown agents from `loadMarkdownAgents`.
- `system?` (`string`, from `Omit`): Override the preset system prompt for this run.
- `thinking?` (`'off' | 'minimal' | 'low' | 'medium' | 'high' | 'xhigh' | 'max' | 'adaptive'`, from `Omit`): Thinking effort for the run. Defaults to `'off'`.
- `timeoutMs?` (`number`, from `Omit`): Wall-clock cap; on expiry the run is aborted and `status` becomes `'timeout'`.
- `tools?` (`Record<string, ToolDef>`, from `Omit`): Tool overrides. Omit to use the basic preset's tools.

Source: [src/eval-agent.ts:74](https://github.com/Tahul/zidane/blob/main/src/eval-agent.ts#L74)

### `EvalAgentRunResult`

Interface.

Result of one [`EvalAgent`](https://zidane.si/docs/reference/eval.md#EvalAgent) run.

```ts
interface EvalAgentRunResult
```

**Properties**

- `newTranscript` (`SessionTurn[]`): Transcript turns added by this run (same view as `result.transcript`).
- `result` (`HeadlessResult`): Full headless result. Its transcript is scoped to this run.
- `stats` (`EvalAgentRunStats`): Per-run usage and trajectory counters.

Source: [src/eval-agent.ts:35](https://github.com/Tahul/zidane/blob/main/src/eval-agent.ts#L35)

### `EvalAgentRunStats`

Interface.

Usage and counters of one [`EvalAgent`](https://zidane.si/docs/reference/eval.md#EvalAgent) run.

```ts
interface EvalAgentRunStats extends EvalRunUsage
```

**Properties**

- `cacheCreation` (`number`, from `EvalRunUsage`): Tokens written to the prompt cache.
- `cacheRead` (`number`, from `EvalRunUsage`): Tokens read from the prompt cache.
- `cost` (`number`, from `EvalRunUsage`): Cost in USD.
- `durationMs` (`number`): Wall-clock duration of the run in milliseconds.
- `input` (`number`, from `EvalRunUsage`): Input tokens, excluding cache tokens.
- `output` (`number`, from `EvalRunUsage`): Output tokens.
- `toolCalls` (`number`): Number of tool calls in the run's transcript.
- `turns` (`number`): Number of turns of the run.

Source: [src/eval-agent.ts:13](https://github.com/Tahul/zidane/blob/main/src/eval-agent.ts#L13)

### `EvalAgentStats`

Interface.

Totals across every run of an [`EvalAgent`](https://zidane.si/docs/reference/eval.md#EvalAgent).

```ts
interface EvalAgentStats extends EvalRunUsage
```

**Properties**

- `cacheCreation` (`number`, from `EvalRunUsage`): Tokens written to the prompt cache.
- `cacheRead` (`number`, from `EvalRunUsage`): Tokens read from the prompt cache.
- `cost` (`number`, from `EvalRunUsage`): Cost in USD.
- `durationMs` (`number`): Sum of per-run wall-clock durations.
- `input` (`number`, from `EvalRunUsage`): Input tokens, excluding cache tokens.
- `output` (`number`, from `EvalRunUsage`): Output tokens.
- `runs` (`number`): Number of completed `run()` calls.
- `toolCalls` (`number`): Sum of per-run tool calls.
- `turns` (`number`): Sum of per-run agent turns.

Source: [src/eval-agent.ts:23](https://github.com/Tahul/zidane/blob/main/src/eval-agent.ts#L23)

### `EvalArtifacts`

Interface.

Paths of the artifact files written for one case.

```ts
interface EvalArtifacts
```

**Properties**

- `dir` (`string`): Directory that holds the case's artifacts.
- `events?` (`string`): Path of `events.jsonl`.
- `result?` (`string`): Path of `result.json`.
- `transcript?` (`string`): Path of `transcript.json`.
- `workspace?` (`string`): Path of `workspace.json`. Only written when a workspace snapshot exists.

Source: [src/eval-types.ts:91](https://github.com/Tahul/zidane/blob/main/src/eval-types.ts#L91)

### `EvalCaseOptions`

Interface.

Definition of one eval case. Extends [`HeadlessOptions`](https://zidane.si/docs/reference/headless.md#HeadlessOptions) with an id, scorers, metrics and workspace setup, and is run by [`runEvalCase`](https://zidane.si/docs/reference/eval.md#runEvalCase).

```ts
interface EvalCaseOptions extends Omit<HeadlessOptions, 'onEvent'>
```

**Properties**

- `artifactDir?` (`string`): Directory where artifacts of this case are written. Nothing is written when omitted.
- `behavior?` (`AgentBehavior`, from `Omit`): Agent behavior defaults for this headless run. Top-level shortcuts below override matching fields.
- `cwd?` (`string`, from `Omit`): Working directory for the default process context (ignored if `execution` is set).
- `durable?` (`DurableRuntime | DurableOptions`, from `Omit`): Durable runtime the session is stored on (`createAgent({ durable })`), or it with `onInterrupted`: runs keep checkpoints.
- `execution?` (`ExecutionContext`, from `Omit`): Execution context. Defaults to a process context rooted at `cwd`.
- `extensions?` (`ExtensionRegistry`, from `Omit`): Pre-built extension registry folded into the agent via `createAgent`
  (build it with `buildExtensionRegistry(defs.map(inlineLoadedExtension))`).
  Contributes the registry's tools / hooks / mcpServers / skills /
  system-fragments / subagents, with the explicit options above winning on
  collision. Because hooks fold through `createAgent`, a headless run with
  `extensions` also gets extension hooks (e.g. a `tool:transform` that
  rewrites results), not just tools.
- `id` (`string`): Case id, unique within its suite.
- `includeThinking?` (`boolean`, from `Omit`): Keep `thinking` blocks in `result.transcript` (default true).
- `maxTokens?` (`number`, from `Omit`): Maximum output tokens per turn. Sets `behavior.maxTokens`.
- `maxTurns?` (`number`, from `Omit`): Maximum number of loop turns. Sets `behavior.maxTurns`.
- `mcpServers?` (`McpServerConfig[]`, from `Omit`): MCP servers to connect and expose as tools.
- `metrics?` (`MetricSpecMap`): Declared metric set. Auto-merged with `EFFICIENCY_METRICS` (`./eval-metrics`) so every
  eval gets comparable efficiency metrics for free. A declared metric that is
  never emitted (via auto-emit or `ctx.metric`) is recorded as missing with a
  score of `0`.
- `model?` (`string`, from `Omit`): Model id. Defaults to the provider's default model.
- `onEvent?` (`(event: HeadlessEvent) => void`): Live event callback for the run.
- `preset?` (`Preset`, from `Omit`): Base [`Preset`](https://zidane.si/docs/reference/presets.md#Preset) for the agent — spread in place of the default `basic`
  preset (use it to swap the system prompt / tool set wholesale). The
  top-level shortcuts above (`system`, `behavior`, `tools`, `mcpServers`,
  `skills`) still override matching preset fields.
- `prompt` (`string | PromptPart[]`, from `Omit`): User prompt — plain string or multimodal `PromptPart[]`.
- `provider` (`Provider`, from `Omit`): Built provider (e.g. `local()`, `openaiCompat(...)`, `anthropic(...)`).
- `resumeRunId?` (`string`, from `Omit`): Continue this checkpoint run of `session` (`agent.resume`) before running `prompt`. Needs `durable`.
- `schema?` (`Record<string, unknown>`, from `Omit`): JSON Schema for structured-output enforcement → populates `result.output`.
- `scorers?` (`EvalScorer[]`, default `[]`): Scorers run after the agent finishes.
- `session?` (`Session`, from `Omit`): Reuse / resume an existing session.
- `signal?` (`AbortSignal`, from `Omit`): External abort signal — chained with the internal timeout controller.
- `skills?` (`SkillsConfig`, from `Omit`): Skills configuration.
- `sourceFile?` (`string`): Absolute path to the eval definition file. Set via `import.meta.url`.
- `store?` (`SessionStore`, from `Omit`): Session store for a fresh session (defaults to an in-memory store).
- `subagents?` (`SubagentRegistry`, from `Omit`): Subagents folded into the preset's spawn tool, filling gaps only (an
  agent the preset or an extension already defines wins). The CLI passes
  builtin + markdown agents from `loadMarkdownAgents`.
- `suite?` (`string`): Suite the case belongs to. Used in artifact paths and reports.
- `system?` (`string`, from `Omit`): Override the preset system prompt for this run.
- `tags?` (`string[]`, default `[]`): Tags for grouping and filtering cases.
- `thinking?` (`'off' | 'minimal' | 'low' | 'medium' | 'high' | 'xhigh' | 'max' | 'adaptive'`, from `Omit`): Thinking effort for the run. Defaults to `'off'`.
- `timeoutMs?` (`number`, from `Omit`): Wall-clock cap; on expiry the run is aborted and `status` becomes `'timeout'`.
- `tools?` (`Record<string, ToolDef>`, from `Omit`): Tool overrides. Omit to use the basic preset's tools.
- `variant?` (`string`): Run variant label when the same eval runs against several provider/model
  targets in one run (e.g. `baseten:zai-org/GLM-5.1`). Flows into
  `EvalCaseResult.variant`, namespaces artifacts, and drives the
  reporter's per-variant tables + comparison matrix.
- `workspace?` (`EvalWorkspaceOptions`): Workspace setup for the case.

Source: [src/eval-types.ts:141](https://github.com/Tahul/zidane/blob/main/src/eval-types.ts#L141)

### `EvalCaseResult`

Interface.

Outcome of one eval case: the headless result plus scores, metrics, trajectory and artifacts.

```ts
interface EvalCaseResult
```

**Properties**

- `artifacts?` (`EvalArtifacts`): Artifact files, when an artifact directory was set.
- `events` (`HeadlessEvent[]`): Events emitted during the run, in order.
- `id` (`string`): Case id.
- `metrics` (`EvalMetric[]`): Normalized metrics for this case (empty when no metrics declared).
- `passed` (`boolean`): True when every scorer passed.
- `result` (`HeadlessResult`): Result of the headless run.
- `score` (`number`): Mean normalized metric score in \[0, 1] when metrics are declared; otherwise mean scorer score.
- `scores` (`EvalScore[]`): Result of each scorer.
- `sourceFile?` (`string`): Absolute path to the eval definition file, for editor links.
- `suite?` (`string`): Suite the case belongs to.
- `tags` (`string[]`): Tags of the case.
- `tagScores` (`Record<string, number>`): Mean normalized score per tag, e.g. `{ '@efficiency': 0.8 }`.
- `trajectory` (`Trajectory`): Ordered, grouped step timeline derived from the transcript.
- `variant?` (`string`): Run variant this result belongs to (e.g. `baseten:zai-org/GLM-5.1`).
- `workspace?` (`EvalWorkspaceSnapshot`): Files captured from the workspace, when the case set up one.
- `workspaceError?` (`string`): Error raised while capturing the workspace, when there was one.

Source: [src/eval-types.ts:105](https://github.com/Tahul/zidane/blob/main/src/eval-types.ts#L105)

### `EvalDefinitionContext`

Interface.

Context handed to an eval definition factory at registration time.

```ts
interface EvalDefinitionContext
```

**Properties**

- `judge` (`Provider`): Provider used by `llmJudge` scorers. Defaults to `provider`.
- `judgeModel?` (`string`): Optional judge model override; evals should pass it to `llmJudge({ model })`.
- `model?` (`string`): Optional agent model override; evals should pass it to `EvalCaseOptions.model`.
- `provider` (`Provider`): Agent-under-test provider.

Source: [src/eval-types.ts:318](https://github.com/Tahul/zidane/blob/main/src/eval-types.ts#L318)

### `EvalMetric`

Interface.

A normalized metric value attached to one eval case.

```ts
interface EvalMetric
```

**Properties**

- `description?` (`string`): Human-readable description, from the spec.
- `direction` (`MetricDirection`): Whether higher or lower raw values score better.
- `id` (`string`): Metric id, the key in the [`MetricSpecMap`](https://zidane.si/docs/reference/eval.md#MetricSpecMap).
- `max` (`number`): Upper bound of the raw range, from the spec.
- `min` (`number`): Lower bound of the raw range, from the spec.
- `missing?` (`boolean`): True when the metric was declared but never emitted (e.g. its scorer threw
  before calling `ctx.metric`). Recorded as `normalized: 0` so the run still
  produces results instead of crashing.
- `normalized` (`number`): Score from `0` to `1`. `0` when the metric is missing.
- `raw` (`number`): Raw value the run emitted. `0` when the metric is missing.
- `tags` (`string[]`): Grouping tags, from the spec. Empty when the spec has none.

Source: [src/eval-metrics.ts:24](https://github.com/Tahul/zidane/blob/main/src/eval-metrics.ts#L24)

### `EvalRunMetricAggregate`

Interface.

Statistics of one metric across every case that emitted it.

```ts
interface EvalRunMetricAggregate
```

**Properties**

- `direction` (`MetricDirection`): Whether higher or lower raw values score better.
- `id` (`string`): Metric id.
- `normalized` (`MetricStats`): Statistics of the normalized scores.
- `raw` (`MetricStats`): Statistics of the raw values.
- `tags` (`string[]`): Grouping tags of the metric.

Source: [src/eval-types.ts:239](https://github.com/Tahul/zidane/blob/main/src/eval-types.ts#L239)

### `EvalRunReporter`

Interface.

Collects case results during a run and produces the run summary.

```ts
interface EvalRunReporter
```

**Properties**

- `flush` (`() => Promise<EvalRunSummary>`): Builds the run summary. Writes `run-summary.json` when `outputDir` is set.
- `format` (`() => string`): Formats the recorded results as a terminal table.
- `record` (`(result: EvalCaseResult) => Promise<void>`): Records one case result. Writes its JSON file when `outputDir` is set.
- `results` (`readonly EvalCaseResult[]`, readonly): Results recorded so far.

Source: [src/eval-types.ts:306](https://github.com/Tahul/zidane/blob/main/src/eval-types.ts#L306)

### `EvalRunReporterOptions`

Interface.

Options for [`createEvalRunReporter`](https://zidane.si/docs/reference/eval.md#createEvalRunReporter).

```ts
interface EvalRunReporterOptions
```

**Properties**

- `color?` (`boolean`): Enable ANSI colors. Defaults to TTY-aware auto color.
- `outputDir?` (`string`): Directory where per-case results and `run-summary.json` are written.

Source: [src/eval-types.ts:298](https://github.com/Tahul/zidane/blob/main/src/eval-types.ts#L298)

### `EvalRunSummary`

Interface.

Summary of an eval run, built by [`buildEvalRunSummary`](https://zidane.si/docs/reference/eval.md#buildEvalRunSummary).

```ts
interface EvalRunSummary
```

**Properties**

- `cases` (`EvalRunSummaryCase[]`): One row per case, sorted by variant, then suite and id.
- `count` (`number`): Number of cases.
- `durationMs` (`number`): Summed wall-clock duration in milliseconds.
- `metrics` (`EvalRunMetricAggregate[]`): Per-metric stats across every case that emitted the metric.
- `passed` (`number`): Number of cases that passed.
- `score` (`number`): Mean case score.
- `tagScores` (`Record<string, number>`): Mean normalized score per tag across all cases.
- `usage` (`EvalRunUsage`): Summed usage.
- `variants?` (`EvalVariantSummary[]`): Per-variant rollups, present when any case carries a variant label.

Source: [src/eval-types.ts:276](https://github.com/Tahul/zidane/blob/main/src/eval-types.ts#L276)

### `EvalRunSummaryCase`

Interface.

One row of `EvalRunSummary.cases`.

```ts
interface EvalRunSummaryCase
```

**Properties**

- `durationMs` (`number`): Wall-clock duration of the run in milliseconds.
- `id` (`string`): Case id.
- `metrics` (`EvalMetric[]`): Normalized metrics of the case.
- `passed` (`boolean`): Whether the case passed.
- `score` (`number`): Case score from `0` to `1`.
- `scores` (`EvalScore[]`): Result of each scorer.
- `status` (`HeadlessStatus`): How the headless run ended.
- `suite?` (`string`): Suite the case belongs to.
- `tagScores` (`Record<string, number>`): Mean normalized score per tag.
- `trajectory` (`Trajectory`): Grouped step timeline of the run.
- `variant?` (`string`): Run variant the case ran under.

Source: [src/eval-types.ts:195](https://github.com/Tahul/zidane/blob/main/src/eval-types.ts#L195)

### `EvalRunUsage`

Interface.

Token and cost totals.

```ts
interface EvalRunUsage
```

**Properties**

- `cacheCreation` (`number`): Tokens written to the prompt cache.
- `cacheRead` (`number`): Tokens read from the prompt cache.
- `cost` (`number`): Cost in USD.
- `input` (`number`): Input tokens, excluding cache tokens.
- `output` (`number`): Output tokens.

Source: [src/eval-types.ts:181](https://github.com/Tahul/zidane/blob/main/src/eval-types.ts#L181)

### `EvalScore`

Interface.

Result of one scorer.

```ts
interface EvalScore
```

**Properties**

- `details?` (`string | Record<string, unknown>`): Explanation shown in reports: a message or a structured record.
- `name` (`string`): Scorer name.
- `passed` (`boolean`): Whether the case passes this scorer.
- `score?` (`number`): Score from `0` to `1`. Treated as `1` or `0` from `passed` when absent.

Source: [src/eval-types.ts:56](https://github.com/Tahul/zidane/blob/main/src/eval-types.ts#L56)

### `EvalScorerContext`

Interface.

Everything a scorer can inspect after a run.

```ts
interface EvalScorerContext
```

**Properties**

- `artifactDir?` (`string`): Directory where artifacts of this case are written, when artifacts are enabled.
- `events` (`readonly HeadlessEvent[]`): Events emitted during the run, in order.
- `id` (`string`): Case id.
- `metric` (`MetricEmitter`): Emit a declared metric's raw value. An undeclared id throws an `EvalMetricError`.
- `result` (`HeadlessResult`): Result of the headless run.
- `suite?` (`string`): Suite the case belongs to.
- `tags` (`string[]`): Tags of the case.
- `workspace?` (`EvalWorkspaceSnapshot`): Files captured from the workspace, when the case set up one.

Source: [src/eval-types.ts:68](https://github.com/Tahul/zidane/blob/main/src/eval-types.ts#L68)

### `EvalTestRunner`

Interface.

Minimal subset of a test runner's API the eval harness needs.

```ts
interface EvalTestRunner
```

**Properties**

- `afterAll` (`(fn: () => void | Promise<void>) => void`): Registers a callback that runs once after every test.
- `it` (`(name: string, fn: () => void | Promise<void>) => void`): Registers one test.

Source: [src/eval.ts:163](https://github.com/Tahul/zidane/blob/main/src/eval.ts#L163)

### `EvalVariantSummary`

Interface.

Aggregate rollup for one run variant (provider/model target).

```ts
interface EvalVariantSummary
```

**Properties**

- `count` (`number`): Number of cases.
- `durationMs` (`number`): Summed wall-clock duration in milliseconds.
- `passed` (`number`): Number of cases that passed.
- `score` (`number`): Mean case score.
- `toolCalls` (`number`): Tool calls the model issued, summed. Calls a codemode script makes are
  not counted: they never reach the transcript, which is the point.
- `turns` (`number`): Model turns, summed across the variant's cases.
- `usage` (`EvalRunUsage`): Summed usage.
- `variant` (`string`): Variant label.

Source: [src/eval-types.ts:253](https://github.com/Tahul/zidane/blob/main/src/eval-types.ts#L253)

### `EvalWorkspaceFile`

Interface.

One file captured from the workspace after a run.

```ts
interface EvalWorkspaceFile
```

**Properties**

- `binary?` (`boolean`): `true` when the file looks binary. Its content is not captured.
- `content?` (`string`): Text content, cut to `maxFileChars`. Absent for binary files.
- `path` (`string`): Path relative to the workspace root.
- `size` (`number`): Size of the file in characters.
- `truncated?` (`boolean`): `true` when `content` was cut to `maxFileChars`.

Source: [src/eval-types.ts:34](https://github.com/Tahul/zidane/blob/main/src/eval-types.ts#L34)

### `EvalWorkspaceOptions`

Interface.

Workspace setup for an eval case: where the agent works, what is copied in, and what is captured afterwards.

```ts
interface EvalWorkspaceOptions
```

**Properties**

- `capture?` (`string[]`): Relative files/directories to capture after the run. Defaults to `.`.
- `cwd?` (`string`): Working directory inside the execution context. When omitted, process-based
  evals get a fresh host temp dir; Docker evals use the context default.

  Note: under the shared-Docker harness (`withDocker`), per-case working dirs
  are assigned by the container (`/workspace/case-N`) and a custom `cwd` is
  ignored — `withDocker` strips it so cases stay isolated.
- `maxFileChars?` (`number`): Max text chars retained per captured file. Defaults to 256 KiB.
- `retain?` (`boolean`): Keep a temp process workspace on disk after the eval completes.
- `seedDir?` (`string`): Directory visible inside the execution context, copied into the workspace
  before the agent runs. For Docker, mount local fixtures read-only and point
  this at the container path, e.g. `/fixtures/react-empty`.
- `seedTarget?` (`string`): Relative destination inside the workspace. Defaults to `.`.

Source: [src/eval-types.ts:7](https://github.com/Tahul/zidane/blob/main/src/eval-types.ts#L7)

### `EvalWorkspaceSnapshot`

Interface.

Files captured from the workspace after a run.

```ts
interface EvalWorkspaceSnapshot
```

**Properties**

- `cwd` (`string`): Working directory the agent used.
- `files` (`EvalWorkspaceFile[]`): Captured files.

Source: [src/eval-types.ts:48](https://github.com/Tahul/zidane/blob/main/src/eval-types.ts#L48)

### `LlmJudgeOptions`

Interface.

Options for [`llmJudge`](https://zidane.si/docs/reference/eval.md#llmJudge).

```ts
interface LlmJudgeOptions
```

**Properties**

- `input?` (`(ctx: EvalScorerContext) => string`): Select what the judge sees. Defaults to final text plus captured workspace files.
- `maxTokens?` (`number`): Output budget for the grade call. Default `DEFAULT_JUDGE_MAX_TOKENS`.
  A judge that runs out mid-`submit_grade` loses the whole call (providers
  drop an unfinished tool call), so it is retried once with twice the budget.
- `metric?` (`string`): Declared metric id to emit the judge's `0..1` score into.
- `model?` (`string`): Judge model id. Defaults to the provider's default model.
- `name?` (`string`): Scorer name. Defaults to `'llm.judge'`.
- `provider` (`Provider`): Provider that runs the judge model.
- `rubric` (`string`): Rubric the output is graded against.
- `system?` (`string`): System prompt for the judge. Defaults to a strict grading prompt.
- `timeoutMs?` (`number`): Wall-clock cap for the judge provider call. Defaults to 60s.

Source: [src/eval-judge.ts:5](https://github.com/Tahul/zidane/blob/main/src/eval-judge.ts#L5)

### `MetricSpec`

Interface.

Declared metric: a raw signal plus how to normalize it to a `0..1` score.

```ts
interface MetricSpec
```

**Properties**

- `description?` (`string`): Human-readable description of the metric.
- `direction` (`MetricDirection`): Whether higher or lower raw values score better.
- `max` (`number`): Upper bound of the raw range. Raw values above it are clamped.
- `min` (`number`): Lower bound of the raw range. Raw values below it are clamped.
- `tags?` (`string[]`): Grouping tags, e.g. `@efficiency`, `@functionality`, `@quality`.

Source: [src/eval-metrics.ts:7](https://github.com/Tahul/zidane/blob/main/src/eval-metrics.ts#L7)

### `MetricStats`

Interface.

Descriptive statistics of one metric across cases.

```ts
interface MetricStats
```

**Properties**

- `max` (`number`): Largest value.
- `mean` (`number`): Arithmetic mean.
- `min` (`number`): Smallest value.
- `p50` (`number`): Median.
- `p90` (`number`): 90th percentile.
- `values` (`number[]`): Every value, one per case.
- `zeroCount` (`number`): Number of values equal to `0`.

Source: [src/eval-types.ts:221](https://github.com/Tahul/zidane/blob/main/src/eval-types.ts#L221)

### `RegisterEvalTestsOptions`

Interface.

Options for [`registerEvalTests`](https://zidane.si/docs/reference/eval.md#registerEvalTests).

```ts
interface RegisterEvalTestsOptions
```

**Properties**

- `artifactDir?` (`string`): Directory where per-case artifacts are written. Nothing is written when omitted.
- `cases` (`EvalCaseOptions[]`): Eval cases to register, one test each.
- `concurrency?` (`number`): Max eval cases to run concurrently. Each case keeps its own workspace dir,
  so a shared container runs them in parallel safely. Default 1 (sequential).
- `dispose?` (`() => void | Promise<void>`): Teardown run once after all cases (e.g. dispose a shared container).
- `failOnIncomplete?` (`boolean`): When true (default), each case asserts only that the agent run reached a
  terminal `completed` status — scores are recorded but never fail the test.
  Evals are scoring tools, not deterministic assertions.
- `printSummary?` (`boolean`): Print the aggregated run summary after all cases. Default true.
- `repeat?` (`number`): Repeat each case N times (each repetition is its own test). Default 1.
- `reporter?` (`EvalRunReporter`): Reporter that collects the results. A new one is created when omitted.
- `runner` (`EvalTestRunner`): Test runner API the tests are registered on, for example `{ it, afterAll }` from `bun:test`.

Source: [src/eval.ts:171](https://github.com/Tahul/zidane/blob/main/src/eval.ts#L171)

### `ReusableExecutionContext`

Interface.

Execution context that shares one handle across runs, plus the hooks to inspect and dispose it. Created by [`createReusableExecutionContext`](https://zidane.si/docs/reference/eval.md#createReusableExecutionContext).

```ts
interface ReusableExecutionContext
```

**Properties**

- `dispose` (`() => Promise<void>`): Destroy the underlying handle if it was spawned. Idempotent.
- `execution` (`ExecutionContext`): Execution context facade that reuses one underlying handle. Its
  `destroy()` is intentionally a no-op; call `dispose()` when the surrounding
  fixture/run owns teardown.
- `handle` (`() => ExecutionHandle | undefined`): Current underlying handle, if the agent has spawned one.

Source: [src/eval-execution.ts:4](https://github.com/Tahul/zidane/blob/main/src/eval-execution.ts#L4)

### `Trajectory`

Interface.

Ordered, grouped view of what an agent did during an eval run.

```ts
interface Trajectory
```

**Properties**

- `steps` (`TrajectoryStep[]`): Grouped steps in the order they happened.
- `totalBlocks` (`number`): Total blocks across all steps (sum of `count`).

Source: [src/eval-trajectory.ts:30](https://github.com/Tahul/zidane/blob/main/src/eval-trajectory.ts#L30)

### `TrajectoryStep`

Interface.

One grouped step in an eval trajectory. Consecutive blocks of the same kind
(and, for tools, the same tool name) collapse into a single step with a
`count`. A different kind interrupting the run breaks the group.

Serializable by design so trajectories round-trip to disk and can be diffed
or replayed by other tooling.

```ts
interface TrajectoryStep
```

**Properties**

- `count` (`number`): Number of consecutive occurrences grouped into this step.
- `durationMs?` (`number`): Approx wall-clock ms attributed to this step's turns, when derivable.
- `kind` (`TrajectoryStepKind`): What the grouped blocks are.
- `name?` (`string`): Tool name for `kind: 'tool'` steps (canonical, e.g. `read_file`).

Source: [src/eval-trajectory.ts:18](https://github.com/Tahul/zidane/blob/main/src/eval-trajectory.ts#L18)

## Types

### `EvalAgentMcpServers`

Type.

MCP servers for an [`EvalAgent`](https://zidane.si/docs/reference/eval.md#EvalAgent): a fixed list, or a function that returns the list before each run.

```ts
type EvalAgentMcpServers =
  | readonly McpServerConfig[]
  | (() => readonly McpServerConfig[] | Promise<readonly McpServerConfig[]>)
```

Source: [src/eval-agent.ts:45](https://github.com/Tahul/zidane/blob/main/src/eval-agent.ts#L45)

### `EvalDefinition`

Type.

Factory that builds an eval case from an [`EvalDefinitionContext`](https://zidane.si/docs/reference/eval.md#EvalDefinitionContext). Register one with `defineEval`.

```ts
type EvalDefinition = (ctx: EvalDefinitionContext) => EvalCaseOptions
```

Source: [src/eval-types.ts:330](https://github.com/Tahul/zidane/blob/main/src/eval-types.ts#L330)

### `EvalScorer`

Type.

Function that scores one finished run. Returns an [`EvalScore`](https://zidane.si/docs/reference/eval.md#EvalScore), or a promise of one.

```ts
type EvalScorer = (ctx: EvalScorerContext) => EvalScore | Promise<EvalScore>
```

Source: [src/eval-types.ts:88](https://github.com/Tahul/zidane/blob/main/src/eval-types.ts#L88)

### `MetricDirection`

Type.

Whether a larger raw value is better or worse.

```ts
type MetricDirection = 'higher-is-better' | 'lower-is-better'
```

Source: [src/eval-metrics.ts:4](https://github.com/Tahul/zidane/blob/main/src/eval-metrics.ts#L4)

### `MetricEmitter`

Type.

Emit a raw metric value during a run.

```ts
type MetricEmitter = (id: string, raw: number) => void
```

Source: [src/eval-metrics.ts:50](https://github.com/Tahul/zidane/blob/main/src/eval-metrics.ts#L50)

### `MetricSpecMap`

Type.

Metric specs keyed by metric id.

```ts
type MetricSpecMap = Record<string, MetricSpec>
```

Source: [src/eval-metrics.ts:21](https://github.com/Tahul/zidane/blob/main/src/eval-metrics.ts#L21)

### `TrajectoryStepKind`

Type.

Kind of a trajectory step: thinking, text output, or a tool call.

```ts
type TrajectoryStepKind = 'think' | 'text' | 'tool'
```

Source: [src/eval-trajectory.ts:8](https://github.com/Tahul/zidane/blob/main/src/eval-trajectory.ts#L8)