zidanezidane/
Eval runner primitives that score headless agent runs and write artifacts.
import { … } from 'zidane/eval'src/eval.ts
Use this entry point to test agent behavior against a real provider. runEvalCase prepares a workspace, runs the agent headless, applies scorers, computes metrics and the trajectory, and writes artifacts. Built-in scorers check the run status and the files in the workspace. llmJudge grades a run with a model. defineEval registers a case factory, and buildRegisteredEvals builds the cases with the provider and judge that the harness passes.
import { fileExists, runEvalCase, statusCompleted } from 'zidane/eval'
import { anthropic } from 'zidane/providers/anthropic'
const result = await runEvalCase({
id: 'creates-readme',
suite: 'files',
prompt: 'Create README.md with a one-line project description.',
provider: anthropic(),
maxTurns: 8,
workspace: { capture: ['.'] },
scorers: [statusCompleted(), fileExists('README.md')],
})
console.log(result.passed, result.score)Functions#
artifactPathFunction#
Resolves the artifact directory of a case: <root>/<suite>/<id>. Characters that are unsafe in a path segment are replaced.
buildEvalRunSummaryFunction#
Builds the summary of an eval run: case counts, mean score, duration and usage totals, one row per case, metric and tag aggregates, and per-variant summaries when cases have variants.
buildRegisteredEvalsFunction#
Snapshot of every registered eval, materialized for one provider/judge pair.
buildTrajectoryFunction#
Build an ordered, grouped trajectory from a transcript. Walks assistant
turns in order (an aborted one, output a process death cut off and sent
again, is skipped), emitting think / text / tool:<name> steps, grouping
only consecutive identical kinds. Per-step duration is approximated from the
owning turn's createdAt delta to the next turn.
clearRegisteredEvalsFunction#
Drop every registered eval — test isolation helper.
computeEvalTagScoresFunction#
Computes the mean normalized score for each tag across the given metrics.
createEvalAgentFunction#
Multi-turn eval agent over the low-level headless runner. It keeps session
and execution lifetime outside any Playwright-specific fixture layer,
so downstream platforms can wrap it with their own agent.run(...) shape.
createEvalRunReporterFunction#
Creates a reporter that collects case results, builds the run summary and formats it as a terminal table.
createReusableExecutionContextFunction#
Wrap an execution context so repeated agent runs share one handle until the
caller disposes it. This is useful for eval fixtures that want per-test
Docker/process lifetime while still using runHeadless() per turn.
defineEvalFunction#
Register an eval definition. Eval files call this at module load; the harness
(buildRegisteredEvals) materializes them with a provider at run time. The
factory keeps definitions lazy so a single registry serves any provider/judge
pairing without re-importing eval files.
Returns the same factory so a file can also export default it for direct,
registry-free use (e.g. hermetic unit tests).
defineMetricsFunction#
Declare an eval's metric set. Returns the same map, typed; pass it to
EvalCaseOptions.metrics. Declared metrics that are not emitted are recorded
as missing with a normalized score of 0; emitting an undeclared metric is an
authoring error and throws.
efficiencyMetricValuesFunction#
Computes the raw values of the EFFICIENCY_METRICS for a run: wall-clock time, input plus output tokens, and the share of input read from cache.
emitEfficiencyMetricsFunction#
Emits every efficiency metric for a run through emit.
fileContainsFunction#
Creates a scorer that passes when the captured content of path contains expected. A missing file fails.
fileContentQualityFunction#
Creates a scorer that grades how closely a file matches expected: 1 for an exact match after trimming, 0.75 when the trimmed content contains expected, 0.5 when the raw content does, otherwise 0. It passes at 0.75 or more.
fileExistsFunction#
Creates a scorer that passes when the workspace snapshot contains path.
fileExistsOneOfFunction#
Creates a scorer that passes when the workspace snapshot contains at least one of paths.
finalizeEvalMetricsFunction#
Builds the normalized metrics of a case. A declared metric that was never emitted is recorded as missing with a score of 0.
formatEvalCaseSummaryFunction#
Formats one case result as plain text: status, score, duration, usage, workspace and artifact info, and per-scorer results.
formatEvalRunSummaryFunction#
Formats an eval run as a terminal table, with a comparison matrix when the cases have more than one variant.
formatTrajectoryLineFunction#
Render a trajectory as a one-line timeline string (no color).
functionalityMetricFunction#
Aggregate a group of boolean scorers into a single 0..1 functionality
metric (fraction passed) and emit it. Useful for "N of M files present".
llmJudgeFunction#
Creates a scorer that asks a model to grade the run against a rubric. The judge answers through a forced submit_grade tool call with a score from 0 to 1 and a short reasoning. The scorer passes at a score of 0.7 or more.
normalizeMetricFunction#
Maps a raw value to a score from 0 to 1, clamped to the spec's range and flipped for 'lower-is-better'. Returns 1 when min equals max.
registerEvalTestsFunction#
Register one test per eval case against a test runner, funnel every result into a shared reporter, and print the aggregated summary once after all cases.
A failing test means the agent run itself broke (provider/tool/timeout). Low
scores are reported, not asserted — see RegisterEvalTestsOptions.failOnIncomplete.
relativeArtifactPathFunction#
Returns path relative to root.
runEvalCaseFunction#
Runs one eval case: prepares the workspace, runs the agent headless, runs the scorers, computes metrics and the trajectory, and writes artifacts when artifactDir is set. The workspace is cleaned up when the run ends.
Throws EvalMetricError when a scorer emits a metric that was not declared or a non-finite value. Other scorer failures are recorded as failed scores.
statusCompletedFunction#
Creates a scorer that passes when the run status is 'completed'.
Classes#
EvalMetricErrorClass#
Thrown by runEvalCase when a scorer emits a metric id that was not declared
in defineMetrics. This is an authoring error (a typo'd or stale metric id),
not a low score — it fails the eval test rather than being recorded as a
failed scorer. Exported so downstream harnesses can type-narrow on it.
Constants#
EFFICIENCY_METRICSConstant#
Always-available efficiency metrics derived from HeadlessResult.
Interfaces#
CreateEvalAgentOptionsInterface#
Options for createEvalAgent. The HeadlessOptions fields set here are the defaults for every run.
EvalAgentInterface#
Multi-turn agent for evals. Every run shares one session and one execution handle.
EvalAgentRunOptionsInterface#
Options for one EvalAgent.run call. Overrides the defaults set in createEvalAgent; mcpServers are added to the agent's own.
EvalAgentRunResultInterface#
Result of one EvalAgent run.
EvalAgentRunStatsInterface#
Usage and counters of one EvalAgent run.
EvalAgentStatsInterface#
Totals across every run of an EvalAgent.
EvalArtifactsInterface#
Paths of the artifact files written for one case.
EvalCaseOptionsInterface#
Definition of one eval case. Extends HeadlessOptions with an id, scorers, metrics and workspace setup, and is run by runEvalCase.
EvalCaseResultInterface#
Outcome of one eval case: the headless result plus scores, metrics, trajectory and artifacts.
EvalDefinitionContextInterface#
Context handed to an eval definition factory at registration time.
EvalMetricInterface#
A normalized metric value attached to one eval case.
EvalRunMetricAggregateInterface#
Statistics of one metric across every case that emitted it.
EvalRunReporterInterface#
Collects case results during a run and produces the run summary.
EvalRunReporterOptionsInterface#
Options for createEvalRunReporter.
EvalRunSummaryInterface#
Summary of an eval run, built by buildEvalRunSummary.
EvalRunSummaryCaseInterface#
One row of EvalRunSummary.cases.
EvalRunUsageInterface#
Token and cost totals.
EvalScoreInterface#
Result of one scorer.
EvalScorerContextInterface#
Everything a scorer can inspect after a run.
EvalTestRunnerInterface#
Minimal subset of a test runner's API the eval harness needs.
EvalVariantSummaryInterface#
Aggregate rollup for one run variant (provider/model target).
EvalWorkspaceFileInterface#
One file captured from the workspace after a run.
EvalWorkspaceOptionsInterface#
Workspace setup for an eval case: where the agent works, what is copied in, and what is captured afterwards.
EvalWorkspaceSnapshotInterface#
Files captured from the workspace after a run.
LlmJudgeOptionsInterface#
Options for llmJudge.
MetricSpecInterface#
Declared metric: a raw signal plus how to normalize it to a 0..1 score.
MetricStatsInterface#
Descriptive statistics of one metric across cases.
RegisterEvalTestsOptionsInterface#
Options for registerEvalTests.
ReusableExecutionContextInterface#
Execution context that shares one handle across runs, plus the hooks to inspect and dispose it. Created by createReusableExecutionContext.
TrajectoryInterface#
Ordered, grouped view of what an agent did during an eval run.
TrajectoryStepInterface#
One grouped step in an eval trajectory. Consecutive blocks of the same kind
(and, for tools, the same tool name) collapse into a single step with a
count. A different kind interrupting the run breaks the group.
Serializable by design so trajectories round-trip to disk and can be diffed or replayed by other tooling.
Types#
EvalAgentMcpServersType#
MCP servers for an EvalAgent: a fixed list, or a function that returns the list before each run.
EvalDefinitionType#
Factory that builds an eval case from an EvalDefinitionContext. Register one with defineEval.
EvalScorerType#
Function that scores one finished run. Returns an EvalScore, or a promise of one.
MetricDirectionType#
Whether a larger raw value is better or worse.
MetricEmitterType#
Emit a raw metric value during a run.
MetricSpecMapType#
Metric specs keyed by metric id.
TrajectoryStepKindType#
Kind of a trajectory step: thinking, text output, or a tool call.