$worker
eval
v0.2.2-experimentalCompare live objective metrics for 2-5 root sessions; retain durable prompt experiments as an advanced surface.
iiiverifiedinstall
$iii worker add eval
- macOS: arm64 · x64
- Linux: arm64 · armv7 · x64
- Windows: arm64 · x64 · x86
repoiii-hq/workers
exact versions are immutable; binary and bundle artifacts are digest-pinned.
agent-ready brief for v0.2.2-experimental
install + config + dependencies + readme + api reference, all in one place. fetch as agent-context.md for an llm to consume.
full markdown
the same content rendered as discrete blocks below is exposed as a single markdown document at
/workers/eval.md. paste it into an llm prompt or pipe it through curl from a worker.install
install
$iii worker add eval@0.2.2-experimental
dependencies
readme
README.md
eval
eval primarily compares 2–5 existing root sessions live. It reads the
current session lifecycle and harness::metrics values, then displays
objective deltas against a user-selected reference. The user decides whether
the sessions are comparable; eval does not score, rank, validate equivalence,
or persist a comparison. Prompt and system-prompt experiments remain available
as an advanced surface.
Start an evaluation
{
"dimension": "prompt",
"model": {
"model": "codex/gpt-5.6-luna",
"provider": "openai-codex",
"system_prompt_strategy": "override"
},
"control": {
"label": "baseline",
"prompt": "Reply with exactly OK.",
"system_prompt": "Follow the user request exactly."
},
"treatment": {
"label": "candidate",
"prompt": "Return exactly the text OK.",
"system_prompt": "Follow the user request exactly."
},
"evaluator": {
"function_id": "eval::assert::exact",
"arguments": {
"expected": "OK"
}
}
}Call eval::start with the request. It returns an evaluation_id
immediately. Use eval::status for occasional progress checks and
eval::result for the terminal report, or bind to the eval::completed
trigger type.
One run per variant is scheduled by default. Their order alternates by pair to reduce order bias. A candidate is eligible only when every treatment run passes and its pass count does not regress against control. Efficiency metrics are descriptive and never select a winner automatically.
Public functions
eval::compare-sessions— read 2–5 root sessions concurrently and return objective metrics plus arithmetic deltas against the selected reference.eval::start— validate, persist, and enqueue an evaluation.eval::list— list recent evaluations as lightweight summaries.eval::status— inspect progress without loading the full report.eval::result— read the normalized request and terminal comparison report.eval::cancel— cancel the active harness session and remaining runs.eval::delete— delete a terminal evaluation and its session indexes.eval::assert::exact— built-in deep JSON/string equality evaluator.
Evaluator functions receive the output, harness::metrics, run identity, and
caller-supplied arguments. They return { passed, score?, reason?, details? }.
Evaluators should be deterministic and idempotent because durable delivery is
at-least-once.
Console UI
When the console worker is running, eval injects an eval page at
#/ext/eval-benchmarks. The default Sessions tab lists visible root
sessions, accepts active sessions, and renders a live matrix grouped by
efficiency, reliability, orchestration, and context. Prompt experiments
keeps the durable prompt/system-prompt workflow for advanced use.
The model picker reads the live router::models::list catalog and falls back
to manual model/provider entry when the catalog is unavailable. Exact-value
evaluation is built in; any evaluator function can also be selected by id with
JSON arguments. Harness policies and output/metadata options remain collapsed
until needed.
Boundaries
The worker intentionally does not implement an agent loop, model router, metrics collector, trace collector, test DSL, or LLM judge. Those concerns remain in the harness, engine, and user-provided evaluator functions.
api reference (json)
agent-api-reference.json
{
"functions": [
{
"description": "Built-in deterministic evaluator: deep-compare output with arguments.expected.",
"metadata": {},
"name": "eval::assert::exact",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ContextSnapshotV1": {
"description": "One generation's context accounting. `free = usable - total`, floored at zero: once provider usage lands, `total` is what was billed, which can exceed the `usable` budget the window was fit into — that budget was derived before the generation from an estimate.",
"properties": {
"categories": {
"$ref": "#/definitions/SnapshotCategoriesV1"
},
"compacted": {
"type": "boolean"
},
"effective_max_output_tokens": {
"description": "Output allocation `usable` was derived against.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"estimator": {
"description": "Which estimator produced the numbers (`heuristic` until the context-manager resolves a real tokenizer). Absent when the context-manager predates the breakdown response.",
"type": [
"string",
"null"
]
},
"free": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"model": {
"type": "string"
},
"provider": {
"type": [
"string",
"null"
]
},
"session_cost_usd": {
"description": "Running cost of the whole session in USD, accumulated across every generation step. `usage.cost_usd` is one step's bill — on providers with steep cache discounts the per-step number swings two orders of magnitude, so a chip showing it alone reads as a bouncing total.",
"format": "double",
"type": [
"number",
"null"
]
},
"session_id": {
"type": "string"
},
"step": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"summarized_head_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"timestamp": {
"format": "int64",
"type": "integer"
},
"total": {
"description": "Final request estimate: categories plus post-assembly growth.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"turn_id": {
"type": "string"
},
"usable": {
"description": "The input budget the window was fit into.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"usage": {
"anyOf": [
{
"$ref": "#/definitions/Usage"
},
{
"type": "null"
}
],
"description": "Actual provider usage for this generation, stamped after the terminal frame; absent when the provider returned none (or the generation never completed)."
}
},
"required": [
"categories",
"compacted",
"effective_max_output_tokens",
"free",
"model",
"session_id",
"step",
"timestamp",
"total",
"turn_id",
"usable"
],
"type": "object"
},
"SessionMetricsResponseV1": {
"additionalProperties": false,
"properties": {
"by_session": {
"items": {
"$ref": "#/definitions/SessionUsageV1"
},
"type": "array"
},
"complete": {
"type": "boolean"
},
"root_session_id": {
"type": "string"
},
"totals": {
"$ref": "#/definitions/SessionUsageTotalsV1"
},
"traces": {
"anyOf": [
{
"$ref": "#/definitions/SessionTraceMetricsV1"
},
{
"type": "null"
}
],
"description": "Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not."
}
},
"required": [
"by_session",
"complete",
"root_session_id",
"totals"
],
"type": "object"
},
"SessionTraceMetricsV1": {
"additionalProperties": false,
"properties": {
"by_session": {
"items": {
"$ref": "#/definitions/SessionTraceUsageV1"
},
"type": "array"
},
"duration_ms": {
"description": "Elapsed window from the first observed span to the last observed span.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"error_span_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"span_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"trace_count": {
"description": "Distinct traces across the root session and all descendants.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"by_session",
"duration_ms",
"error_span_count",
"span_count",
"trace_count"
],
"type": "object"
},
"SessionTraceUsageV1": {
"additionalProperties": false,
"properties": {
"depth": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"duration_ms": {
"description": "Elapsed window from the session's first observed span to its last.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"error_span_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"parent_session_id": {
"type": [
"string",
"null"
]
},
"session_id": {
"type": "string"
},
"span_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"trace_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"depth",
"duration_ms",
"error_span_count",
"session_id",
"span_count",
"trace_count"
],
"type": "object"
},
"SessionUsageTotalsV1": {
"additionalProperties": false,
"properties": {
"cache_read_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"reasoning_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"sessions": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"turns": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"function_call_errors",
"function_calls",
"sessions",
"turns"
],
"type": "object"
},
"SessionUsageV1": {
"additionalProperties": false,
"properties": {
"cache_read_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"context": {
"anyOf": [
{
"$ref": "#/definitions/ContextSnapshotV1"
},
{
"type": "null"
}
],
"description": "The session's latest per-generation context snapshot (categories, budget, usage) — absent for sessions that have not generated since snapshots landed."
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"depth": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"parent_session_id": {
"type": [
"string",
"null"
]
},
"reasoning_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"session_id": {
"type": "string"
},
"turns": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"depth",
"function_call_errors",
"function_calls",
"session_id",
"turns"
],
"type": "object"
},
"SnapshotCategoriesV1": {
"description": "Where the request's tokens sit. Categories are assembly-time estimates; `hook_guidance` is the measured growth after assembly (pre-generate hook appends and orphan-repair patches), 0 when the request left assembly unchanged.",
"properties": {
"hook_guidance": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"messages": {
"$ref": "#/definitions/SnapshotMessagesV1"
},
"overhead": {
"description": "Provider framing plus response_format / provider_options fields.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"system_prompt": {
"description": "Final assembled system prompt: mode paragraph, identity, per-step aids, and any compaction summary section.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"tools": {
"description": "Function schemas exposed to the model.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"messages",
"overhead",
"system_prompt",
"tools"
],
"type": "object"
},
"SnapshotMessagesV1": {
"description": "Estimated tokens of the assembled window's messages, by role.",
"properties": {
"assistant": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"custom": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_result": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"user": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"assistant",
"custom",
"function_result",
"user"
],
"type": "object"
},
"Usage": {
"properties": {
"cache_read": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"input": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"reasoning": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"VariantRoleV1": {
"enum": [
"control",
"treatment"
],
"type": "string"
}
},
"properties": {
"arguments": {
"default": {}
},
"evaluation_id": {
"type": "string"
},
"metrics": {
"$ref": "#/definitions/SessionMetricsResponseV1"
},
"output": true,
"role": {
"$ref": "#/definitions/VariantRoleV1"
},
"run_id": {
"type": "string"
},
"session_id": {
"type": "string"
}
},
"required": [
"evaluation_id",
"metrics",
"output",
"role",
"run_id",
"session_id"
],
"title": "EvaluatorInputV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"details": true,
"passed": {
"type": "boolean"
},
"reason": {
"type": [
"string",
"null"
]
},
"score": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"passed"
],
"title": "EvaluatorResponseV1",
"type": "object"
}
},
{
"description": "Built-in deterministic evaluator: compare text after normalizing case, whitespace, and surrounding punctuation.",
"metadata": {},
"name": "eval::assert::normalized_text",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ContextSnapshotV1": {
"description": "One generation's context accounting. `free = usable - total`, floored at zero: once provider usage lands, `total` is what was billed, which can exceed the `usable` budget the window was fit into — that budget was derived before the generation from an estimate.",
"properties": {
"categories": {
"$ref": "#/definitions/SnapshotCategoriesV1"
},
"compacted": {
"type": "boolean"
},
"effective_max_output_tokens": {
"description": "Output allocation `usable` was derived against.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"estimator": {
"description": "Which estimator produced the numbers (`heuristic` until the context-manager resolves a real tokenizer). Absent when the context-manager predates the breakdown response.",
"type": [
"string",
"null"
]
},
"free": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"model": {
"type": "string"
},
"provider": {
"type": [
"string",
"null"
]
},
"session_cost_usd": {
"description": "Running cost of the whole session in USD, accumulated across every generation step. `usage.cost_usd` is one step's bill — on providers with steep cache discounts the per-step number swings two orders of magnitude, so a chip showing it alone reads as a bouncing total.",
"format": "double",
"type": [
"number",
"null"
]
},
"session_id": {
"type": "string"
},
"step": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"summarized_head_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"timestamp": {
"format": "int64",
"type": "integer"
},
"total": {
"description": "Final request estimate: categories plus post-assembly growth.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"turn_id": {
"type": "string"
},
"usable": {
"description": "The input budget the window was fit into.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"usage": {
"anyOf": [
{
"$ref": "#/definitions/Usage"
},
{
"type": "null"
}
],
"description": "Actual provider usage for this generation, stamped after the terminal frame; absent when the provider returned none (or the generation never completed)."
}
},
"required": [
"categories",
"compacted",
"effective_max_output_tokens",
"free",
"model",
"session_id",
"step",
"timestamp",
"total",
"turn_id",
"usable"
],
"type": "object"
},
"SessionMetricsResponseV1": {
"additionalProperties": false,
"properties": {
"by_session": {
"items": {
"$ref": "#/definitions/SessionUsageV1"
},
"type": "array"
},
"complete": {
"type": "boolean"
},
"root_session_id": {
"type": "string"
},
"totals": {
"$ref": "#/definitions/SessionUsageTotalsV1"
},
"traces": {
"anyOf": [
{
"$ref": "#/definitions/SessionTraceMetricsV1"
},
{
"type": "null"
}
],
"description": "Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not."
}
},
"required": [
"by_session",
"complete",
"root_session_id",
"totals"
],
"type": "object"
},
"SessionTraceMetricsV1": {
"additionalProperties": false,
"properties": {
"by_session": {
"items": {
"$ref": "#/definitions/SessionTraceUsageV1"
},
"type": "array"
},
"duration_ms": {
"description": "Elapsed window from the first observed span to the last observed span.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"error_span_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"span_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"trace_count": {
"description": "Distinct traces across the root session and all descendants.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"by_session",
"duration_ms",
"error_span_count",
"span_count",
"trace_count"
],
"type": "object"
},
"SessionTraceUsageV1": {
"additionalProperties": false,
"properties": {
"depth": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"duration_ms": {
"description": "Elapsed window from the session's first observed span to its last.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"error_span_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"parent_session_id": {
"type": [
"string",
"null"
]
},
"session_id": {
"type": "string"
},
"span_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"trace_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"depth",
"duration_ms",
"error_span_count",
"session_id",
"span_count",
"trace_count"
],
"type": "object"
},
"SessionUsageTotalsV1": {
"additionalProperties": false,
"properties": {
"cache_read_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"reasoning_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"sessions": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"turns": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"function_call_errors",
"function_calls",
"sessions",
"turns"
],
"type": "object"
},
"SessionUsageV1": {
"additionalProperties": false,
"properties": {
"cache_read_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"context": {
"anyOf": [
{
"$ref": "#/definitions/ContextSnapshotV1"
},
{
"type": "null"
}
],
"description": "The session's latest per-generation context snapshot (categories, budget, usage) — absent for sessions that have not generated since snapshots landed."
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"depth": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"parent_session_id": {
"type": [
"string",
"null"
]
},
"reasoning_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"session_id": {
"type": "string"
},
"turns": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"depth",
"function_call_errors",
"function_calls",
"session_id",
"turns"
],
"type": "object"
},
"SnapshotCategoriesV1": {
"description": "Where the request's tokens sit. Categories are assembly-time estimates; `hook_guidance` is the measured growth after assembly (pre-generate hook appends and orphan-repair patches), 0 when the request left assembly unchanged.",
"properties": {
"hook_guidance": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"messages": {
"$ref": "#/definitions/SnapshotMessagesV1"
},
"overhead": {
"description": "Provider framing plus response_format / provider_options fields.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"system_prompt": {
"description": "Final assembled system prompt: mode paragraph, identity, per-step aids, and any compaction summary section.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"tools": {
"description": "Function schemas exposed to the model.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"messages",
"overhead",
"system_prompt",
"tools"
],
"type": "object"
},
"SnapshotMessagesV1": {
"description": "Estimated tokens of the assembled window's messages, by role.",
"properties": {
"assistant": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"custom": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_result": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"user": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"assistant",
"custom",
"function_result",
"user"
],
"type": "object"
},
"Usage": {
"properties": {
"cache_read": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"input": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"reasoning": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"VariantRoleV1": {
"enum": [
"control",
"treatment"
],
"type": "string"
}
},
"properties": {
"arguments": {
"default": {}
},
"evaluation_id": {
"type": "string"
},
"metrics": {
"$ref": "#/definitions/SessionMetricsResponseV1"
},
"output": true,
"role": {
"$ref": "#/definitions/VariantRoleV1"
},
"run_id": {
"type": "string"
},
"session_id": {
"type": "string"
}
},
"required": [
"evaluation_id",
"metrics",
"output",
"role",
"run_id",
"session_id"
],
"title": "EvaluatorInputV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"details": true,
"passed": {
"type": "boolean"
},
"reason": {
"type": [
"string",
"null"
]
},
"score": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"passed"
],
"title": "EvaluatorResponseV1",
"type": "object"
}
},
{
"description": "Cancel an evaluation and its active harness turn.",
"metadata": {},
"name": "eval::cancel",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"evaluation_id": {
"type": "string"
}
},
"required": [
"evaluation_id"
],
"title": "EvaluationIdRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"EvalStatusV1": {
"enum": [
"queued",
"running",
"completed",
"failed",
"cancelled"
],
"type": "string"
}
},
"properties": {
"cancelled": {
"type": "boolean"
},
"status": {
"$ref": "#/definitions/EvalStatusV1"
}
},
"required": [
"cancelled",
"status"
],
"title": "EvalCancelResponseV1",
"type": "object"
}
},
{
"description": "Compare 2–5 existing root sessions from live session, lifecycle, and Harness metrics. The response contains objective values and arithmetic deltas against the chosen baseline; it does not score, rank, validate equivalence, or persist a snapshot.",
"metadata": {},
"name": "eval::compare-sessions",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"description": "The only input needed to compare sessions. The caller is responsible for choosing sessions that are meaningful to compare; the worker only checks shape and root identity.",
"properties": {
"baseline_session_id": {
"type": "string"
},
"session_ids": {
"items": {
"type": "string"
},
"maxItems": 5,
"minItems": 2,
"type": "array"
}
},
"required": [
"baseline_session_id",
"session_ids"
],
"title": "CompareSessionsRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ContextSnapshotV1": {
"description": "One generation's context accounting. `free = usable - total`, floored at zero: once provider usage lands, `total` is what was billed, which can exceed the `usable` budget the window was fit into — that budget was derived before the generation from an estimate.",
"properties": {
"categories": {
"$ref": "#/definitions/SnapshotCategoriesV1"
},
"compacted": {
"type": "boolean"
},
"effective_max_output_tokens": {
"description": "Output allocation `usable` was derived against.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"estimator": {
"description": "Which estimator produced the numbers (`heuristic` until the context-manager resolves a real tokenizer). Absent when the context-manager predates the breakdown response.",
"type": [
"string",
"null"
]
},
"free": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"model": {
"type": "string"
},
"provider": {
"type": [
"string",
"null"
]
},
"session_cost_usd": {
"description": "Running cost of the whole session in USD, accumulated across every generation step. `usage.cost_usd` is one step's bill — on providers with steep cache discounts the per-step number swings two orders of magnitude, so a chip showing it alone reads as a bouncing total.",
"format": "double",
"type": [
"number",
"null"
]
},
"session_id": {
"type": "string"
},
"step": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"summarized_head_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"timestamp": {
"format": "int64",
"type": "integer"
},
"total": {
"description": "Final request estimate: categories plus post-assembly growth.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"turn_id": {
"type": "string"
},
"usable": {
"description": "The input budget the window was fit into.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"usage": {
"anyOf": [
{
"$ref": "#/definitions/Usage"
},
{
"type": "null"
}
],
"description": "Actual provider usage for this generation, stamped after the terminal frame; absent when the provider returned none (or the generation never completed)."
}
},
"required": [
"categories",
"compacted",
"effective_max_output_tokens",
"free",
"model",
"session_id",
"step",
"timestamp",
"total",
"turn_id",
"usable"
],
"type": "object"
},
"MetricDeltaV1": {
"properties": {
"absolute": {
"format": "double",
"type": [
"number",
"null"
]
},
"percent": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"type": "object"
},
"ObjectiveSummaryV1": {
"properties": {
"cache_read_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"compacted_sessions": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"context_free_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"context_occupancy": {
"format": "double",
"type": [
"number",
"null"
]
},
"context_total_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"context_usable_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_per_generation_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"descendants": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"error_span_count": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"function_error_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"generations": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_depth": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"reasoning_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"sessions": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"span_count": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"subject_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"tokens_per_generation": {
"format": "double",
"type": [
"number",
"null"
]
},
"total_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"trace_count": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"trace_duration_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"SessionComparisonItemV1": {
"properties": {
"deltas": {
"additionalProperties": {
"$ref": "#/definitions/MetricDeltaV1"
},
"default": {},
"description": "One entry per objective numeric metric. A missing value is represented by null in either delta field, never by zero.",
"type": "object"
},
"errors": {
"items": {
"type": "string"
},
"type": "array"
},
"lifecycle": {
"$ref": "#/definitions/SessionLifecycleV1"
},
"metrics": {
"anyOf": [
{
"$ref": "#/definitions/SessionMetricsResponseV1"
},
{
"type": "null"
}
]
},
"session": {
"$ref": "#/definitions/SessionMetaProjectionV1"
},
"summary": {
"anyOf": [
{
"$ref": "#/definitions/ObjectiveSummaryV1"
},
{
"type": "null"
}
]
}
},
"required": [
"lifecycle",
"session"
],
"type": "object"
},
"SessionLifecycleV1": {
"properties": {
"complete": {
"description": "Null means the metrics read was unavailable.",
"type": [
"boolean",
"null"
]
},
"expects_wake": {
"description": "Null means the status read was unavailable.",
"type": [
"boolean",
"null"
]
},
"partial": {
"type": "boolean"
},
"result_error": {
"type": [
"string",
"null"
]
},
"session_status": {
"$ref": "#/definitions/SessionStatusV1"
},
"terminal": {
"description": "Null means `harness::status` did not provide a turn record.",
"type": [
"boolean",
"null"
]
},
"turn_id": {
"type": [
"string",
"null"
]
},
"turn_status": {
"anyOf": [
{
"$ref": "#/definitions/TurnStatus"
},
{
"type": "null"
}
]
}
},
"required": [
"partial",
"session_status"
],
"type": "object"
},
"SessionMetaProjectionV1": {
"properties": {
"created_at": {
"default": 0,
"format": "int64",
"type": "integer"
},
"description": {
"default": "",
"type": "string"
},
"draft": {
"type": [
"string",
"null"
]
},
"forked_from": {
"type": [
"string",
"null"
]
},
"message_count": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"metadata": true,
"session_id": {
"type": "string"
},
"status": {
"$ref": "#/definitions/SessionStatusV1"
},
"status_reason": {
"type": [
"string",
"null"
]
},
"title": {
"default": "",
"type": "string"
},
"updated_at": {
"default": 0,
"format": "int64",
"type": "integer"
}
},
"required": [
"session_id",
"status"
],
"type": "object"
},
"SessionMetricsResponseV1": {
"additionalProperties": false,
"properties": {
"by_session": {
"items": {
"$ref": "#/definitions/SessionUsageV1"
},
"type": "array"
},
"complete": {
"type": "boolean"
},
"root_session_id": {
"type": "string"
},
"totals": {
"$ref": "#/definitions/SessionUsageTotalsV1"
},
"traces": {
"anyOf": [
{
"$ref": "#/definitions/SessionTraceMetricsV1"
},
{
"type": "null"
}
],
"description": "Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not."
}
},
"required": [
"by_session",
"complete",
"root_session_id",
"totals"
],
"type": "object"
},
"SessionStatusV1": {
"enum": [
"idle",
"working",
"done",
"error"
],
"type": "string"
},
"SessionTraceMetricsV1": {
"additionalProperties": false,
"properties": {
"by_session": {
"items": {
"$ref": "#/definitions/SessionTraceUsageV1"
},
"type": "array"
},
"duration_ms": {
"description": "Elapsed window from the first observed span to the last observed span.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"error_span_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"span_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"trace_count": {
"description": "Distinct traces across the root session and all descendants.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"by_session",
"duration_ms",
"error_span_count",
"span_count",
"trace_count"
],
"type": "object"
},
"SessionTraceUsageV1": {
"additionalProperties": false,
"properties": {
"depth": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"duration_ms": {
"description": "Elapsed window from the session's first observed span to its last.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"error_span_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"parent_session_id": {
"type": [
"string",
"null"
]
},
"session_id": {
"type": "string"
},
"span_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"trace_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"depth",
"duration_ms",
"error_span_count",
"session_id",
"span_count",
"trace_count"
],
"type": "object"
},
"SessionUsageTotalsV1": {
"additionalProperties": false,
"properties": {
"cache_read_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"reasoning_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"sessions": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"turns": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"function_call_errors",
"function_calls",
"sessions",
"turns"
],
"type": "object"
},
"SessionUsageV1": {
"additionalProperties": false,
"properties": {
"cache_read_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"context": {
"anyOf": [
{
"$ref": "#/definitions/ContextSnapshotV1"
},
{
"type": "null"
}
],
"description": "The session's latest per-generation context snapshot (categories, budget, usage) — absent for sessions that have not generated since snapshots landed."
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"depth": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"parent_session_id": {
"type": [
"string",
"null"
]
},
"reasoning_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"session_id": {
"type": "string"
},
"turns": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"depth",
"function_call_errors",
"function_calls",
"session_id",
"turns"
],
"type": "object"
},
"SnapshotCategoriesV1": {
"description": "Where the request's tokens sit. Categories are assembly-time estimates; `hook_guidance` is the measured growth after assembly (pre-generate hook appends and orphan-repair patches), 0 when the request left assembly unchanged.",
"properties": {
"hook_guidance": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"messages": {
"$ref": "#/definitions/SnapshotMessagesV1"
},
"overhead": {
"description": "Provider framing plus response_format / provider_options fields.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"system_prompt": {
"description": "Final assembled system prompt: mode paragraph, identity, per-step aids, and any compaction summary section.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"tools": {
"description": "Function schemas exposed to the model.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"messages",
"overhead",
"system_prompt",
"tools"
],
"type": "object"
},
"SnapshotMessagesV1": {
"description": "Estimated tokens of the assembled window's messages, by role.",
"properties": {
"assistant": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"custom": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_result": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"user": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"assistant",
"custom",
"function_result",
"user"
],
"type": "object"
},
"TurnStatus": {
"description": "The coarse, harness-internal turn lifecycle (harness.md § API Reference). Finer-grained than the session's `status`, which the loop derives from it.",
"enum": [
"running",
"awaiting_functions",
"completed",
"cancelled",
"failed"
],
"type": "string"
},
"Usage": {
"properties": {
"cache_read": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"input": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"reasoning": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
}
},
"properties": {
"baseline_session_id": {
"type": "string"
},
"captured_at": {
"format": "int64",
"type": "integer"
},
"schema_version": {
"type": "string"
},
"sessions": {
"items": {
"$ref": "#/definitions/SessionComparisonItemV1"
},
"type": "array"
}
},
"required": [
"baseline_session_id",
"captured_at",
"schema_version",
"sessions"
],
"title": "SessionComparisonResponseV1",
"type": "object"
}
},
{
"description": "Delete a terminal evaluation report and its session indexes.",
"metadata": {},
"name": "eval::delete",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"evaluation_id": {
"type": "string"
}
},
"required": [
"evaluation_id"
],
"title": "EvaluationIdRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"deleted": {
"type": "boolean"
}
},
"required": [
"deleted"
],
"title": "EvalDeleteResponseV1",
"type": "object"
}
},
{
"description": "List recent evaluations as lightweight summaries, newest first.",
"metadata": {},
"name": "eval::list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"limit": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"title": "EvalListRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"ComparisonDimensionV1": {
"enum": [
"prompt",
"system_prompt"
],
"type": "string"
},
"EvalStatusV1": {
"enum": [
"queued",
"running",
"completed",
"failed",
"cancelled"
],
"type": "string"
},
"EvalSummaryV1": {
"additionalProperties": false,
"properties": {
"completed_at": {
"format": "int64",
"type": [
"integer",
"null"
]
},
"control_label": {
"type": [
"string",
"null"
]
},
"created_at": {
"format": "int64",
"type": "integer"
},
"dimension": {
"$ref": "#/definitions/ComparisonDimensionV1"
},
"eligible": {
"type": [
"boolean",
"null"
]
},
"error": {
"type": [
"string",
"null"
]
},
"evaluation_id": {
"type": "string"
},
"failed_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"model": {
"type": "string"
},
"passed_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"provider": {
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/EvalStatusV1"
},
"terminal_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"total_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"treatment_label": {
"type": [
"string",
"null"
]
},
"updated_at": {
"format": "int64",
"type": "integer"
}
},
"required": [
"created_at",
"dimension",
"evaluation_id",
"failed_runs",
"model",
"passed_runs",
"status",
"terminal_runs",
"total_runs",
"updated_at"
],
"type": "object"
}
},
"properties": {
"evaluations": {
"items": {
"$ref": "#/definitions/EvalSummaryV1"
},
"type": "array"
}
},
"required": [
"evaluations"
],
"title": "EvalListResponseV1",
"type": "object"
}
},
{
"description": "Internal harness turn-completed wake-up.",
"metadata": {
"internal": true,
"trace_hidden": true
},
"name": "eval::on-turn-completed",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"session_id": {
"default": "",
"type": "string"
},
"terminal": {
"default": false,
"type": "boolean"
}
},
"title": "WakeEventV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"woke": {
"type": "boolean"
}
},
"required": [
"woke"
],
"title": "WakeResponseV1",
"type": "object"
}
},
{
"description": "Repeat a terminal evaluation from its persisted request. Set reverse_order to invert the balanced A/B order. The rerun receives a new evaluation ID and fresh sessions.",
"metadata": {},
"name": "eval::rerun",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"evaluation_id": {
"type": "string"
},
"reverse_order": {
"default": false,
"type": "boolean"
}
},
"required": [
"evaluation_id"
],
"title": "EvalRerunRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"EvalStatusV1": {
"enum": [
"queued",
"running",
"completed",
"failed",
"cancelled"
],
"type": "string"
}
},
"properties": {
"evaluation_id": {
"type": "string"
},
"status": {
"$ref": "#/definitions/EvalStatusV1"
}
},
"required": [
"evaluation_id",
"status"
],
"title": "EvalStartResponseV1",
"type": "object"
}
},
{
"description": "Read an evaluation result. The report is present only after terminal completion.",
"metadata": {},
"name": "eval::result",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"evaluation_id": {
"type": "string"
}
},
"required": [
"evaluation_id"
],
"title": "EvaluationIdRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"anyOf": [
{
"$ref": "#/definitions/EvalResultResponseV1"
},
{
"type": "null"
}
],
"definitions": {
"AggregateDeltaV1": {
"additionalProperties": false,
"properties": {
"median_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_error_span_count": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_call_errors": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_reasoning_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_span_count": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_total_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_trace_count": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_trace_duration_ms": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_wall_time_ms": {
"format": "double",
"type": [
"number",
"null"
]
},
"pass_rate": {
"format": "double",
"type": "number"
}
},
"required": [
"pass_rate"
],
"type": "object"
},
"ComparisonDimensionV1": {
"enum": [
"prompt",
"system_prompt"
],
"type": "string"
},
"ContextSnapshotV1": {
"description": "One generation's context accounting. `free = usable - total`, floored at zero: once provider usage lands, `total` is what was billed, which can exceed the `usable` budget the window was fit into — that budget was derived before the generation from an estimate.",
"properties": {
"categories": {
"$ref": "#/definitions/SnapshotCategoriesV1"
},
"compacted": {
"type": "boolean"
},
"effective_max_output_tokens": {
"description": "Output allocation `usable` was derived against.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"estimator": {
"description": "Which estimator produced the numbers (`heuristic` until the context-manager resolves a real tokenizer). Absent when the context-manager predates the breakdown response.",
"type": [
"string",
"null"
]
},
"free": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"model": {
"type": "string"
},
"provider": {
"type": [
"string",
"null"
]
},
"session_cost_usd": {
"description": "Running cost of the whole session in USD, accumulated across every generation step. `usage.cost_usd` is one step's bill — on providers with steep cache discounts the per-step number swings two orders of magnitude, so a chip showing it alone reads as a bouncing total.",
"format": "double",
"type": [
"number",
"null"
]
},
"session_id": {
"type": "string"
},
"step": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"summarized_head_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"timestamp": {
"format": "int64",
"type": "integer"
},
"total": {
"description": "Final request estimate: categories plus post-assembly growth.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"turn_id": {
"type": "string"
},
"usable": {
"description": "The input budget the window was fit into.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"usage": {
"anyOf": [
{
"$ref": "#/definitions/Usage"
},
{
"type": "null"
}
],
"description": "Actual provider usage for this generation, stamped after the terminal frame; absent when the provider returned none (or the generation never completed)."
}
},
"required": [
"categories",
"compacted",
"effective_max_output_tokens",
"free",
"model",
"session_id",
"step",
"timestamp",
"total",
"turn_id",
"usable"
],
"type": "object"
},
"EvalBenchmarkV1": {
"additionalProperties": false,
"properties": {
"cache_read_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"error_span_count": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"reasoning_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"sessions": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"span_count": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"total_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"trace_count": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"trace_duration_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"turns": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"wall_time_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"function_call_errors",
"function_calls",
"sessions",
"turns",
"wall_time_ms"
],
"type": "object"
},
"EvalFailureV1": {
"additionalProperties": false,
"properties": {
"code": {
"type": [
"string",
"null"
]
},
"function_id": {
"type": [
"string",
"null"
]
},
"message": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/EvalPhaseV1"
}
},
"required": [
"message",
"phase"
],
"type": "object"
},
"EvalLimitsV1": {
"additionalProperties": false,
"properties": {
"evaluation": {
"allOf": [
{
"$ref": "#/definitions/EvaluationLimitsV1"
}
],
"default": {}
},
"execution": {
"allOf": [
{
"$ref": "#/definitions/ExecutionLimitsV1"
}
],
"default": {
"invocation_timeout_seconds": 120,
"max_output_tokens_per_call": 8192,
"max_turns": 1000,
"scenario_timeout_seconds": 600
}
}
},
"type": "object"
},
"EvalModelConfigV1": {
"additionalProperties": false,
"properties": {
"mode": {
"anyOf": [
{
"$ref": "#/definitions/Mode"
},
{
"type": "null"
}
]
},
"model": {
"type": "string"
},
"provider": {
"type": [
"string",
"null"
]
},
"provider_options": {
"additionalProperties": true,
"type": [
"object",
"null"
]
},
"system_prompt_strategy": {
"allOf": [
{
"$ref": "#/definitions/SystemPromptStrategy"
}
],
"default": "enrich"
},
"thinking_level": {
"anyOf": [
{
"$ref": "#/definitions/ThinkingLevel"
},
{
"type": "null"
}
]
}
},
"required": [
"model"
],
"type": "object"
},
"EvalPhaseV1": {
"enum": [
"setup",
"send",
"await",
"collect",
"evaluate",
"limit",
"cancel"
],
"type": "string"
},
"EvalProgressV1": {
"additionalProperties": false,
"properties": {
"control_aggregate": {
"$ref": "#/definitions/VariantAggregateV1"
},
"delta": {
"$ref": "#/definitions/AggregateDeltaV1"
},
"effective_execution_order": {
"items": {
"type": "string"
},
"type": "array"
},
"order_sensitivity": {
"$ref": "#/definitions/OrderSensitivityV1"
},
"runs": {
"items": {
"$ref": "#/definitions/EvalRunReportV1"
},
"type": "array"
},
"treatment_aggregate": {
"$ref": "#/definitions/VariantAggregateV1"
}
},
"required": [
"control_aggregate",
"delta",
"effective_execution_order",
"order_sensitivity",
"runs",
"treatment_aggregate"
],
"type": "object"
},
"EvalReportV1": {
"additionalProperties": false,
"properties": {
"completed_at": {
"format": "int64",
"type": "integer"
},
"control": {
"$ref": "#/definitions/VariantArtifactV1"
},
"control_aggregate": {
"$ref": "#/definitions/VariantAggregateV1"
},
"created_at": {
"format": "int64",
"type": "integer"
},
"delta": {
"allOf": [
{
"$ref": "#/definitions/AggregateDeltaV1"
}
],
"description": "Treatment minus control. Negative efficiency deltas are improvements."
},
"dimension": {
"$ref": "#/definitions/ComparisonDimensionV1"
},
"effective_execution_order": {
"items": {
"type": "string"
},
"type": "array"
},
"eligible": {
"type": [
"boolean",
"null"
]
},
"evaluation_id": {
"type": "string"
},
"evaluator": {
"anyOf": [
{
"$ref": "#/definitions/EvaluatorArtifactV1"
},
{
"type": "null"
}
]
},
"execution_order_policy": {
"$ref": "#/definitions/ExecutionOrderV1"
},
"limits": {
"$ref": "#/definitions/EvalLimitsV1"
},
"model": {
"$ref": "#/definitions/EvalModelConfigV1"
},
"order_sensitivity": {
"$ref": "#/definitions/OrderSensitivityV1"
},
"runs": {
"items": {
"$ref": "#/definitions/EvalRunReportV1"
},
"type": "array"
},
"schema_version": {
"type": "string"
},
"shared_artifacts": {
"$ref": "#/definitions/SharedArtifactsV1"
},
"source_evaluation_id": {
"type": [
"string",
"null"
]
},
"treatment": {
"$ref": "#/definitions/VariantArtifactV1"
},
"treatment_aggregate": {
"$ref": "#/definitions/VariantAggregateV1"
}
},
"required": [
"completed_at",
"control",
"control_aggregate",
"created_at",
"delta",
"dimension",
"effective_execution_order",
"evaluation_id",
"execution_order_policy",
"limits",
"model",
"order_sensitivity",
"runs",
"schema_version",
"shared_artifacts",
"treatment",
"treatment_aggregate"
],
"type": "object"
},
"EvalResultResponseV1": {
"additionalProperties": false,
"properties": {
"progress": {
"$ref": "#/definitions/EvalProgressV1"
},
"report": {
"anyOf": [
{
"$ref": "#/definitions/EvalReportV1"
},
{
"type": "null"
}
]
},
"request": {
"$ref": "#/definitions/NormalizedEvalRequestV1"
},
"status": {
"$ref": "#/definitions/EvalStatusV1"
}
},
"required": [
"progress",
"request",
"status"
],
"type": "object"
},
"EvalRunReportV1": {
"additionalProperties": false,
"properties": {
"benchmark": {
"anyOf": [
{
"$ref": "#/definitions/EvalBenchmarkV1"
},
{
"type": "null"
}
]
},
"completed_at": {
"format": "int64",
"type": [
"integer",
"null"
]
},
"evaluation": {
"anyOf": [
{
"$ref": "#/definitions/EvaluatorResponseV1"
},
{
"type": "null"
}
]
},
"execution_position": {
"default": 0,
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"failures": {
"items": {
"$ref": "#/definitions/EvalFailureV1"
},
"type": "array"
},
"iteration": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"metrics": {
"anyOf": [
{
"$ref": "#/definitions/SessionMetricsResponseV1"
},
{
"type": "null"
}
]
},
"output": true,
"pair_position": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"passed": {
"type": [
"boolean",
"null"
]
},
"role": {
"$ref": "#/definitions/VariantRoleV1"
},
"run_id": {
"type": "string"
},
"session_id": {
"type": "string"
},
"started_at": {
"format": "int64",
"type": "integer"
},
"status": {
"$ref": "#/definitions/EvalRunStatusV1"
},
"turn_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"iteration",
"role",
"run_id",
"session_id",
"started_at",
"status"
],
"type": "object"
},
"EvalRunStatusV1": {
"enum": [
"pending",
"running",
"completed",
"failed",
"cancelled"
],
"type": "string"
},
"EvalStatusV1": {
"enum": [
"queued",
"running",
"completed",
"failed",
"cancelled"
],
"type": "string"
},
"EvalVariantV1": {
"additionalProperties": false,
"properties": {
"label": {
"type": [
"string",
"null"
]
},
"prompt": {
"type": "string"
},
"system_prompt": {
"description": "`null` disables the system prompt for this variant. An empty string keeps the harness behavior of resolving the provider or built-in prompt.",
"type": [
"string",
"null"
]
}
},
"required": [
"prompt"
],
"type": "object"
},
"EvaluationLimitsV1": {
"additionalProperties": false,
"properties": {
"max_error_spans": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"EvaluatorArtifactV1": {
"additionalProperties": false,
"properties": {
"arguments_sha256": {
"type": "string"
},
"function_id": {
"type": "string"
}
},
"required": [
"arguments_sha256",
"function_id"
],
"type": "object"
},
"EvaluatorResponseV1": {
"additionalProperties": false,
"properties": {
"details": true,
"passed": {
"type": "boolean"
},
"reason": {
"type": [
"string",
"null"
]
},
"score": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"passed"
],
"type": "object"
},
"EvaluatorSpecV1": {
"additionalProperties": false,
"properties": {
"arguments": {
"default": {}
},
"function_id": {
"type": "string"
}
},
"required": [
"function_id"
],
"type": "object"
},
"ExecutionLimitsV1": {
"additionalProperties": false,
"properties": {
"invocation_timeout_seconds": {
"default": 120,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"max_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"max_output_tokens_per_call": {
"default": 8192,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"max_total_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_turns": {
"default": 1000,
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenario_timeout_seconds": {
"default": 600,
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"type": "object"
},
"ExecutionOrderV1": {
"enum": [
"balanced_control_first",
"balanced_treatment_first"
],
"type": "string"
},
"ExposeMode": {
"description": "How allowed functions reach the model (harness.md § Exposure modes).",
"enum": [
"agent_trigger",
"native"
],
"type": "string"
},
"FunctionPolicy": {
"description": "The fail-closed dispatch policy (harness.md § Functions). Absent on the send => every call denied (a plain chat loop).",
"properties": {
"allow": {
"default": [],
"items": {
"type": "string"
},
"type": "array"
},
"deny": {
"default": [],
"items": {
"type": "string"
},
"type": "array"
},
"expose": {
"allOf": [
{
"$ref": "#/definitions/ExposeMode"
}
],
"default": "agent_trigger"
}
},
"type": "object"
},
"Mode": {
"description": "Console / send operating mode — prepends a short paragraph before the shared identity prompt. `ask` is also structural: the turn's dispatch policy is capped at the configured default policy, never widened.",
"enum": [
"ask",
"agent"
],
"type": "string"
},
"NormalizedEvalRequestV1": {
"additionalProperties": false,
"properties": {
"control": {
"$ref": "#/definitions/EvalVariantV1"
},
"dimension": {
"$ref": "#/definitions/ComparisonDimensionV1"
},
"evaluator": {
"anyOf": [
{
"$ref": "#/definitions/EvaluatorSpecV1"
},
{
"type": "null"
}
]
},
"execution_order": {
"allOf": [
{
"$ref": "#/definitions/ExecutionOrderV1"
}
],
"default": "balanced_control_first"
},
"functions": {
"$ref": "#/definitions/FunctionPolicy"
},
"limits": {
"$ref": "#/definitions/EvalLimitsV1"
},
"metadata": true,
"model": {
"$ref": "#/definitions/EvalModelConfigV1"
},
"output": {
"$ref": "#/definitions/OutputContract"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"source_evaluation_id": {
"type": [
"string",
"null"
]
},
"treatment": {
"$ref": "#/definitions/EvalVariantV1"
}
},
"required": [
"control",
"dimension",
"functions",
"limits",
"model",
"output",
"runs",
"treatment"
],
"type": "object"
},
"OrderSensitivityV1": {
"additionalProperties": false,
"properties": {
"control": {
"$ref": "#/definitions/RoleOrderSensitivityV1"
},
"detected": {
"type": "boolean"
},
"treatment": {
"$ref": "#/definitions/RoleOrderSensitivityV1"
}
},
"required": [
"control",
"detected",
"treatment"
],
"type": "object"
},
"OutputContract": {
"description": "Free text by default; `json` constrains the final answer to a JSON value, validated against `schema` when supplied.",
"oneOf": [
{
"properties": {
"type": {
"enum": [
"text"
],
"type": "string"
}
},
"required": [
"type"
],
"type": "object"
},
{
"properties": {
"schema": true,
"type": {
"enum": [
"json"
],
"type": "string"
}
},
"required": [
"type"
],
"type": "object"
}
]
},
"RoleOrderSensitivityV1": {
"additionalProperties": false,
"properties": {
"differs": {
"type": "boolean"
},
"first_passed": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"first_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"second_passed": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"second_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"differs",
"first_passed",
"first_runs",
"second_passed",
"second_runs"
],
"type": "object"
},
"SessionMetricsResponseV1": {
"additionalProperties": false,
"properties": {
"by_session": {
"items": {
"$ref": "#/definitions/SessionUsageV1"
},
"type": "array"
},
"complete": {
"type": "boolean"
},
"root_session_id": {
"type": "string"
},
"totals": {
"$ref": "#/definitions/SessionUsageTotalsV1"
},
"traces": {
"anyOf": [
{
"$ref": "#/definitions/SessionTraceMetricsV1"
},
{
"type": "null"
}
],
"description": "Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not."
}
},
"required": [
"by_session",
"complete",
"root_session_id",
"totals"
],
"type": "object"
},
"SessionTraceMetricsV1": {
"additionalProperties": false,
"properties": {
"by_session": {
"items": {
"$ref": "#/definitions/SessionTraceUsageV1"
},
"type": "array"
},
"duration_ms": {
"description": "Elapsed window from the first observed span to the last observed span.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"error_span_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"span_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"trace_count": {
"description": "Distinct traces across the root session and all descendants.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"by_session",
"duration_ms",
"error_span_count",
"span_count",
"trace_count"
],
"type": "object"
},
"SessionTraceUsageV1": {
"additionalProperties": false,
"properties": {
"depth": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"duration_ms": {
"description": "Elapsed window from the session's first observed span to its last.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"error_span_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"parent_session_id": {
"type": [
"string",
"null"
]
},
"session_id": {
"type": "string"
},
"span_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"trace_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"depth",
"duration_ms",
"error_span_count",
"session_id",
"span_count",
"trace_count"
],
"type": "object"
},
"SessionUsageTotalsV1": {
"additionalProperties": false,
"properties": {
"cache_read_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"reasoning_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"sessions": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"turns": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"function_call_errors",
"function_calls",
"sessions",
"turns"
],
"type": "object"
},
"SessionUsageV1": {
"additionalProperties": false,
"properties": {
"cache_read_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"context": {
"anyOf": [
{
"$ref": "#/definitions/ContextSnapshotV1"
},
{
"type": "null"
}
],
"description": "The session's latest per-generation context snapshot (categories, budget, usage) — absent for sessions that have not generated since snapshots landed."
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"depth": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"parent_session_id": {
"type": [
"string",
"null"
]
},
"reasoning_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"session_id": {
"type": "string"
},
"turns": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"depth",
"function_call_errors",
"function_calls",
"session_id",
"turns"
],
"type": "object"
},
"SharedArtifactsV1": {
"additionalProperties": false,
"properties": {
"function_policy_sha256": {
"type": "string"
},
"limits_sha256": {
"type": "string"
},
"model_sha256": {
"type": "string"
},
"output_sha256": {
"type": "string"
}
},
"required": [
"function_policy_sha256",
"limits_sha256",
"model_sha256",
"output_sha256"
],
"type": "object"
},
"SnapshotCategoriesV1": {
"description": "Where the request's tokens sit. Categories are assembly-time estimates; `hook_guidance` is the measured growth after assembly (pre-generate hook appends and orphan-repair patches), 0 when the request left assembly unchanged.",
"properties": {
"hook_guidance": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"messages": {
"$ref": "#/definitions/SnapshotMessagesV1"
},
"overhead": {
"description": "Provider framing plus response_format / provider_options fields.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"system_prompt": {
"description": "Final assembled system prompt: mode paragraph, identity, per-step aids, and any compaction summary section.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"tools": {
"description": "Function schemas exposed to the model.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"messages",
"overhead",
"system_prompt",
"tools"
],
"type": "object"
},
"SnapshotMessagesV1": {
"description": "Estimated tokens of the assembled window's messages, by role.",
"properties": {
"assistant": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"custom": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_result": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"user": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"assistant",
"custom",
"function_result",
"user"
],
"type": "object"
},
"SystemPromptStrategy": {
"description": "How a caller-supplied system prompt combines with the built-in identity prompt.",
"oneOf": [
{
"description": "Caller prompt replaces the built-in prompt verbatim.",
"enum": [
"override"
],
"type": "string"
},
{
"description": "Caller prompt is appended to the built-in identity prompt.",
"enum": [
"enrich"
],
"type": "string"
},
{
"description": "No system prompt is sent to the model.",
"enum": [
"disabled"
],
"type": "string"
}
]
},
"ThinkingLevel": {
"enum": [
"minimal",
"low",
"medium",
"high",
"xhigh"
],
"type": "string"
},
"Usage": {
"properties": {
"cache_read": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"input": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"reasoning": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"VariantAggregateV1": {
"additionalProperties": false,
"properties": {
"benchmarked_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"evaluated_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"median_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_error_span_count": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"median_function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"median_function_calls": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"median_input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"median_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"median_reasoning_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"median_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_span_count": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"median_total_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"median_trace_count": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"median_trace_duration_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"median_wall_time_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"pass_rate": {
"format": "double",
"type": "number"
},
"passed": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"benchmarked_runs",
"evaluated_runs",
"pass_rate",
"passed",
"runs"
],
"type": "object"
},
"VariantArtifactV1": {
"additionalProperties": false,
"properties": {
"label": {
"type": [
"string",
"null"
]
},
"prompt_sha256": {
"type": "string"
},
"system_prompt_sha256": {
"type": [
"string",
"null"
]
}
},
"required": [
"prompt_sha256"
],
"type": "object"
},
"VariantRoleV1": {
"enum": [
"control",
"treatment"
],
"type": "string"
}
},
"title": "Nullable_EvalResultResponseV1"
}
},
{
"description": "Start a durable same-model A/B evaluation. Exactly one dimension may change: `prompt` or `system_prompt`. Returns evaluation_id immediately; use eval::status, eval::result, or the eval::completed trigger instead of polling in a tight loop.",
"metadata": {},
"name": "eval::start",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ComparisonDimensionV1": {
"enum": [
"prompt",
"system_prompt"
],
"type": "string"
},
"EvalLimitsV1": {
"additionalProperties": false,
"properties": {
"evaluation": {
"allOf": [
{
"$ref": "#/definitions/EvaluationLimitsV1"
}
],
"default": {}
},
"execution": {
"allOf": [
{
"$ref": "#/definitions/ExecutionLimitsV1"
}
],
"default": {
"invocation_timeout_seconds": 120,
"max_output_tokens_per_call": 8192,
"max_turns": 1000,
"scenario_timeout_seconds": 600
}
}
},
"type": "object"
},
"EvalModelConfigV1": {
"additionalProperties": false,
"properties": {
"mode": {
"anyOf": [
{
"$ref": "#/definitions/Mode"
},
{
"type": "null"
}
]
},
"model": {
"type": "string"
},
"provider": {
"type": [
"string",
"null"
]
},
"provider_options": {
"additionalProperties": true,
"type": [
"object",
"null"
]
},
"system_prompt_strategy": {
"allOf": [
{
"$ref": "#/definitions/SystemPromptStrategy"
}
],
"default": "enrich"
},
"thinking_level": {
"anyOf": [
{
"$ref": "#/definitions/ThinkingLevel"
},
{
"type": "null"
}
]
}
},
"required": [
"model"
],
"type": "object"
},
"EvalVariantV1": {
"additionalProperties": false,
"properties": {
"label": {
"type": [
"string",
"null"
]
},
"prompt": {
"type": "string"
},
"system_prompt": {
"description": "`null` disables the system prompt for this variant. An empty string keeps the harness behavior of resolving the provider or built-in prompt.",
"type": [
"string",
"null"
]
}
},
"required": [
"prompt"
],
"type": "object"
},
"EvaluationLimitsV1": {
"additionalProperties": false,
"properties": {
"max_error_spans": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"EvaluatorSpecV1": {
"additionalProperties": false,
"properties": {
"arguments": {
"default": {}
},
"function_id": {
"type": "string"
}
},
"required": [
"function_id"
],
"type": "object"
},
"ExecutionLimitsV1": {
"additionalProperties": false,
"properties": {
"invocation_timeout_seconds": {
"default": 120,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"max_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"max_output_tokens_per_call": {
"default": 8192,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"max_total_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_turns": {
"default": 1000,
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenario_timeout_seconds": {
"default": 600,
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"type": "object"
},
"ExecutionOrderV1": {
"enum": [
"balanced_control_first",
"balanced_treatment_first"
],
"type": "string"
},
"ExposeMode": {
"description": "How allowed functions reach the model (harness.md § Exposure modes).",
"enum": [
"agent_trigger",
"native"
],
"type": "string"
},
"FunctionPolicy": {
"description": "The fail-closed dispatch policy (harness.md § Functions). Absent on the send => every call denied (a plain chat loop).",
"properties": {
"allow": {
"default": [],
"items": {
"type": "string"
},
"type": "array"
},
"deny": {
"default": [],
"items": {
"type": "string"
},
"type": "array"
},
"expose": {
"allOf": [
{
"$ref": "#/definitions/ExposeMode"
}
],
"default": "agent_trigger"
}
},
"type": "object"
},
"Mode": {
"description": "Console / send operating mode — prepends a short paragraph before the shared identity prompt. `ask` is also structural: the turn's dispatch policy is capped at the configured default policy, never widened.",
"enum": [
"ask",
"agent"
],
"type": "string"
},
"OutputContract": {
"description": "Free text by default; `json` constrains the final answer to a JSON value, validated against `schema` when supplied.",
"oneOf": [
{
"properties": {
"type": {
"enum": [
"text"
],
"type": "string"
}
},
"required": [
"type"
],
"type": "object"
},
{
"properties": {
"schema": true,
"type": {
"enum": [
"json"
],
"type": "string"
}
},
"required": [
"type"
],
"type": "object"
}
]
},
"SystemPromptStrategy": {
"description": "How a caller-supplied system prompt combines with the built-in identity prompt.",
"oneOf": [
{
"description": "Caller prompt replaces the built-in prompt verbatim.",
"enum": [
"override"
],
"type": "string"
},
{
"description": "Caller prompt is appended to the built-in identity prompt.",
"enum": [
"enrich"
],
"type": "string"
},
{
"description": "No system prompt is sent to the model.",
"enum": [
"disabled"
],
"type": "string"
}
]
},
"ThinkingLevel": {
"enum": [
"minimal",
"low",
"medium",
"high",
"xhigh"
],
"type": "string"
}
},
"properties": {
"control": {
"$ref": "#/definitions/EvalVariantV1"
},
"dimension": {
"$ref": "#/definitions/ComparisonDimensionV1"
},
"evaluator": {
"anyOf": [
{
"$ref": "#/definitions/EvaluatorSpecV1"
},
{
"type": "null"
}
]
},
"execution_order": {
"anyOf": [
{
"$ref": "#/definitions/ExecutionOrderV1"
},
{
"type": "null"
}
]
},
"functions": {
"anyOf": [
{
"$ref": "#/definitions/FunctionPolicy"
},
{
"type": "null"
}
],
"description": "Shared function policy for both variants. Omitted means deny all."
},
"limits": {
"allOf": [
{
"$ref": "#/definitions/EvalLimitsV1"
}
],
"default": {
"evaluation": {},
"execution": {
"invocation_timeout_seconds": 120,
"max_output_tokens_per_call": 8192,
"max_turns": 1000,
"scenario_timeout_seconds": 600
}
}
},
"metadata": {
"description": "Shared harness metadata, including an optional filesystem scope."
},
"model": {
"$ref": "#/definitions/EvalModelConfigV1"
},
"output": {
"anyOf": [
{
"$ref": "#/definitions/OutputContract"
},
{
"type": "null"
}
],
"description": "Shared output contract for both variants. Omitted means text."
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"treatment": {
"$ref": "#/definitions/EvalVariantV1"
}
},
"required": [
"control",
"dimension",
"model",
"treatment"
],
"title": "EvalStartRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"EvalStatusV1": {
"enum": [
"queued",
"running",
"completed",
"failed",
"cancelled"
],
"type": "string"
}
},
"properties": {
"evaluation_id": {
"type": "string"
},
"status": {
"$ref": "#/definitions/EvalStatusV1"
}
},
"required": [
"evaluation_id",
"status"
],
"title": "EvalStartResponseV1",
"type": "object"
}
},
{
"description": "Read evaluation progress without loading the complete report.",
"metadata": {},
"name": "eval::status",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"evaluation_id": {
"type": "string"
}
},
"required": [
"evaluation_id"
],
"title": "EvaluationIdRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"anyOf": [
{
"$ref": "#/definitions/EvalStatusResponseV1"
},
{
"type": "null"
}
],
"definitions": {
"ActiveRunV1": {
"additionalProperties": false,
"properties": {
"iteration": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"role": {
"$ref": "#/definitions/VariantRoleV1"
},
"run_id": {
"type": "string"
},
"session_id": {
"type": "string"
},
"started_at": {
"format": "int64",
"type": "integer"
},
"turn_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"iteration",
"role",
"run_id",
"session_id",
"started_at"
],
"type": "object"
},
"EvalStatusResponseV1": {
"additionalProperties": false,
"properties": {
"active": {
"anyOf": [
{
"$ref": "#/definitions/ActiveRunV1"
},
{
"type": "null"
}
]
},
"completed_at": {
"format": "int64",
"type": [
"integer",
"null"
]
},
"created_at": {
"format": "int64",
"type": "integer"
},
"error": {
"type": [
"string",
"null"
]
},
"evaluation_id": {
"type": "string"
},
"failed_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"passed_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"status": {
"$ref": "#/definitions/EvalStatusV1"
},
"terminal_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"total_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"updated_at": {
"format": "int64",
"type": "integer"
}
},
"required": [
"created_at",
"evaluation_id",
"failed_runs",
"passed_runs",
"status",
"terminal_runs",
"total_runs",
"updated_at"
],
"type": "object"
},
"EvalStatusV1": {
"enum": [
"queued",
"running",
"completed",
"failed",
"cancelled"
],
"type": "string"
},
"VariantRoleV1": {
"enum": [
"control",
"treatment"
],
"type": "string"
}
},
"title": "Nullable_EvalStatusResponseV1"
}
},
{
"description": "Internal durable evaluation step.",
"metadata": {
"internal": true,
"trace_hidden": true
},
"name": "eval::step",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"evaluation_id": {
"type": "string"
},
"step": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"evaluation_id",
"step"
],
"title": "StepRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"EvalStatusV1": {
"enum": [
"queued",
"running",
"completed",
"failed",
"cancelled"
],
"type": "string"
}
},
"properties": {
"skipped": {
"type": "boolean"
},
"status": {
"$ref": "#/definitions/EvalStatusV1"
}
},
"required": [
"skipped",
"status"
],
"title": "StepResponseV1",
"type": "object"
}
},
{
"description": "Internal recovery and timeout sweep.",
"metadata": {
"internal": true,
"trace_hidden": true
},
"name": "eval::sweep",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"scheduled_at": {
"default": null,
"format": "int64",
"type": [
"integer",
"null"
]
},
"scheduled_time": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "SweepEventV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"swept": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"swept"
],
"title": "SweepResponseV1",
"type": "object"
}
},
{
"description": "Serve the eval worker's injected console UI assets (content function for its console:script / console:style triggers).",
"metadata": {
"internal": true
},
"name": "eval::ui-content",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input of the content function: the console asks for one asset by path.",
"properties": {
"path": {
"description": "The asset path from the trigger config (e.g. `state/page.js`).",
"type": "string"
}
},
"required": [
"path"
],
"title": "UiContentInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Output of the content function.",
"properties": {
"content": {
"description": "The asset source, verbatim.",
"type": "string"
},
"content_type": {
"description": "MIME type the console should serve the asset with.",
"type": "string"
}
},
"required": [
"content",
"content_type"
],
"title": "UiContentResult",
"type": "object"
}
}
],
"triggers": [
{
"description": "An evaluation reached a terminal status.",
"invocation_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"evaluation_id": {
"type": [
"string",
"null"
]
}
},
"title": "CompletedBindingConfigV1",
"type": "object"
},
"metadata": {},
"name": "eval::completed",
"return_schema": {}
}
]
}