# eval > Compare live objective metrics for 2-5 root sessions; retain durable prompt experiments as an advanced surface. | field | value | |-------|-------| | version | 0.1.0 | | type | binary | | repo | https://github.com/iii-hq/workers | | supported_targets | x86_64-apple-darwin, aarch64-apple-darwin, i686-pc-windows-msvc, x86_64-pc-windows-msvc, aarch64-pc-windows-msvc, x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu, x86_64-unknown-linux-musl, armv7-unknown-linux-gnueabihf | | author | iii | ## installation ```sh iii worker add eval@0.1.0 ``` ## dependencies - `harness` @ `^1.5.5` - `state` @ `^0.21.3` - `queue` @ `^0.2.0` - `cron` @ `^0.21.0` - `iii-observability` @ `^0.21.6` ## readme # eval `eval` compares two prompt or system-prompt variants under the same model and execution settings. It runs control and treatment through the iii harness, evaluates each result with an iii function, and reports correctness together with tokens, cost, latency, function-call, trace, and span metrics. ## Start an evaluation ```json { "dimension": "prompt", "model": { "model": "codex/gpt-5.6-luna", "provider": "openai-codex", "system_prompt_strategy": "override" }, "control": { "label": "baseline", "prompt": "Reply with exactly OK.", "system_prompt": "Follow the user request exactly." }, "treatment": { "label": "candidate", "prompt": "Return exactly the text OK.", "system_prompt": "Follow the user request exactly." }, "evaluator": { "function_id": "eval::assert::exact", "arguments": { "expected": "OK" } } } ``` Call `eval::start` with the request. It returns an `evaluation_id` immediately. Use `eval::status` for occasional progress checks and `eval::result` for the terminal report, or bind to the `eval::completed` trigger type. Three runs per variant are scheduled by default. Their order alternates by pair to reduce order bias. A candidate is eligible only when every treatment run passes and its pass count does not regress against control. Efficiency metrics are descriptive and never select a winner automatically. ## Public functions - `eval::start` — validate, persist, and enqueue an evaluation. - `eval::list` — list recent evaluations as lightweight summaries. - `eval::status` — inspect progress without loading the full report. - `eval::result` — read the normalized request and terminal comparison report. - `eval::cancel` — cancel the active harness session and remaining runs. - `eval::delete` — delete a terminal evaluation and its session indexes. - `eval::assert::exact` — built-in deep JSON/string equality evaluator. Evaluator functions receive the output, `harness::metrics`, run identity, and caller-supplied arguments. They return `{ passed, score?, reason?, details? }`. Evaluators should be deterministic and idempotent because durable delivery is at-least-once. ## Console UI When the console worker is running, `eval` injects an **eval** page at `#/ext/eval-benchmarks`. The page creates prompt or system-prompt comparisons, tracks durable progress, restores recent reports after reload, and compares correctness, token, cost, latency, function-call, trace, and span metrics. The model picker reads the live `router::models::list` catalog and falls back to manual model/provider entry when the catalog is unavailable. Exact-value evaluation is built in; any evaluator function can also be selected by id with JSON arguments. Harness policies and output/metadata options remain collapsed until needed. ## Boundaries The worker intentionally does not implement an agent loop, model router, metrics collector, trace collector, test DSL, or LLM judge. Those concerns remain in the harness, engine, and user-provided evaluator functions. ## api reference ```json { "functions": [ { "description": "Built-in deterministic evaluator: deep-compare output with arguments.expected.", "metadata": {}, "name": "eval::assert::exact", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "SessionMetricsResponseV1": { "additionalProperties": false, "properties": { "by_session": { "items": { "$ref": "#/definitions/SessionUsageV1" }, "type": "array" }, "complete": { "type": "boolean" }, "root_session_id": { "type": "string" }, "totals": { "$ref": "#/definitions/SessionUsageTotalsV1" }, "traces": { "anyOf": [ { "$ref": "#/definitions/SessionTraceMetricsV1" }, { "type": "null" } ], "description": "Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not." } }, "required": [ "by_session", "complete", "root_session_id", "totals" ], "type": "object" }, "SessionTraceMetricsV1": { "additionalProperties": false, "properties": { "by_session": { "items": { "$ref": "#/definitions/SessionTraceUsageV1" }, "type": "array" }, "duration_ms": { "description": "Elapsed window from the first observed span to the last observed span.", "format": "uint64", "minimum": 0, "type": "integer" }, "error_span_count": { "format": "uint64", "minimum": 0, "type": "integer" }, "span_count": { "format": "uint64", "minimum": 0, "type": "integer" }, "trace_count": { "description": "Distinct traces across the root session and all descendants.", "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "by_session", "duration_ms", "error_span_count", "span_count", "trace_count" ], "type": "object" }, "SessionTraceUsageV1": { "additionalProperties": false, "properties": { "depth": { "format": "uint32", "minimum": 0, "type": "integer" }, "duration_ms": { "description": "Elapsed window from the session's first observed span to its last.", "format": "uint64", "minimum": 0, "type": "integer" }, "error_span_count": { "format": "uint64", "minimum": 0, "type": "integer" }, "parent_session_id": { "type": [ "string", "null" ] }, "session_id": { "type": "string" }, "span_count": { "format": "uint64", "minimum": 0, "type": "integer" }, "trace_count": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "depth", "duration_ms", "error_span_count", "session_id", "span_count", "trace_count" ], "type": "object" }, "SessionUsageTotalsV1": { "additionalProperties": false, "properties": { "cache_read_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cache_write_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cost_usd": { "format": "double", "type": [ "number", "null" ] }, "function_call_errors": { "format": "uint64", "minimum": 0, "type": "integer" }, "function_calls": { "format": "uint64", "minimum": 0, "type": "integer" }, "input_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "output_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "reasoning_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "sessions": { "format": "uint64", "minimum": 0, "type": "integer" }, "turns": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "function_call_errors", "function_calls", "sessions", "turns" ], "type": "object" }, "SessionUsageV1": { "additionalProperties": false, "properties": { "cache_read_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cache_write_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cost_usd": { "format": "double", "type": [ "number", "null" ] }, "depth": { "format": "uint32", "minimum": 0, "type": "integer" }, "function_call_errors": { "format": "uint64", "minimum": 0, "type": "integer" }, "function_calls": { "format": "uint64", "minimum": 0, "type": "integer" }, "input_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "output_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "parent_session_id": { "type": [ "string", "null" ] }, "reasoning_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "session_id": { "type": "string" }, "turns": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "depth", "function_call_errors", "function_calls", "session_id", "turns" ], "type": "object" }, "VariantRoleV1": { "enum": [ "control", "treatment" ], "type": "string" } }, "properties": { "arguments": { "default": {} }, "evaluation_id": { "type": "string" }, "metrics": { "$ref": "#/definitions/SessionMetricsResponseV1" }, "output": true, "role": { "$ref": "#/definitions/VariantRoleV1" }, "run_id": { "type": "string" }, "session_id": { "type": "string" } }, "required": [ "evaluation_id", "metrics", "output", "role", "run_id", "session_id" ], "title": "EvaluatorInputV1", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": false, "properties": { "details": true, "passed": { "type": "boolean" }, "reason": { "type": [ "string", "null" ] }, "score": { "format": "double", "type": [ "number", "null" ] } }, "required": [ "passed" ], "title": "EvaluatorResponseV1", "type": "object" } }, { "description": "Built-in deterministic evaluator: compare text after normalizing case, whitespace, and surrounding punctuation.", "metadata": {}, "name": "eval::assert::normalized_text", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "SessionMetricsResponseV1": { "additionalProperties": false, "properties": { "by_session": { "items": { "$ref": "#/definitions/SessionUsageV1" }, "type": "array" }, "complete": { "type": "boolean" }, "root_session_id": { "type": "string" }, "totals": { "$ref": "#/definitions/SessionUsageTotalsV1" }, "traces": { "anyOf": [ { "$ref": "#/definitions/SessionTraceMetricsV1" }, { "type": "null" } ], "description": "Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not." } }, "required": [ "by_session", "complete", "root_session_id", "totals" ], "type": "object" }, "SessionTraceMetricsV1": { "additionalProperties": false, "properties": { "by_session": { "items": { "$ref": "#/definitions/SessionTraceUsageV1" }, "type": "array" }, "duration_ms": { "description": "Elapsed window from the first observed span to the last observed span.", "format": "uint64", "minimum": 0, "type": "integer" }, "error_span_count": { "format": "uint64", "minimum": 0, "type": "integer" }, "span_count": { "format": "uint64", "minimum": 0, "type": "integer" }, "trace_count": { "description": "Distinct traces across the root session and all descendants.", "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "by_session", "duration_ms", "error_span_count", "span_count", "trace_count" ], "type": "object" }, "SessionTraceUsageV1": { "additionalProperties": false, "properties": { "depth": { "format": "uint32", "minimum": 0, "type": "integer" }, "duration_ms": { "description": "Elapsed window from the session's first observed span to its last.", "format": "uint64", "minimum": 0, "type": "integer" }, "error_span_count": { "format": "uint64", "minimum": 0, "type": "integer" }, "parent_session_id": { "type": [ "string", "null" ] }, "session_id": { "type": "string" }, "span_count": { "format": "uint64", "minimum": 0, "type": "integer" }, "trace_count": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "depth", "duration_ms", "error_span_count", "session_id", "span_count", "trace_count" ], "type": "object" }, "SessionUsageTotalsV1": { "additionalProperties": false, "properties": { "cache_read_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cache_write_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cost_usd": { "format": "double", "type": [ "number", "null" ] }, "function_call_errors": { "format": "uint64", "minimum": 0, "type": "integer" }, "function_calls": { "format": "uint64", "minimum": 0, "type": "integer" }, "input_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "output_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "reasoning_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "sessions": { "format": "uint64", "minimum": 0, "type": "integer" }, "turns": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "function_call_errors", "function_calls", "sessions", "turns" ], "type": "object" }, "SessionUsageV1": { "additionalProperties": false, "properties": { "cache_read_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cache_write_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cost_usd": { "format": "double", "type": [ "number", "null" ] }, "depth": { "format": "uint32", "minimum": 0, "type": "integer" }, "function_call_errors": { "format": "uint64", "minimum": 0, "type": "integer" }, "function_calls": { "format": "uint64", "minimum": 0, "type": "integer" }, "input_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "output_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "parent_session_id": { "type": [ "string", "null" ] }, "reasoning_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "session_id": { "type": "string" }, "turns": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "depth", "function_call_errors", "function_calls", "session_id", "turns" ], "type": "object" }, "VariantRoleV1": { "enum": [ "control", "treatment" ], "type": "string" } }, "properties": { "arguments": { "default": {} }, "evaluation_id": { "type": "string" }, "metrics": { "$ref": "#/definitions/SessionMetricsResponseV1" }, "output": true, "role": { "$ref": "#/definitions/VariantRoleV1" }, "run_id": { "type": "string" }, "session_id": { "type": "string" } }, "required": [ "evaluation_id", "metrics", "output", "role", "run_id", "session_id" ], "title": "EvaluatorInputV1", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": false, "properties": { "details": true, "passed": { "type": "boolean" }, "reason": { "type": [ "string", "null" ] }, "score": { "format": "double", "type": [ "number", "null" ] } }, "required": [ "passed" ], "title": "EvaluatorResponseV1", "type": "object" } }, { "description": "Cancel an evaluation and its active harness turn.", "metadata": {}, "name": "eval::cancel", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "evaluation_id": { "type": "string" } }, "required": [ "evaluation_id" ], "title": "EvaluationIdRequestV1", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": false, "definitions": { "EvalStatusV1": { "enum": [ "queued", "running", "completed", "failed", "cancelled" ], "type": "string" } }, "properties": { "cancelled": { "type": "boolean" }, "status": { "$ref": "#/definitions/EvalStatusV1" } }, "required": [ "cancelled", "status" ], "title": "EvalCancelResponseV1", "type": "object" } }, { "description": "Delete a terminal evaluation report and its session indexes.", "metadata": {}, "name": "eval::delete", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "evaluation_id": { "type": "string" } }, "required": [ "evaluation_id" ], "title": "EvaluationIdRequestV1", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": false, "properties": { "deleted": { "type": "boolean" } }, "required": [ "deleted" ], "title": "EvalDeleteResponseV1", "type": "object" } }, { "description": "List recent evaluations as lightweight summaries, newest first.", "metadata": {}, "name": "eval::list", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "limit": { "format": "uint32", "minimum": 0, "type": [ "integer", "null" ] } }, "title": "EvalListRequestV1", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": false, "definitions": { "ComparisonDimensionV1": { "enum": [ "prompt", "system_prompt" ], "type": "string" }, "EvalStatusV1": { "enum": [ "queued", "running", "completed", "failed", "cancelled" ], "type": "string" }, "EvalSummaryV1": { "additionalProperties": false, "properties": { "completed_at": { "format": "int64", "type": [ "integer", "null" ] }, "control_label": { "type": [ "string", "null" ] }, "created_at": { "format": "int64", "type": "integer" }, "dimension": { "$ref": "#/definitions/ComparisonDimensionV1" }, "eligible": { "type": [ "boolean", "null" ] }, "error": { "type": [ "string", "null" ] }, "evaluation_id": { "type": "string" }, "failed_runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "model": { "type": "string" }, "passed_runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "provider": { "type": [ "string", "null" ] }, "status": { "$ref": "#/definitions/EvalStatusV1" }, "terminal_runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "total_runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "treatment_label": { "type": [ "string", "null" ] }, "updated_at": { "format": "int64", "type": "integer" } }, "required": [ "created_at", "dimension", "evaluation_id", "failed_runs", "model", "passed_runs", "status", "terminal_runs", "total_runs", "updated_at" ], "type": "object" } }, "properties": { "evaluations": { "items": { "$ref": "#/definitions/EvalSummaryV1" }, "type": "array" } }, "required": [ "evaluations" ], "title": "EvalListResponseV1", "type": "object" } }, { "description": "Internal harness turn-completed wake-up.", "metadata": { "internal": true, "trace_hidden": true }, "name": "eval::on-turn-completed", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "session_id": { "default": "", "type": "string" }, "terminal": { "default": false, "type": "boolean" } }, "title": "WakeEventV1", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": false, "properties": { "woke": { "type": "boolean" } }, "required": [ "woke" ], "title": "WakeResponseV1", "type": "object" } }, { "description": "Repeat a terminal evaluation from its persisted request. Set reverse_order to invert the balanced A/B order. The rerun receives a new evaluation ID and fresh sessions.", "metadata": {}, "name": "eval::rerun", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "evaluation_id": { "type": "string" }, "reverse_order": { "default": false, "type": "boolean" } }, "required": [ "evaluation_id" ], "title": "EvalRerunRequestV1", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": false, "definitions": { "EvalStatusV1": { "enum": [ "queued", "running", "completed", "failed", "cancelled" ], "type": "string" } }, "properties": { "evaluation_id": { "type": "string" }, "status": { "$ref": "#/definitions/EvalStatusV1" } }, "required": [ "evaluation_id", "status" ], "title": "EvalStartResponseV1", "type": "object" } }, { "description": "Read an evaluation result. The report is present only after terminal completion.", "metadata": {}, "name": "eval::result", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "evaluation_id": { "type": "string" } }, "required": [ "evaluation_id" ], "title": "EvaluationIdRequestV1", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "anyOf": [ { "$ref": "#/definitions/EvalResultResponseV1" }, { "type": "null" } ], "definitions": { "AggregateDeltaV1": { "additionalProperties": false, "properties": { "median_cost_usd": { "format": "double", "type": [ "number", "null" ] }, "median_error_span_count": { "format": "double", "type": [ "number", "null" ] }, "median_function_call_errors": { "format": "double", "type": [ "number", "null" ] }, "median_function_calls": { "format": "double", "type": [ "number", "null" ] }, "median_reasoning_tokens": { "format": "double", "type": [ "number", "null" ] }, "median_score": { "format": "double", "type": [ "number", "null" ] }, "median_span_count": { "format": "double", "type": [ "number", "null" ] }, "median_total_tokens": { "format": "double", "type": [ "number", "null" ] }, "median_trace_count": { "format": "double", "type": [ "number", "null" ] }, "median_trace_duration_ms": { "format": "double", "type": [ "number", "null" ] }, "median_wall_time_ms": { "format": "double", "type": [ "number", "null" ] }, "pass_rate": { "format": "double", "type": "number" } }, "required": [ "pass_rate" ], "type": "object" }, "ComparisonDimensionV1": { "enum": [ "prompt", "system_prompt" ], "type": "string" }, "EvalBenchmarkV1": { "additionalProperties": false, "properties": { "cache_read_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cache_write_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cost_usd": { "format": "double", "type": [ "number", "null" ] }, "error_span_count": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "function_call_errors": { "format": "uint64", "minimum": 0, "type": "integer" }, "function_calls": { "format": "uint64", "minimum": 0, "type": "integer" }, "input_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "output_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "reasoning_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "sessions": { "format": "uint64", "minimum": 0, "type": "integer" }, "span_count": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "total_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "trace_count": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "trace_duration_ms": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "turns": { "format": "uint64", "minimum": 0, "type": "integer" }, "wall_time_ms": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "function_call_errors", "function_calls", "sessions", "turns", "wall_time_ms" ], "type": "object" }, "EvalFailureV1": { "additionalProperties": false, "properties": { "code": { "type": [ "string", "null" ] }, "function_id": { "type": [ "string", "null" ] }, "message": { "type": "string" }, "phase": { "$ref": "#/definitions/EvalPhaseV1" } }, "required": [ "message", "phase" ], "type": "object" }, "EvalLimitsV1": { "additionalProperties": false, "properties": { "evaluation": { "allOf": [ { "$ref": "#/definitions/EvaluationLimitsV1" } ], "default": {} }, "execution": { "allOf": [ { "$ref": "#/definitions/ExecutionLimitsV1" } ], "default": { "invocation_timeout_seconds": 120, "max_output_tokens_per_call": 8192, "max_turns": 1000, "scenario_timeout_seconds": 600 } } }, "type": "object" }, "EvalModelConfigV1": { "additionalProperties": false, "properties": { "mode": { "anyOf": [ { "$ref": "#/definitions/Mode" }, { "type": "null" } ] }, "model": { "type": "string" }, "provider": { "type": [ "string", "null" ] }, "provider_options": { "additionalProperties": true, "type": [ "object", "null" ] }, "system_prompt_strategy": { "allOf": [ { "$ref": "#/definitions/SystemPromptStrategy" } ], "default": "enrich" }, "thinking_level": { "anyOf": [ { "$ref": "#/definitions/ThinkingLevel" }, { "type": "null" } ] } }, "required": [ "model" ], "type": "object" }, "EvalPhaseV1": { "enum": [ "setup", "send", "await", "collect", "evaluate", "limit", "cancel" ], "type": "string" }, "EvalProgressV1": { "additionalProperties": false, "properties": { "control_aggregate": { "$ref": "#/definitions/VariantAggregateV1" }, "delta": { "$ref": "#/definitions/AggregateDeltaV1" }, "effective_execution_order": { "items": { "type": "string" }, "type": "array" }, "order_sensitivity": { "$ref": "#/definitions/OrderSensitivityV1" }, "runs": { "items": { "$ref": "#/definitions/EvalRunReportV1" }, "type": "array" }, "treatment_aggregate": { "$ref": "#/definitions/VariantAggregateV1" } }, "required": [ "control_aggregate", "delta", "effective_execution_order", "order_sensitivity", "runs", "treatment_aggregate" ], "type": "object" }, "EvalReportV1": { "additionalProperties": false, "properties": { "completed_at": { "format": "int64", "type": "integer" }, "control": { "$ref": "#/definitions/VariantArtifactV1" }, "control_aggregate": { "$ref": "#/definitions/VariantAggregateV1" }, "created_at": { "format": "int64", "type": "integer" }, "delta": { "allOf": [ { "$ref": "#/definitions/AggregateDeltaV1" } ], "description": "Treatment minus control. Negative efficiency deltas are improvements." }, "dimension": { "$ref": "#/definitions/ComparisonDimensionV1" }, "effective_execution_order": { "items": { "type": "string" }, "type": "array" }, "eligible": { "type": [ "boolean", "null" ] }, "evaluation_id": { "type": "string" }, "evaluator": { "anyOf": [ { "$ref": "#/definitions/EvaluatorArtifactV1" }, { "type": "null" } ] }, "execution_order_policy": { "$ref": "#/definitions/ExecutionOrderV1" }, "limits": { "$ref": "#/definitions/EvalLimitsV1" }, "model": { "$ref": "#/definitions/EvalModelConfigV1" }, "order_sensitivity": { "$ref": "#/definitions/OrderSensitivityV1" }, "runs": { "items": { "$ref": "#/definitions/EvalRunReportV1" }, "type": "array" }, "schema_version": { "type": "string" }, "shared_artifacts": { "$ref": "#/definitions/SharedArtifactsV1" }, "source_evaluation_id": { "type": [ "string", "null" ] }, "treatment": { "$ref": "#/definitions/VariantArtifactV1" }, "treatment_aggregate": { "$ref": "#/definitions/VariantAggregateV1" } }, "required": [ "completed_at", "control", "control_aggregate", "created_at", "delta", "dimension", "effective_execution_order", "evaluation_id", "execution_order_policy", "limits", "model", "order_sensitivity", "runs", "schema_version", "shared_artifacts", "treatment", "treatment_aggregate" ], "type": "object" }, "EvalResultResponseV1": { "additionalProperties": false, "properties": { "progress": { "$ref": "#/definitions/EvalProgressV1" }, "report": { "anyOf": [ { "$ref": "#/definitions/EvalReportV1" }, { "type": "null" } ] }, "request": { "$ref": "#/definitions/NormalizedEvalRequestV1" }, "status": { "$ref": "#/definitions/EvalStatusV1" } }, "required": [ "progress", "request", "status" ], "type": "object" }, "EvalRunReportV1": { "additionalProperties": false, "properties": { "benchmark": { "anyOf": [ { "$ref": "#/definitions/EvalBenchmarkV1" }, { "type": "null" } ] }, "completed_at": { "format": "int64", "type": [ "integer", "null" ] }, "evaluation": { "anyOf": [ { "$ref": "#/definitions/EvaluatorResponseV1" }, { "type": "null" } ] }, "execution_position": { "default": 0, "format": "uint32", "minimum": 0, "type": "integer" }, "failures": { "items": { "$ref": "#/definitions/EvalFailureV1" }, "type": "array" }, "iteration": { "format": "uint32", "minimum": 0, "type": "integer" }, "metrics": { "anyOf": [ { "$ref": "#/definitions/SessionMetricsResponseV1" }, { "type": "null" } ] }, "output": true, "pair_position": { "default": 0, "format": "uint8", "minimum": 0, "type": "integer" }, "passed": { "type": [ "boolean", "null" ] }, "role": { "$ref": "#/definitions/VariantRoleV1" }, "run_id": { "type": "string" }, "session_id": { "type": "string" }, "started_at": { "format": "int64", "type": "integer" }, "status": { "$ref": "#/definitions/EvalRunStatusV1" }, "turn_id": { "type": [ "string", "null" ] } }, "required": [ "iteration", "role", "run_id", "session_id", "started_at", "status" ], "type": "object" }, "EvalRunStatusV1": { "enum": [ "pending", "running", "completed", "failed", "cancelled" ], "type": "string" }, "EvalStatusV1": { "enum": [ "queued", "running", "completed", "failed", "cancelled" ], "type": "string" }, "EvalVariantV1": { "additionalProperties": false, "properties": { "label": { "type": [ "string", "null" ] }, "prompt": { "type": "string" }, "system_prompt": { "description": "`null` disables the system prompt for this variant. An empty string keeps the harness behavior of resolving the provider or built-in prompt.", "type": [ "string", "null" ] } }, "required": [ "prompt" ], "type": "object" }, "EvaluationLimitsV1": { "additionalProperties": false, "properties": { "max_error_spans": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "max_function_call_errors": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] } }, "type": "object" }, "EvaluatorArtifactV1": { "additionalProperties": false, "properties": { "arguments_sha256": { "type": "string" }, "function_id": { "type": "string" } }, "required": [ "arguments_sha256", "function_id" ], "type": "object" }, "EvaluatorResponseV1": { "additionalProperties": false, "properties": { "details": true, "passed": { "type": "boolean" }, "reason": { "type": [ "string", "null" ] }, "score": { "format": "double", "type": [ "number", "null" ] } }, "required": [ "passed" ], "type": "object" }, "EvaluatorSpecV1": { "additionalProperties": false, "properties": { "arguments": { "default": {} }, "function_id": { "type": "string" } }, "required": [ "function_id" ], "type": "object" }, "ExecutionLimitsV1": { "additionalProperties": false, "properties": { "invocation_timeout_seconds": { "default": 120, "format": "uint64", "minimum": 0, "type": "integer" }, "max_cost_usd": { "format": "double", "type": [ "number", "null" ] }, "max_output_tokens_per_call": { "default": 8192, "format": "uint64", "minimum": 0, "type": "integer" }, "max_total_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "max_turns": { "default": 1000, "format": "uint32", "minimum": 0, "type": "integer" }, "scenario_timeout_seconds": { "default": 600, "format": "uint64", "minimum": 0, "type": "integer" } }, "type": "object" }, "ExecutionOrderV1": { "enum": [ "balanced_control_first", "balanced_treatment_first" ], "type": "string" }, "ExposeMode": { "description": "How allowed functions reach the model (harness.md § Exposure modes).", "enum": [ "agent_trigger", "native" ], "type": "string" }, "FunctionPolicy": { "description": "The fail-closed dispatch policy (harness.md § Functions). Absent on the send => every call denied (a plain chat loop).", "properties": { "allow": { "default": [], "items": { "type": "string" }, "type": "array" }, "deny": { "default": [], "items": { "type": "string" }, "type": "array" }, "expose": { "allOf": [ { "$ref": "#/definitions/ExposeMode" } ], "default": "agent_trigger" } }, "type": "object" }, "Mode": { "description": "Console / send operating mode — prepends a short paragraph before the shared identity prompt. `ask` is also structural: the turn's dispatch policy is capped at the configured default policy, never widened.", "enum": [ "ask", "agent" ], "type": "string" }, "NormalizedEvalRequestV1": { "additionalProperties": false, "properties": { "control": { "$ref": "#/definitions/EvalVariantV1" }, "dimension": { "$ref": "#/definitions/ComparisonDimensionV1" }, "evaluator": { "anyOf": [ { "$ref": "#/definitions/EvaluatorSpecV1" }, { "type": "null" } ] }, "execution_order": { "allOf": [ { "$ref": "#/definitions/ExecutionOrderV1" } ], "default": "balanced_control_first" }, "functions": { "$ref": "#/definitions/FunctionPolicy" }, "limits": { "$ref": "#/definitions/EvalLimitsV1" }, "metadata": true, "model": { "$ref": "#/definitions/EvalModelConfigV1" }, "output": { "$ref": "#/definitions/OutputContract" }, "runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "source_evaluation_id": { "type": [ "string", "null" ] }, "treatment": { "$ref": "#/definitions/EvalVariantV1" } }, "required": [ "control", "dimension", "functions", "limits", "model", "output", "runs", "treatment" ], "type": "object" }, "OrderSensitivityV1": { "additionalProperties": false, "properties": { "control": { "$ref": "#/definitions/RoleOrderSensitivityV1" }, "detected": { "type": "boolean" }, "treatment": { "$ref": "#/definitions/RoleOrderSensitivityV1" } }, "required": [ "control", "detected", "treatment" ], "type": "object" }, "OutputContract": { "description": "Free text by default; `json` constrains the final answer to a JSON value, validated against `schema` when supplied.", "oneOf": [ { "properties": { "type": { "enum": [ "text" ], "type": "string" } }, "required": [ "type" ], "type": "object" }, { "properties": { "schema": true, "type": { "enum": [ "json" ], "type": "string" } }, "required": [ "type" ], "type": "object" } ] }, "RoleOrderSensitivityV1": { "additionalProperties": false, "properties": { "differs": { "type": "boolean" }, "first_passed": { "format": "uint32", "minimum": 0, "type": "integer" }, "first_runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "second_passed": { "format": "uint32", "minimum": 0, "type": "integer" }, "second_runs": { "format": "uint32", "minimum": 0, "type": "integer" } }, "required": [ "differs", "first_passed", "first_runs", "second_passed", "second_runs" ], "type": "object" }, "SessionMetricsResponseV1": { "additionalProperties": false, "properties": { "by_session": { "items": { "$ref": "#/definitions/SessionUsageV1" }, "type": "array" }, "complete": { "type": "boolean" }, "root_session_id": { "type": "string" }, "totals": { "$ref": "#/definitions/SessionUsageTotalsV1" }, "traces": { "anyOf": [ { "$ref": "#/definitions/SessionTraceMetricsV1" }, { "type": "null" } ], "description": "Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not." } }, "required": [ "by_session", "complete", "root_session_id", "totals" ], "type": "object" }, "SessionTraceMetricsV1": { "additionalProperties": false, "properties": { "by_session": { "items": { "$ref": "#/definitions/SessionTraceUsageV1" }, "type": "array" }, "duration_ms": { "description": "Elapsed window from the first observed span to the last observed span.", "format": "uint64", "minimum": 0, "type": "integer" }, "error_span_count": { "format": "uint64", "minimum": 0, "type": "integer" }, "span_count": { "format": "uint64", "minimum": 0, "type": "integer" }, "trace_count": { "description": "Distinct traces across the root session and all descendants.", "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "by_session", "duration_ms", "error_span_count", "span_count", "trace_count" ], "type": "object" }, "SessionTraceUsageV1": { "additionalProperties": false, "properties": { "depth": { "format": "uint32", "minimum": 0, "type": "integer" }, "duration_ms": { "description": "Elapsed window from the session's first observed span to its last.", "format": "uint64", "minimum": 0, "type": "integer" }, "error_span_count": { "format": "uint64", "minimum": 0, "type": "integer" }, "parent_session_id": { "type": [ "string", "null" ] }, "session_id": { "type": "string" }, "span_count": { "format": "uint64", "minimum": 0, "type": "integer" }, "trace_count": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "depth", "duration_ms", "error_span_count", "session_id", "span_count", "trace_count" ], "type": "object" }, "SessionUsageTotalsV1": { "additionalProperties": false, "properties": { "cache_read_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cache_write_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cost_usd": { "format": "double", "type": [ "number", "null" ] }, "function_call_errors": { "format": "uint64", "minimum": 0, "type": "integer" }, "function_calls": { "format": "uint64", "minimum": 0, "type": "integer" }, "input_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "output_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "reasoning_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "sessions": { "format": "uint64", "minimum": 0, "type": "integer" }, "turns": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "function_call_errors", "function_calls", "sessions", "turns" ], "type": "object" }, "SessionUsageV1": { "additionalProperties": false, "properties": { "cache_read_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cache_write_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cost_usd": { "format": "double", "type": [ "number", "null" ] }, "depth": { "format": "uint32", "minimum": 0, "type": "integer" }, "function_call_errors": { "format": "uint64", "minimum": 0, "type": "integer" }, "function_calls": { "format": "uint64", "minimum": 0, "type": "integer" }, "input_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "output_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "parent_session_id": { "type": [ "string", "null" ] }, "reasoning_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "session_id": { "type": "string" }, "turns": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "depth", "function_call_errors", "function_calls", "session_id", "turns" ], "type": "object" }, "SharedArtifactsV1": { "additionalProperties": false, "properties": { "function_policy_sha256": { "type": "string" }, "limits_sha256": { "type": "string" }, "model_sha256": { "type": "string" }, "output_sha256": { "type": "string" } }, "required": [ "function_policy_sha256", "limits_sha256", "model_sha256", "output_sha256" ], "type": "object" }, "SystemPromptStrategy": { "description": "How a caller-supplied system prompt combines with the built-in identity prompt.", "oneOf": [ { "description": "Caller prompt replaces the built-in prompt verbatim.", "enum": [ "override" ], "type": "string" }, { "description": "Caller prompt is appended to the built-in identity prompt.", "enum": [ "enrich" ], "type": "string" }, { "description": "No system prompt is sent to the model.", "enum": [ "disabled" ], "type": "string" } ] }, "ThinkingLevel": { "enum": [ "minimal", "low", "medium", "high", "xhigh" ], "type": "string" }, "VariantAggregateV1": { "additionalProperties": false, "properties": { "benchmarked_runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "evaluated_runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "median_cost_usd": { "format": "double", "type": [ "number", "null" ] }, "median_error_span_count": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "median_function_call_errors": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "median_function_calls": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "median_input_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "median_output_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "median_reasoning_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "median_score": { "format": "double", "type": [ "number", "null" ] }, "median_span_count": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "median_total_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "median_trace_count": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "median_trace_duration_ms": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "median_wall_time_ms": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "pass_rate": { "format": "double", "type": "number" }, "passed": { "format": "uint32", "minimum": 0, "type": "integer" }, "runs": { "format": "uint32", "minimum": 0, "type": "integer" } }, "required": [ "benchmarked_runs", "evaluated_runs", "pass_rate", "passed", "runs" ], "type": "object" }, "VariantArtifactV1": { "additionalProperties": false, "properties": { "label": { "type": [ "string", "null" ] }, "prompt_sha256": { "type": "string" }, "system_prompt_sha256": { "type": [ "string", "null" ] } }, "required": [ "prompt_sha256" ], "type": "object" }, "VariantRoleV1": { "enum": [ "control", "treatment" ], "type": "string" } }, "title": "Nullable_EvalResultResponseV1" } }, { "description": "Start a durable same-model A/B evaluation. Exactly one dimension may change: `prompt` or `system_prompt`. Returns evaluation_id immediately; use eval::status, eval::result, or the eval::completed trigger instead of polling in a tight loop.", "metadata": {}, "name": "eval::start", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "ComparisonDimensionV1": { "enum": [ "prompt", "system_prompt" ], "type": "string" }, "EvalLimitsV1": { "additionalProperties": false, "properties": { "evaluation": { "allOf": [ { "$ref": "#/definitions/EvaluationLimitsV1" } ], "default": {} }, "execution": { "allOf": [ { "$ref": "#/definitions/ExecutionLimitsV1" } ], "default": { "invocation_timeout_seconds": 120, "max_output_tokens_per_call": 8192, "max_turns": 1000, "scenario_timeout_seconds": 600 } } }, "type": "object" }, "EvalModelConfigV1": { "additionalProperties": false, "properties": { "mode": { "anyOf": [ { "$ref": "#/definitions/Mode" }, { "type": "null" } ] }, "model": { "type": "string" }, "provider": { "type": [ "string", "null" ] }, "provider_options": { "additionalProperties": true, "type": [ "object", "null" ] }, "system_prompt_strategy": { "allOf": [ { "$ref": "#/definitions/SystemPromptStrategy" } ], "default": "enrich" }, "thinking_level": { "anyOf": [ { "$ref": "#/definitions/ThinkingLevel" }, { "type": "null" } ] } }, "required": [ "model" ], "type": "object" }, "EvalVariantV1": { "additionalProperties": false, "properties": { "label": { "type": [ "string", "null" ] }, "prompt": { "type": "string" }, "system_prompt": { "description": "`null` disables the system prompt for this variant. An empty string keeps the harness behavior of resolving the provider or built-in prompt.", "type": [ "string", "null" ] } }, "required": [ "prompt" ], "type": "object" }, "EvaluationLimitsV1": { "additionalProperties": false, "properties": { "max_error_spans": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "max_function_call_errors": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] } }, "type": "object" }, "EvaluatorSpecV1": { "additionalProperties": false, "properties": { "arguments": { "default": {} }, "function_id": { "type": "string" } }, "required": [ "function_id" ], "type": "object" }, "ExecutionLimitsV1": { "additionalProperties": false, "properties": { "invocation_timeout_seconds": { "default": 120, "format": "uint64", "minimum": 0, "type": "integer" }, "max_cost_usd": { "format": "double", "type": [ "number", "null" ] }, "max_output_tokens_per_call": { "default": 8192, "format": "uint64", "minimum": 0, "type": "integer" }, "max_total_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "max_turns": { "default": 1000, "format": "uint32", "minimum": 0, "type": "integer" }, "scenario_timeout_seconds": { "default": 600, "format": "uint64", "minimum": 0, "type": "integer" } }, "type": "object" }, "ExecutionOrderV1": { "enum": [ "balanced_control_first", "balanced_treatment_first" ], "type": "string" }, "ExposeMode": { "description": "How allowed functions reach the model (harness.md § Exposure modes).", "enum": [ "agent_trigger", "native" ], "type": "string" }, "FunctionPolicy": { "description": "The fail-closed dispatch policy (harness.md § Functions). Absent on the send => every call denied (a plain chat loop).", "properties": { "allow": { "default": [], "items": { "type": "string" }, "type": "array" }, "deny": { "default": [], "items": { "type": "string" }, "type": "array" }, "expose": { "allOf": [ { "$ref": "#/definitions/ExposeMode" } ], "default": "agent_trigger" } }, "type": "object" }, "Mode": { "description": "Console / send operating mode — prepends a short paragraph before the shared identity prompt. `ask` is also structural: the turn's dispatch policy is capped at the configured default policy, never widened.", "enum": [ "ask", "agent" ], "type": "string" }, "OutputContract": { "description": "Free text by default; `json` constrains the final answer to a JSON value, validated against `schema` when supplied.", "oneOf": [ { "properties": { "type": { "enum": [ "text" ], "type": "string" } }, "required": [ "type" ], "type": "object" }, { "properties": { "schema": true, "type": { "enum": [ "json" ], "type": "string" } }, "required": [ "type" ], "type": "object" } ] }, "SystemPromptStrategy": { "description": "How a caller-supplied system prompt combines with the built-in identity prompt.", "oneOf": [ { "description": "Caller prompt replaces the built-in prompt verbatim.", "enum": [ "override" ], "type": "string" }, { "description": "Caller prompt is appended to the built-in identity prompt.", "enum": [ "enrich" ], "type": "string" }, { "description": "No system prompt is sent to the model.", "enum": [ "disabled" ], "type": "string" } ] }, "ThinkingLevel": { "enum": [ "minimal", "low", "medium", "high", "xhigh" ], "type": "string" } }, "properties": { "control": { "$ref": "#/definitions/EvalVariantV1" }, "dimension": { "$ref": "#/definitions/ComparisonDimensionV1" }, "evaluator": { "anyOf": [ { "$ref": "#/definitions/EvaluatorSpecV1" }, { "type": "null" } ] }, "execution_order": { "anyOf": [ { "$ref": "#/definitions/ExecutionOrderV1" }, { "type": "null" } ] }, "functions": { "anyOf": [ { "$ref": "#/definitions/FunctionPolicy" }, { "type": "null" } ], "description": "Shared function policy for both variants. Omitted means deny all." }, "limits": { "allOf": [ { "$ref": "#/definitions/EvalLimitsV1" } ], "default": { "evaluation": {}, "execution": { "invocation_timeout_seconds": 120, "max_output_tokens_per_call": 8192, "max_turns": 1000, "scenario_timeout_seconds": 600 } } }, "metadata": { "description": "Shared harness metadata, including an optional filesystem scope." }, "model": { "$ref": "#/definitions/EvalModelConfigV1" }, "output": { "anyOf": [ { "$ref": "#/definitions/OutputContract" }, { "type": "null" } ], "description": "Shared output contract for both variants. Omitted means text." }, "runs": { "format": "uint32", "minimum": 0, "type": [ "integer", "null" ] }, "treatment": { "$ref": "#/definitions/EvalVariantV1" } }, "required": [ "control", "dimension", "model", "treatment" ], "title": "EvalStartRequestV1", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": false, "definitions": { "EvalStatusV1": { "enum": [ "queued", "running", "completed", "failed", "cancelled" ], "type": "string" } }, "properties": { "evaluation_id": { "type": "string" }, "status": { "$ref": "#/definitions/EvalStatusV1" } }, "required": [ "evaluation_id", "status" ], "title": "EvalStartResponseV1", "type": "object" } }, { "description": "Read evaluation progress without loading the complete report.", "metadata": {}, "name": "eval::status", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "evaluation_id": { "type": "string" } }, "required": [ "evaluation_id" ], "title": "EvaluationIdRequestV1", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "anyOf": [ { "$ref": "#/definitions/EvalStatusResponseV1" }, { "type": "null" } ], "definitions": { "ActiveRunV1": { "additionalProperties": false, "properties": { "iteration": { "format": "uint32", "minimum": 0, "type": "integer" }, "role": { "$ref": "#/definitions/VariantRoleV1" }, "run_id": { "type": "string" }, "session_id": { "type": "string" }, "started_at": { "format": "int64", "type": "integer" }, "turn_id": { "type": [ "string", "null" ] } }, "required": [ "iteration", "role", "run_id", "session_id", "started_at" ], "type": "object" }, "EvalStatusResponseV1": { "additionalProperties": false, "properties": { "active": { "anyOf": [ { "$ref": "#/definitions/ActiveRunV1" }, { "type": "null" } ] }, "completed_at": { "format": "int64", "type": [ "integer", "null" ] }, "created_at": { "format": "int64", "type": "integer" }, "error": { "type": [ "string", "null" ] }, "evaluation_id": { "type": "string" }, "failed_runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "passed_runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "status": { "$ref": "#/definitions/EvalStatusV1" }, "terminal_runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "total_runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "updated_at": { "format": "int64", "type": "integer" } }, "required": [ "created_at", "evaluation_id", "failed_runs", "passed_runs", "status", "terminal_runs", "total_runs", "updated_at" ], "type": "object" }, "EvalStatusV1": { "enum": [ "queued", "running", "completed", "failed", "cancelled" ], "type": "string" }, "VariantRoleV1": { "enum": [ "control", "treatment" ], "type": "string" } }, "title": "Nullable_EvalStatusResponseV1" } }, { "description": "Internal durable evaluation step.", "metadata": { "internal": true, "trace_hidden": true }, "name": "eval::step", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "evaluation_id": { "type": "string" }, "step": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "evaluation_id", "step" ], "title": "StepRequestV1", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": false, "definitions": { "EvalStatusV1": { "enum": [ "queued", "running", "completed", "failed", "cancelled" ], "type": "string" } }, "properties": { "skipped": { "type": "boolean" }, "status": { "$ref": "#/definitions/EvalStatusV1" } }, "required": [ "skipped", "status" ], "title": "StepResponseV1", "type": "object" } }, { "description": "Internal recovery and timeout sweep.", "metadata": { "internal": true, "trace_hidden": true }, "name": "eval::sweep", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "scheduled_at": { "default": null, "format": "int64", "type": [ "integer", "null" ] }, "scheduled_time": { "default": null, "type": [ "string", "null" ] } }, "title": "SweepEventV1", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": false, "properties": { "swept": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "swept" ], "title": "SweepResponseV1", "type": "object" } }, { "description": "Serve the eval worker's injected console UI assets (content function for its console:script / console:style triggers).", "metadata": { "internal": true }, "name": "eval::ui-content", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "description": "Input of the content function: the console asks for one asset by path.", "properties": { "path": { "description": "The asset path from the trigger config (e.g. `state/page.js`).", "type": "string" } }, "required": [ "path" ], "title": "UiContentInput", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "description": "Output of the content function.", "properties": { "content": { "description": "The asset source, verbatim.", "type": "string" }, "content_type": { "description": "MIME type the console should serve the asset with.", "type": "string" } }, "required": [ "content", "content_type" ], "title": "UiContentResult", "type": "object" } } ], "triggers": [ { "description": "An evaluation reached a terminal status.", "invocation_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": false, "properties": { "evaluation_id": { "type": [ "string", "null" ] } }, "title": "CompletedBindingConfigV1", "type": "object" }, "metadata": {}, "name": "eval::completed", "return_schema": {} } ] } ```