$worker
harness-e2e
v0.5.2-experimentalMeasure Harness capability, retain reproducible evidence, and expose the E2E dashboard.
iiiverifiedinstall
$iii trigger compose::add worker=harness-e2e@0.5.2-experimental
- macOS: arm64 · x64
- Linux: arm64 · armv7 · x64
- Windows: arm64 · x64 · x86
exact versions are immutable; binary and bundle artifacts are digest-pinned.
agent-ready brief for v0.5.2-experimental
install + config + dependencies + readme + api reference, all in one place. fetch as agent-context.md for an llm to consume.
full markdown
the same content rendered as discrete blocks below is exposed as a single markdown document at
/workers/harness-e2e.md?version=0.5.2-experimental. paste it into an llm prompt or pipe it through curl from a worker.install
install
$iii trigger compose::add worker=harness-e2e@0.5.2-experimental
configuration
iii-config.yaml
- data_dir: ~/.iii/data/harness-e2edependencies
readme
README.md
Harness E2E
harness-e2e measures which complexity levels a Harness stack can execute
with correct deliverables, structural integrity, bounded work, and repeatable
outcomes.
The repository is intentionally independent from the workers source tree.
Runtime discovery, execution, observation, state access, and cleanup all happen
through functions registered in iii. The only product input is an immutable
subject artifact or an already-running iii stack.
Binaries
harness-e2eis started by Compose and registers the asynchronouse2e::*control plane plus the injectable Console dashboard. Explicit subcommands keep direct scenario execution, report inspection, and the standalone dashboard available from the same binary.
Build and validate the repository:
pnpm --dir dashboard install --frozen-lockfile
pnpm --dir dashboard typecheck
pnpm --dir dashboard lint
pnpm --dir dashboard test
pnpm --dir dashboard build
cargo test --locked --all-targets
cargo clippy --locked --all-targets -- -D warnings
node --test tests/dashboard/*.test.cjs
python3 -m unittest discover -s tests/python -p 'test_*.py'List the materialized scenarios and their scenario versions:
cargo run --locked --bin harness-e2e -- list
cargo run --locked --bin harness-e2e -- catalog
cargo run --locked --bin harness-e2e -- validate-scenariosNew declarative scenarios are authored only as scenarios/*.md. The compiler
embeds the exact source, validates the canonical English section structure,
and exposes the resulting file-stem id through the CLI, worker catalog,
campaign runner, dashboard, and canonical result artifacts. See
docs/markdown-scenarios.md.
Replay an archived input only through its immutable plan (the runner rejects any scenario, model, policy, budget, stack, runner, run-count, or retry drift):
cargo run --locked -- replay-materialized \
target/e2e/evidence/<run-id>/<attempt-id>/materialized-plan.jsonRun against an existing stack:
cargo run --locked --bin harness-e2e -- run \
--url ws://127.0.0.1:49134 \
--model codex/gpt-5.6-luna \
--provider openai-codex \
--scenario todo_worker_simpleValidate one of the checked-in canonical campaign assets:
python3 scripts/run_e2e_campaign.py config/campaigns/post-deploy.json --validate-only
python3 scripts/run_e2e_campaign.py config/campaigns/daily.json --dry-runOperational campaign execution is dispatched only by Release Control through
.github/workflows/exact-stack-e2e.yml. The repository no longer
publishes independent daily, weekly, post-deploy, or fault-stress dispatch
workflows.
Adaptive L5 classification, trusted planning boundaries, resume semantics, and
the canonical incident/release/cross-repository cases are documented in
docs/l5-adaptive-scenarios.md.
Campaign manifests never select or rotate seeds. They separate replay-safe
turns from scripted dialogue and composite flows, persist a summary for every
group, and are advisory by default while their longitudinal history is being
calibrated. Release Control owns scheduling and dispatch; the executor keeps
the result advisory and archives each materialized group through the
environment-owned durable archiver.
The code-focused campaigns use protected disposable checkouts of
iii-hq/e2e-fixture. The engineering handoff uses its dedicated pinned
revision, while shell_coder_sandbox, chess_engine_build, and trend_blog
share a second pinned revision through HARNESS_E2E_FIXTURE_PATH. The protected
launcher and cleanup boundary are described in
docs/engineering-ticket-git-handoff.md.
The team-facing composition and didactic description of every daily, weekly,
and post-deploy scenario is in docs/e2e-test-plans.md.
Release Control dispatches .github/workflows/exact-stack-e2e.yml
directly in this repository. The workflow validates the single strict campaign contract,
executes every common group in an isolated ephemeral stack, routes fault groups
to the protected runner, and produces one root bundle without rebuilding the
native Harness artifacts. workers supplies versioned components of the stack
under test; it does not orchestrate campaigns.
Dashboard
Build and start the dashboard from the repository root:
cargo build --locked --bin harness-e2e
target/debug/harness-e2e dashboardThe Rust build follows the same embedded-SPA contract as workers/console: it
builds the React bundle with pnpm when dashboard/dist/ is missing or stale,
then embeds the Vite output in the binary. Node and pnpm must be available on
PATH. For frontend development with HMR, use pnpm --dir dashboard dev; the
Vite server proxies runtime data, the scoped iii WebSocket, and local-run APIs
to the Rust dashboard on port 4173.
Rust-defined composite scenarios, including the multi-test security_review
example, use the shared result schema v2 and read-only execution projection.
The running Harness must publish request and response schemas compatible with the current typed surface. Missing or incompatible fields fail preflight; no payload-version compatibility mode is available.
The server listens on 0.0.0.0:4173 by default. Open
http://localhost:4173/#/overview on the same machine, or replace localhost
with the machine's address when accessing it remotely. Use --listen 0.0.0.0:PORT to select another port, III_URL to select the running Harness
stack, and --runs-dir to select another local history directory.
Local mode loads data incrementally through iii: 25 compact summaries on the
first overview page, one complete report when an execution is opened, only the
selected pair for comparison, and the model/scenario catalog when the run dialog
opens. Server-side filtering and cursor pagination keep history growth out of
the initial payload. Static published and --view-only presentations preserve
the generated-file fallback.
Local mode exposes controls that can start and cancel E2E runs, so expose the
port only on a trusted network. Use --listen 127.0.0.1:4173 when access should
remain local. See dashboard/README.md for view-only mode
and the complete dashboard behavior.
Compose lifecycle
Release Control materializes the immutable dependency graph and this repository
renders it as worker-compose.yaml. Every execution starts an empty Engine and
a dedicated Compose daemon, then runs compose::validate, compose::up,
compose::status, and compose::down. The project and daemon namespaces are
unique per execution.
Compose supplies III_URL, III_NAMESPACE, III_WORKER_NAME, and III_CONFIG
to the harness-e2e process. All four values are mandatory. The referenced
configuration file contains the execution-specific data_dir; there is no
local fallback, command-line override, or runtime self-registration.
Publication validates the locally built binary through a path:// Compose
container before the package is uploaded. Published campaigns use only exact
Registry package versions. Provider secrets are written to temporary
permission-restricted env_file files and are never included in contract,
Compose, evidence, or archive artifacts.
The worker exposes e2e::run, e2e::status, e2e::cancel,
e2e::results-get, e2e::results-list, e2e::compare,
e2e::scenarios-list, e2e::scenarios-create,
e2e::scenarios-authoring-guide, e2e::archive, e2e::archive-head,
e2e::archive-restore,
e2e::history-list, and e2e::retention-sweep.
Fault supervisors use e2e::fault-plan and e2e::fault-evaluate so plan
materialization and recovery classification stay on the same iii control plane.
Subject policies deny e2e::*.
Durable artifacts are chunked through storage::*, while longitudinal series
are ingested through database::*. The runner has no S3, GCS, R2, SQL-driver,
or Harness dependency.
Weekly Stress materializes deterministic fault plans and evaluates journals from a
protected supervisor. See docs/fault-injection.md.
Lane promotion is governed by
config/policies/cutover.json.
Repository boundaries
src/owns the runner, local wire adapters, scenarios, evaluation, longitudinal comparison, and the E2E control worker.config/owns reviewed comparison and cutover policies, fault profiles, and standalone stack configuration.tests/owns test-only fixtures, golden wire schemas, and the Node/Python validation suites.schemas/contains the public contracts for generated E2E artifacts.dashboard/contains the React, TypeScript, Vite, and Tailwind dashboard embedded in the Rust binary.- generated reports, transcripts, logs, and deliverables stay outside Git.
The crate may depend on the iii SDK and generic libraries. It must not declare
a path or Git dependency on workers, Harness, or another product crate.
Contract compatibility is established at runtime from
engine::functions::list and engine::functions::info; the checked-in schemas
are parity fixtures, not a linked product API.
The assessment and on-demand analysis boundary has one current payload shape,
written only to results.json; scenario contracts are the only versioned
domain.
Deterministic, pre-cleanup asset capture applies explicit safety limits and
writes an unversioned sidecar containing the canonical deterministic validation
portion, which is aggregated into results.json.
Observation
The runner waits for a session tree to finish by binding
harness::turn-completed to an internal sink (e2e::on-turn-completed) before
harness::send. That sink is not a control-plane verb: it is not registered
with e2e::run / e2e::status / e2e::cancel, and it does not appear in
e2e::scenarios-list. Subject policies already deny e2e::*.
A 15s watchdog samples harness::metrics and one root harness::status for
stuck detection, heartbeat logs, and e2e::cancel. If the trigger type is
missing from engine::triggers::list, the run is unsupported infrastructure —
there is no silent fallback to polling harness::status or harness::metrics.
After the tree completes, the runner still collects terminal status, metrics,
transcripts, and deliverables.
Subject artifacts
Cross-repository executions accept a subject manifest matching
schemas/subject-artifact.json. The archive and every declared file are
verified before use. Mutable URLs, shortened Git revisions, unexpected archive
paths, and digest mismatches are rejected.
Untrusted subject artifacts are never given provider, storage, or GitHub credentials in their environment. Provider workers and the trusted E2E worker are started separately. PR execution remains non-blocking shadow evidence until the source repository, revision, E2E ref, and credential boundary are approved.
Comparison
Every completed execution records the subject and E2E revisions, observed wire
contracts, scenario version, materialized inputs, seed, policies, artifacts,
and raw structural evidence. e2e::compare accepts two distinct completed
execution ids (from_execution_id and to_execution_id) and writes a unique
comparisons/ plus e2e-summary.md. Numeric
deltas remain disabled when the case set or canonical contract differs.
Deliverable, structural, technical, cost, latency, turns, retries, and work amplification deltas remain independent. Cost and wall-time are reported as observed metrics and compared only within a compatible baseline/candidate cohort. amplification deltas remain independent. A tier is repeatable after five local runs satisfy the deliverable, structural, and technical thresholds. Cost and wall-time are reported as observed metrics and compared only within a compatible baseline/candidate cohort.
Runtime-only package boundary
This repository executes exact-stack Test Plans and never publishes itself as
a Registry worker. Release Control supplies a digest-locked stack and an
immutable executor SHA to exact-stack-e2e.yml; every Registry version in the
lock is exact, including historical candidates.
The root iii.worker.yaml remains the public manifest for local iii worker
development and package compatibility. The root worker-compose.yaml remains
a normal public Compose document. Release Control and post-prepare workflow
phases deliberately read neither source contract.
api reference (json)
agent-api-reference.json
{
"functions": [
{
"description": "Persist a completed execution through private iii storage and idempotent history.",
"metadata": {
"contract": {
"capabilities": [
"archive"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::archive",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"RetentionClass": {
"enum": [
"temporary",
"pull_request",
"longitudinal",
"canonical"
],
"type": "string"
}
},
"properties": {
"execution_id": {
"type": "string"
},
"retention_class": {
"$ref": "#/definitions/RetentionClass"
}
},
"required": [
"execution_id",
"retention_class"
],
"title": "ArchiveExecutionRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"DurableArchiveReference": {
"additionalProperties": false,
"properties": {
"archive_id": {
"type": "string"
},
"created_at": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"expires_at": {
"type": [
"string",
"null"
]
},
"identity_sha256": {
"type": "string"
},
"manifest": {
"$ref": "#/definitions/StorageObjectReference"
},
"manifest_backup": {
"$ref": "#/definitions/StorageObjectReference"
},
"retention_class": {
"$ref": "#/definitions/RetentionClass"
}
},
"required": [
"archive_id",
"created_at",
"execution_id",
"identity_sha256",
"manifest",
"manifest_backup",
"retention_class"
],
"type": "object"
},
"HistoryRecord": {
"additionalProperties": false,
"properties": {
"archive": {
"$ref": "#/definitions/DurableArchiveReference"
},
"case_count": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"e2e_repository": {
"type": "string"
},
"e2e_revision": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"identity_sha256": {
"type": "string"
},
"ingestion_id": {
"type": "string"
},
"lane": {
"type": "string"
},
"occurred_at": {
"type": "string"
},
"passed": {
"type": "boolean"
},
"stack_mode": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"subject_revision": {
"type": "string"
}
},
"required": [
"archive",
"case_count",
"e2e_repository",
"e2e_revision",
"execution_id",
"identity_sha256",
"ingestion_id",
"lane",
"occurred_at",
"passed",
"stack_mode",
"subject_model",
"subject_provider",
"subject_revision"
],
"type": "object"
},
"RetentionClass": {
"enum": [
"temporary",
"pull_request",
"longitudinal",
"canonical"
],
"type": "string"
},
"StorageObjectReference": {
"properties": {
"bucket": {
"type": "string"
},
"key": {
"type": "string"
},
"media_type": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"uri": {
"type": "string"
}
},
"required": [
"bucket",
"key",
"media_type",
"sha256",
"size_bytes",
"uri"
],
"type": "object"
}
},
"properties": {
"archive": {
"$ref": "#/definitions/DurableArchiveReference"
},
"duplicate_ingestion": {
"type": "boolean"
},
"history": {
"$ref": "#/definitions/HistoryRecord"
}
},
"required": [
"archive",
"duplicate_ingestion",
"history"
],
"title": "ArchiveResponse",
"type": "object"
}
},
{
"description": "Verify availability and immutable metadata for an archived execution.",
"metadata": {
"contract": {
"capabilities": [
"archive-head"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::archive-head",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
}
},
"required": [
"execution_id"
],
"title": "ExecutionRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ArchiveAvailability": {
"enum": [
"available",
"expired"
],
"type": "string"
},
"DurableArchiveReference": {
"additionalProperties": false,
"properties": {
"archive_id": {
"type": "string"
},
"created_at": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"expires_at": {
"type": [
"string",
"null"
]
},
"identity_sha256": {
"type": "string"
},
"manifest": {
"$ref": "#/definitions/StorageObjectReference"
},
"manifest_backup": {
"$ref": "#/definitions/StorageObjectReference"
},
"retention_class": {
"$ref": "#/definitions/RetentionClass"
}
},
"required": [
"archive_id",
"created_at",
"execution_id",
"identity_sha256",
"manifest",
"manifest_backup",
"retention_class"
],
"type": "object"
},
"RetentionClass": {
"enum": [
"temporary",
"pull_request",
"longitudinal",
"canonical"
],
"type": "string"
},
"StorageObjectReference": {
"properties": {
"bucket": {
"type": "string"
},
"key": {
"type": "string"
},
"media_type": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"uri": {
"type": "string"
}
},
"required": [
"bucket",
"key",
"media_type",
"sha256",
"size_bytes",
"uri"
],
"type": "object"
}
},
"properties": {
"archive": {
"$ref": "#/definitions/DurableArchiveReference"
},
"availability": {
"$ref": "#/definitions/ArchiveAvailability"
},
"verified_at": {
"type": "string"
}
},
"required": [
"archive",
"availability",
"verified_at"
],
"title": "ArchiveHeadResponse",
"type": "object"
}
},
{
"description": "Restore and verify an archived execution from its immutable manifest.",
"metadata": {
"contract": {
"capabilities": [
"archive-restore"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::archive-restore",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
}
},
"required": [
"execution_id"
],
"title": "ExecutionRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ArchiveAvailability": {
"enum": [
"available",
"expired"
],
"type": "string"
},
"DurableArchiveReference": {
"additionalProperties": false,
"properties": {
"archive_id": {
"type": "string"
},
"created_at": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"expires_at": {
"type": [
"string",
"null"
]
},
"identity_sha256": {
"type": "string"
},
"manifest": {
"$ref": "#/definitions/StorageObjectReference"
},
"manifest_backup": {
"$ref": "#/definitions/StorageObjectReference"
},
"retention_class": {
"$ref": "#/definitions/RetentionClass"
}
},
"required": [
"archive_id",
"created_at",
"execution_id",
"identity_sha256",
"manifest",
"manifest_backup",
"retention_class"
],
"type": "object"
},
"RetentionClass": {
"enum": [
"temporary",
"pull_request",
"longitudinal",
"canonical"
],
"type": "string"
},
"StorageObjectReference": {
"properties": {
"bucket": {
"type": "string"
},
"key": {
"type": "string"
},
"media_type": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"uri": {
"type": "string"
}
},
"required": [
"bucket",
"key",
"media_type",
"sha256",
"size_bytes",
"uri"
],
"type": "object"
}
},
"properties": {
"archive": {
"$ref": "#/definitions/DurableArchiveReference"
},
"availability": {
"$ref": "#/definitions/ArchiveAvailability"
},
"restored_files": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"restored_from_backup": {
"type": "boolean"
},
"restored_root": {
"type": [
"string",
"null"
]
}
},
"required": [
"archive",
"availability",
"restored_files",
"restored_from_backup"
],
"title": "ArchiveRestoreResponse",
"type": "object"
}
},
{
"description": "Request idempotent cancellation and cleanup of an E2E execution.",
"metadata": {
"contract": {
"capabilities": [
"cancel"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::cancel",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
}
},
"required": [
"execution_id"
],
"title": "ExecutionRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ExecutionPhase": {
"enum": [
"requested",
"admitted",
"preflighting",
"materializing",
"setting_up",
"executing",
"collecting",
"evaluating",
"persisting",
"cleaning_up",
"finalizing",
"completed",
"failed",
"cancelled",
"needs_reconciliation",
"unsupported"
],
"type": "string"
}
},
"properties": {
"accepted": {
"type": "boolean"
},
"execution_id": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
}
},
"required": [
"accepted",
"execution_id",
"phase"
],
"title": "CancelResponse",
"type": "object"
}
},
{
"description": "Compare two completed executions when their identities are eligible.",
"metadata": {
"contract": {
"capabilities": [
"compare"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::compare",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ComparisonPolicy": {
"properties": {
"baseline_id": {
"description": "Identifier of the reviewed baseline these thresholds came from, when one was loaded. Absent when the code-default thresholds are in effect.",
"type": [
"string",
"null"
]
},
"regression": {
"allOf": [
{
"$ref": "#/definitions/RegressionThresholds"
}
],
"default": {
"cost_increase_ratio": 0.2,
"deliverable_success_drop": 0.03,
"p95_turns_increase_ratio": 0.25,
"structural_integrity_drop": 0.02,
"technical_failure_increase": 0.02,
"wall_time_increase_ratio": 0.2
}
},
"scenario_overrides": {
"additionalProperties": {
"$ref": "#/definitions/ThresholdOverrides"
},
"description": "Per-scenario partial threshold overrides carried from the reviewed baseline.",
"type": "object"
}
},
"type": "object"
},
"RegressionThresholds": {
"properties": {
"cost_increase_ratio": {
"format": "double",
"type": "number"
},
"deliverable_success_drop": {
"format": "double",
"type": "number"
},
"p95_turns_increase_ratio": {
"format": "double",
"type": "number"
},
"structural_integrity_drop": {
"format": "double",
"type": "number"
},
"technical_failure_increase": {
"format": "double",
"type": "number"
},
"wall_time_increase_ratio": {
"format": "double",
"type": "number"
}
},
"required": [
"cost_increase_ratio",
"deliverable_success_drop",
"p95_turns_increase_ratio",
"structural_integrity_drop",
"technical_failure_increase",
"wall_time_increase_ratio"
],
"type": "object"
},
"ThresholdOverrides": {
"additionalProperties": false,
"description": "Partial per-scenario replacement for [`RegressionThresholds`]: a named field replaces the baseline value, an omitted field inherits it.",
"properties": {
"cost_increase_ratio": {
"format": "double",
"type": [
"number",
"null"
]
},
"deliverable_success_drop": {
"format": "double",
"type": [
"number",
"null"
]
},
"p95_turns_increase_ratio": {
"format": "double",
"type": [
"number",
"null"
]
},
"structural_integrity_drop": {
"format": "double",
"type": [
"number",
"null"
]
},
"technical_failure_increase": {
"format": "double",
"type": [
"number",
"null"
]
},
"wall_time_increase_ratio": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"type": "object"
}
},
"properties": {
"baseline": {
"default": null,
"description": "Reviewed baseline file to load thresholds from. Omit to use the checked-in `config/baselines/default.json` when it exists, falling back to the code-default thresholds. Ignored when `policy` is set.",
"type": [
"string",
"null"
]
},
"from_execution_id": {
"type": "string"
},
"policy": {
"anyOf": [
{
"$ref": "#/definitions/ComparisonPolicy"
},
{
"type": "null"
}
],
"default": null,
"description": "Explicit policy override. Omit to gate on the reviewed baseline."
},
"to_execution_id": {
"type": "string"
}
},
"required": [
"from_execution_id",
"to_execution_id"
],
"title": "CompareRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ArtifactReference": {
"properties": {
"id": {
"type": "string"
},
"kind": {
"type": "string"
},
"media_type": {
"type": "string"
},
"path": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"id",
"kind",
"media_type",
"path",
"sha256",
"size_bytes"
],
"type": "object"
},
"BenchmarkDelta": {
"properties": {
"deliverable_success_rate": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"flaky_rate": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"median_score": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p50_cost_usd": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p50_turns": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p50_wall_time_ms": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p50_work_amplification": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p95_cost_usd": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p95_turns": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p95_wall_time_ms": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p95_work_amplification": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"retry_rate": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"structural_integrity_rate": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"technical_failure_rate": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"unavailable": {
"additionalProperties": {
"type": "string"
},
"type": "object"
}
},
"type": "object"
},
"CaseCohortKey": {
"properties": {
"case_id": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"execution": {
"$ref": "#/definitions/ExecutionCohortIdentity"
},
"inputs_sha256": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"case_id",
"contract_sha256",
"execution",
"inputs_sha256",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"CaseComparison": {
"properties": {
"complexity_tier": {
"$ref": "#/definitions/ComplexityTier"
},
"delta": {
"$ref": "#/definitions/BenchmarkDelta"
},
"from": {
"$ref": "#/definitions/CaseMetrics"
},
"from_run_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"key": {
"$ref": "#/definitions/CaseCohortKey"
},
"regressions": {
"items": {
"$ref": "#/definitions/RegressionSignal"
},
"type": "array"
},
"to": {
"$ref": "#/definitions/CaseMetrics"
},
"to_run_ids": {
"items": {
"type": "string"
},
"type": "array"
}
},
"required": [
"complexity_tier",
"delta",
"from",
"from_run_ids",
"key",
"regressions",
"to",
"to_run_ids"
],
"type": "object"
},
"CaseMetrics": {
"properties": {
"deliverable_success": {
"anyOf": [
{
"$ref": "#/definitions/RateEstimate"
},
{
"type": "null"
}
]
},
"excluded_infrastructure_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"flaky_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"included_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"median_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"p50_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"p50_turns": {
"format": "double",
"type": [
"number",
"null"
]
},
"p50_wall_time_ms": {
"format": "double",
"type": [
"number",
"null"
]
},
"p50_work_amplification": {
"format": "double",
"type": [
"number",
"null"
]
},
"p95_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"p95_turns": {
"format": "double",
"type": [
"number",
"null"
]
},
"p95_wall_time_ms": {
"format": "double",
"type": [
"number",
"null"
]
},
"p95_work_amplification": {
"format": "double",
"type": [
"number",
"null"
]
},
"retry_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"structural_integrity": {
"anyOf": [
{
"$ref": "#/definitions/RateEstimate"
},
{
"type": "null"
}
]
},
"technical_failure": {
"anyOf": [
{
"$ref": "#/definitions/RateEstimate"
},
{
"type": "null"
}
]
},
"unavailable": {
"additionalProperties": {
"type": "string"
},
"type": "object"
}
},
"required": [
"excluded_infrastructure_runs",
"included_runs"
],
"type": "object"
},
"CohortAudit": {
"properties": {
"excluded_cases": {
"items": {
"$ref": "#/definitions/ExcludedCase"
},
"type": "array"
},
"excluded_runs": {
"items": {
"$ref": "#/definitions/ExcludedRun"
},
"type": "array"
},
"included_case_ids": {
"items": {
"type": "string"
},
"type": "array"
}
},
"required": [
"excluded_cases",
"excluded_runs",
"included_case_ids"
],
"type": "object"
},
"ComparisonSide": {
"enum": [
"from",
"to"
],
"type": "string"
},
"ComparisonSummary": {
"properties": {
"baseline_id": {
"description": "Reviewed baseline the regression thresholds came from; absent when the code-default thresholds gated this comparison.",
"type": [
"string",
"null"
]
},
"cases": {
"items": {
"$ref": "#/definitions/CaseComparison"
},
"type": "array"
},
"cohort": {
"$ref": "#/definitions/CohortAudit"
},
"comparable": {
"type": "boolean"
},
"comparison_id": {
"type": "string"
},
"from_execution_id": {
"type": "string"
},
"from_revision": {
"type": [
"string",
"null"
]
},
"gate_passed": {
"type": "boolean"
},
"reasons": {
"items": {
"type": "string"
},
"type": "array"
},
"regressions": {
"items": {
"$ref": "#/definitions/RegressionSignal"
},
"type": "array"
},
"to_execution_id": {
"type": "string"
},
"to_revision": {
"type": [
"string",
"null"
]
}
},
"required": [
"cases",
"cohort",
"comparable",
"comparison_id",
"from_execution_id",
"gate_passed",
"reasons",
"regressions",
"to_execution_id"
],
"type": "object"
},
"ComplexityTier": {
"enum": [
"l0_atomic",
"l1_sequential",
"l2_stateful",
"l3_concurrent",
"l4_coordinated",
"l5_adaptive"
],
"type": "string"
},
"DeltaValue": {
"properties": {
"absolute": {
"format": "double",
"type": "number"
},
"relative_ratio": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"absolute"
],
"type": "object"
},
"ExcludedCase": {
"properties": {
"case_id": {
"type": "string"
},
"reasons": {
"items": {
"type": "string"
},
"type": "array"
},
"scenario_id": {
"type": "string"
},
"side": {
"$ref": "#/definitions/ComparisonSide"
}
},
"required": [
"case_id",
"reasons",
"scenario_id",
"side"
],
"type": "object"
},
"ExcludedRun": {
"properties": {
"attempt_id": {
"type": "string"
},
"case_id": {
"type": "string"
},
"reason": {
"type": "string"
},
"run_id": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"side": {
"$ref": "#/definitions/ComparisonSide"
}
},
"required": [
"attempt_id",
"case_id",
"reason",
"run_id",
"scenario_id",
"side"
],
"type": "object"
},
"ExecutionCohortIdentity": {
"properties": {
"e2e_repository": {
"type": [
"string",
"null"
]
},
"judge_model": {
"type": [
"string",
"null"
]
},
"judge_protocol": {
"type": [
"string",
"null"
]
},
"judge_provider": {
"type": [
"string",
"null"
]
},
"lane": {
"type": "string"
},
"stack_mode": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
}
},
"required": [
"lane",
"stack_mode",
"subject_model",
"subject_provider"
],
"type": "object"
},
"RateEstimate": {
"properties": {
"ci95_lower": {
"format": "double",
"type": "number"
},
"ci95_upper": {
"format": "double",
"type": "number"
},
"rate": {
"format": "double",
"type": "number"
},
"sample_size": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"successes": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"ci95_lower",
"ci95_upper",
"rate",
"sample_size",
"successes"
],
"type": "object"
},
"RegressionDimension": {
"enum": [
"deliverable",
"structural_integrity",
"technical_reliability",
"cost",
"wall_time",
"turns"
],
"type": "string"
},
"RegressionSignal": {
"properties": {
"blocking": {
"type": "boolean"
},
"case_id": {
"type": "string"
},
"dimension": {
"$ref": "#/definitions/RegressionDimension"
},
"maturity": {
"default": "",
"type": "string"
},
"n_from": {
"default": 0,
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"n_to": {
"default": 0,
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"observed_delta": {
"format": "double",
"type": "number"
},
"reason": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"statistic": {
"default": "",
"type": "string"
},
"threshold": {
"format": "double",
"type": "number"
}
},
"required": [
"blocking",
"case_id",
"dimension",
"observed_delta",
"reason",
"scenario_id",
"threshold"
],
"type": "object"
}
},
"properties": {
"artifacts": {
"items": {
"$ref": "#/definitions/ArtifactReference"
},
"type": "array"
},
"comparison": {
"$ref": "#/definitions/ComparisonSummary"
}
},
"required": [
"artifacts",
"comparison"
],
"title": "ComparisonResponse",
"type": "object"
}
},
{
"description": "Read models and scenarios when the execution dialog opens.",
"metadata": {
"contract": {
"capability": "catalog-get",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::catalog-get",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"url": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "CatalogRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"CatalogModel": {
"properties": {
"id": {
"type": "string"
},
"provider": {
"type": "string"
}
},
"required": [
"id",
"provider"
],
"type": "object"
},
"LocalScenarioSummary": {
"properties": {
"id": {
"type": "string"
},
"source_path": {
"type": "string"
},
"source_sha256": {
"type": "string"
},
"title": {
"type": "string"
},
"version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"id",
"source_path",
"source_sha256",
"title",
"version"
],
"type": "object"
}
},
"properties": {
"local_scenarios": {
"items": {
"$ref": "#/definitions/LocalScenarioSummary"
},
"type": "array"
},
"models": {
"items": {
"$ref": "#/definitions/CatalogModel"
},
"type": "array"
},
"scenarios": {
"items": {
"type": "string"
},
"type": "array"
},
"url": {
"type": "string"
}
},
"required": [
"local_scenarios",
"models",
"scenarios",
"url"
],
"title": "CatalogResponse",
"type": "object"
}
},
{
"description": "List immutable evaluated-system versions and their exact evaluation cohorts.",
"metadata": {
"contract": {
"capability": "evaluated-versions-list",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::evaluated-versions-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"cohort_id": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "EvaluatedVersionsRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"CohortDescriptor": {
"properties": {
"id": {
"type": "string"
},
"judge_model": {
"type": [
"string",
"null"
]
},
"judge_protocol": {
"type": [
"string",
"null"
]
},
"judge_provider": {
"type": [
"string",
"null"
]
},
"lane": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
}
},
"required": [
"id",
"lane",
"subject_model",
"subject_provider"
],
"type": "object"
},
"EvaluatedVersionDescriptor": {
"properties": {
"cohort_id": {
"type": "string"
},
"completed_at": {
"type": "string"
},
"execution_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"id": {
"type": "string"
},
"label": {
"type": "string"
},
"stack_mode": {
"type": "string"
}
},
"required": [
"cohort_id",
"completed_at",
"execution_count",
"id",
"label",
"stack_mode"
],
"type": "object"
}
},
"properties": {
"cohorts": {
"items": {
"$ref": "#/definitions/CohortDescriptor"
},
"type": "array"
},
"revision": {
"type": "string"
},
"versions": {
"items": {
"$ref": "#/definitions/EvaluatedVersionDescriptor"
},
"type": "array"
}
},
"required": [
"cohorts",
"revision",
"versions"
],
"title": "EvaluatedVersionsResponse",
"type": "object"
}
},
{
"description": "Read one retained execution summary and its diagnostic report.",
"metadata": {
"contract": {
"capability": "execution-get",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::execution-get",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
}
},
"required": [
"execution_id"
],
"title": "ExecutionGetRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ExecutionListResponse": {
"properties": {
"executions": {
"items": true,
"type": "array"
},
"last_update": {
"type": "string"
},
"mode": {
"type": "string"
},
"next_cursor": {
"type": [
"string",
"null"
]
},
"repo_url": {
"type": "string"
},
"retention": {
"$ref": "#/definitions/Retention"
},
"total": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"executions",
"last_update",
"mode",
"repo_url",
"retention",
"total"
],
"type": "object"
},
"Retention": {
"properties": {
"details": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"summaries": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"details",
"summaries"
],
"type": "object"
}
},
"properties": {
"detail": true,
"manifest": {
"$ref": "#/definitions/ExecutionListResponse"
}
},
"required": [
"detail",
"manifest"
],
"title": "ExecutionBundle",
"type": "object"
}
},
{
"description": "List compact retained execution summaries with server-side filters and cursor pagination.",
"metadata": {
"contract": {
"capability": "executions-list",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::executions-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"cursor": {
"default": null,
"type": [
"string",
"null"
]
},
"event": {
"default": null,
"type": [
"string",
"null"
]
},
"ids": {
"default": [],
"items": {
"type": "string"
},
"type": "array"
},
"ids_csv": {
"default": null,
"type": [
"string",
"null"
]
},
"limit": {
"default": null,
"format": "uint16",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"query": {
"default": null,
"type": [
"string",
"null"
]
},
"status": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "ExecutionListRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"Retention": {
"properties": {
"details": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"summaries": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"details",
"summaries"
],
"type": "object"
}
},
"properties": {
"executions": {
"items": true,
"type": "array"
},
"last_update": {
"type": "string"
},
"mode": {
"type": "string"
},
"next_cursor": {
"type": [
"string",
"null"
]
},
"repo_url": {
"type": "string"
},
"retention": {
"$ref": "#/definitions/Retention"
},
"total": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"executions",
"last_update",
"mode",
"repo_url",
"retention",
"total"
],
"title": "ExecutionListResponse",
"type": "object"
}
},
{
"description": "Validate and save one local-only Markdown scenario, then return its compiled identity.",
"metadata": {
"contract": {
"capability": "local-scenario-create",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::local-scenario-create",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"file_name": {
"description": "One UTF-8 `.md` file name, without a directory. Use letters, numbers, spaces, hyphens, or underscores; its stem becomes `local_<safe_id>`. Existing files are rejected instead of overwritten.",
"type": "string"
},
"source": {
"description": "Complete Markdown test definition. It must contain one H1 followed by the required H2 sections in order, use only `- local` under Plans, and contain positive `### Name (N%)` validations totaling exactly 100%.",
"type": "string"
}
},
"required": [
"file_name",
"source"
],
"title": "LocalScenarioCreateRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ArtifactExpectation": {
"properties": {
"id": {
"type": "string"
},
"kind": {
"type": "string"
},
"max_size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"media_type": {
"type": "string"
},
"schema": true
},
"required": [
"id",
"kind",
"max_size_bytes",
"media_type",
"schema"
],
"type": "object"
},
"ComplexityClassification": {
"properties": {
"method": {
"allOf": [
{
"$ref": "#/definitions/ComplexityMethod"
}
],
"default": "legacy_v1"
},
"profile": {
"$ref": "#/definitions/ComplexityProfile"
},
"tier": {
"$ref": "#/definitions/ComplexityTier"
}
},
"required": [
"profile",
"tier"
],
"type": "object"
},
"ComplexityMethod": {
"enum": [
"legacy_v1",
"capability_v2"
],
"type": "string"
},
"ComplexityProfile": {
"properties": {
"agent_owned_decomposition": {
"default": false,
"type": "boolean"
},
"ambiguity_level": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"artifact_count": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"coherent_long_horizon": {
"default": false,
"type": "boolean"
},
"compensable_mutations": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"coordination_edges": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"dependency_depth": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"durable_resume_cycles": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"external_systems": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"material_invalidation_events": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"parallel_branches": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"planning_depth": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"replan_loops": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"state_transitions": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"validation_loops": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"wake_cycles": {
"format": "uint8",
"minimum": 0,
"type": "integer"
}
},
"required": [
"ambiguity_level",
"artifact_count",
"coordination_edges",
"dependency_depth",
"external_systems",
"parallel_branches",
"planning_depth",
"state_transitions",
"validation_loops",
"wake_cycles"
],
"type": "object"
},
"ComplexityTier": {
"enum": [
"l0_atomic",
"l1_sequential",
"l2_stateful",
"l3_concurrent",
"l4_coordinated",
"l5_adaptive"
],
"type": "string"
},
"DeliverableContract": {
"properties": {
"artifacts": {
"items": {
"$ref": "#/definitions/ArtifactExpectation"
},
"type": "array"
},
"capture_before_cleanup": {
"type": "boolean"
},
"invariants": {
"items": {
"$ref": "#/definitions/InvariantSpec"
},
"type": "array"
},
"provenance_required": {
"type": "boolean"
}
},
"required": [
"artifacts",
"capture_before_cleanup",
"invariants",
"provenance_required"
],
"type": "object"
},
"ExecutionPolicy": {
"properties": {
"max_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_total_tokens": {
"description": "Shared Harness token budget. `None` leaves the Harness budget unbounded for this scenario.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_turns": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"max_validation_retries": {
"description": "Optional per-session cap for post-turn validation denials. `None` preserves the Harness-configured default.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stuck_timeout_seconds": {
"description": "Stop only after this many seconds without observable useful progress. Large scenarios have no fixed wall-clock deadline.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"max_turns",
"stuck_timeout_seconds"
],
"type": "object"
},
"ExecutionRealism": {
"enum": [
"synthetic",
"realistic_simulator",
"frozen_real_artifact"
],
"type": "string"
},
"HumanHorizon": {
"properties": {
"basis": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizonBasis"
}
],
"default": "unknown"
},
"max_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"min_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"HumanHorizonBasis": {
"enum": [
"unknown",
"author_estimate",
"measured"
],
"type": "string"
},
"InvariantSpec": {
"properties": {
"description": {
"type": "string"
},
"id": {
"type": "string"
}
},
"required": [
"description",
"id"
],
"type": "object"
},
"ScenarioCharacterization": {
"properties": {
"human_horizon": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizon"
}
],
"default": {
"basis": "unknown"
}
},
"realism": {
"allOf": [
{
"$ref": "#/definitions/ScenarioRealism"
}
],
"default": {
"execution": "synthetic",
"shadow": "none"
}
}
},
"type": "object"
},
"ScenarioDescriptor": {
"properties": {
"author_version": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"behavior_sha256": {
"type": [
"string",
"null"
]
},
"case_id": {
"type": "string"
},
"characterization": {
"$ref": "#/definitions/ScenarioCharacterization"
},
"classification": {
"$ref": "#/definitions/ComplexityClassification"
},
"compiled_sha256": {
"type": [
"string",
"null"
]
},
"complexity": {
"$ref": "#/definitions/ComplexityProfile"
},
"contract_sha256": {
"type": "string"
},
"deliverable_contract": {
"$ref": "#/definitions/DeliverableContract"
},
"inputs_sha256": {
"type": "string"
},
"origin": {
"$ref": "#/definitions/ScenarioOrigin"
},
"plans": {
"items": {
"type": "string"
},
"type": "array"
},
"required_capabilities": {
"items": {
"type": "string"
},
"type": "array"
},
"resource_envelope": {
"$ref": "#/definitions/ScenarioResourceEnvelope"
},
"scenario_id": {
"$ref": "#/definitions/ScenarioKey"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"source_path": {
"type": [
"string",
"null"
]
},
"source_sha256": {
"type": [
"string",
"null"
]
},
"title": {
"type": [
"string",
"null"
]
}
},
"required": [
"case_id",
"characterization",
"classification",
"complexity",
"contract_sha256",
"deliverable_contract",
"inputs_sha256",
"origin",
"required_capabilities",
"resource_envelope",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"ScenarioKey": {
"type": "string"
},
"ScenarioOrigin": {
"enum": [
"built_in",
"markdown",
"local"
],
"type": "string"
},
"ScenarioRealism": {
"properties": {
"execution": {
"allOf": [
{
"$ref": "#/definitions/ExecutionRealism"
}
],
"default": "synthetic"
},
"shadow": {
"allOf": [
{
"$ref": "#/definitions/ShadowMode"
}
],
"default": "none"
}
},
"type": "object"
},
"ScenarioResourceEnvelope": {
"properties": {
"execution": {
"$ref": "#/definitions/ExecutionPolicy"
},
"workflow": {
"anyOf": [
{
"$ref": "#/definitions/WorkflowResourceEnvelope"
},
{
"type": "null"
}
]
}
},
"required": [
"execution"
],
"type": "object"
},
"ShadowMode": {
"enum": [
"none",
"read_only"
],
"type": "string"
},
"WorkflowResourceEnvelope": {
"properties": {
"max_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"max_nodes": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"max_parallel": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"max_total_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"step_timeout_seconds": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"workflow_timeout_seconds": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"max_nodes",
"max_parallel",
"step_timeout_seconds",
"technical_retries",
"workflow_timeout_seconds"
],
"type": "object"
}
},
"properties": {
"scenario": {
"$ref": "#/definitions/ScenarioDescriptor"
}
},
"required": [
"scenario"
],
"title": "LocalScenarioCreateResponse",
"type": "object"
}
},
{
"description": "Create a draft local plan with an explicit small test scope.",
"metadata": {
"contract": {
"capability": "plan-create",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::plan-create",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"judge_model": {
"default": "",
"type": "string"
},
"judge_provider": {
"default": "",
"type": "string"
},
"label": {
"default": "",
"type": "string"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"purpose": {
"default": "",
"type": "string"
},
"runs": {
"default": 1,
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenarios": {
"items": {
"type": "string"
},
"type": "array"
},
"seed": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"default": 1,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"url": {
"type": "string"
}
},
"required": [
"model",
"provider",
"scenarios",
"url"
],
"title": "PlanCreateRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"PlanScopeItem": {
"properties": {
"case_id": {
"type": "string"
},
"complexity_tier": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"inputs_sha256": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"case_id",
"complexity_tier",
"contract_sha256",
"inputs_sha256",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"PlanState": {
"enum": [
"draft",
"baseline_running",
"baseline_ready",
"candidate_running",
"comparison_ready"
],
"type": "string"
}
},
"properties": {
"baseline_execution_id": {
"type": [
"string",
"null"
]
},
"candidate_execution_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"candidate_labels": {
"additionalProperties": {
"type": "string"
},
"default": {},
"type": "object"
},
"created_at": {
"type": "string"
},
"id": {
"type": "string"
},
"incomplete_execution_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"judge_model": {
"type": "string"
},
"judge_provider": {
"type": "string"
},
"label": {
"type": "string"
},
"last_attempt_id": {
"type": [
"string",
"null"
]
},
"locked": {
"type": "boolean"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"purpose": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenario_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"scenarios": {
"items": {
"$ref": "#/definitions/PlanScopeItem"
},
"type": "array"
},
"schema_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scope_hash": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"state": {
"$ref": "#/definitions/PlanState"
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"updated_at": {
"type": "string"
},
"url": {
"type": "string"
}
},
"required": [
"candidate_execution_ids",
"created_at",
"id",
"incomplete_execution_ids",
"judge_model",
"judge_provider",
"label",
"locked",
"model",
"provider",
"purpose",
"runs",
"scenario_ids",
"scenarios",
"schema_version",
"scope_hash",
"state",
"technical_retries",
"updated_at",
"url"
],
"title": "LocalPlan",
"type": "object"
}
},
{
"description": "Read one local plan.",
"metadata": {
"contract": {
"capability": "plan-get",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::plan-get",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"plan_id": {
"type": "string"
}
},
"required": [
"plan_id"
],
"title": "PlanGetRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"PlanScopeItem": {
"properties": {
"case_id": {
"type": "string"
},
"complexity_tier": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"inputs_sha256": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"case_id",
"complexity_tier",
"contract_sha256",
"inputs_sha256",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"PlanState": {
"enum": [
"draft",
"baseline_running",
"baseline_ready",
"candidate_running",
"comparison_ready"
],
"type": "string"
}
},
"properties": {
"baseline_execution_id": {
"type": [
"string",
"null"
]
},
"candidate_execution_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"candidate_labels": {
"additionalProperties": {
"type": "string"
},
"default": {},
"type": "object"
},
"created_at": {
"type": "string"
},
"id": {
"type": "string"
},
"incomplete_execution_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"judge_model": {
"type": "string"
},
"judge_provider": {
"type": "string"
},
"label": {
"type": "string"
},
"last_attempt_id": {
"type": [
"string",
"null"
]
},
"locked": {
"type": "boolean"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"purpose": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenario_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"scenarios": {
"items": {
"$ref": "#/definitions/PlanScopeItem"
},
"type": "array"
},
"schema_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scope_hash": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"state": {
"$ref": "#/definitions/PlanState"
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"updated_at": {
"type": "string"
},
"url": {
"type": "string"
}
},
"required": [
"candidate_execution_ids",
"created_at",
"id",
"incomplete_execution_ids",
"judge_model",
"judge_provider",
"label",
"locked",
"model",
"provider",
"purpose",
"runs",
"scenario_ids",
"scenarios",
"schema_version",
"scope_hash",
"state",
"technical_retries",
"updated_at",
"url"
],
"title": "LocalPlan",
"type": "object"
}
},
{
"description": "Start a baseline or candidate from a locked local plan.",
"metadata": {
"contract": {
"capability": "plan-run-start",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::plan-run-start",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"PlanRunRole": {
"enum": [
"baseline",
"candidate"
],
"type": "string"
}
},
"properties": {
"plan_id": {
"default": null,
"type": [
"string",
"null"
]
},
"role": {
"$ref": "#/definitions/PlanRunRole"
}
},
"required": [
"role"
],
"title": "PlanRunRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"PlanScopeItem": {
"properties": {
"case_id": {
"type": "string"
},
"complexity_tier": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"inputs_sha256": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"case_id",
"complexity_tier",
"contract_sha256",
"inputs_sha256",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"PlanState": {
"enum": [
"draft",
"baseline_running",
"baseline_ready",
"candidate_running",
"comparison_ready"
],
"type": "string"
}
},
"properties": {
"baseline_execution_id": {
"type": [
"string",
"null"
]
},
"candidate_execution_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"candidate_labels": {
"additionalProperties": {
"type": "string"
},
"default": {},
"type": "object"
},
"created_at": {
"type": "string"
},
"id": {
"type": "string"
},
"incomplete_execution_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"judge_model": {
"type": "string"
},
"judge_provider": {
"type": "string"
},
"label": {
"type": "string"
},
"last_attempt_id": {
"type": [
"string",
"null"
]
},
"locked": {
"type": "boolean"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"purpose": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenario_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"scenarios": {
"items": {
"$ref": "#/definitions/PlanScopeItem"
},
"type": "array"
},
"schema_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scope_hash": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"state": {
"$ref": "#/definitions/PlanState"
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"updated_at": {
"type": "string"
},
"url": {
"type": "string"
}
},
"required": [
"candidate_execution_ids",
"created_at",
"id",
"incomplete_execution_ids",
"judge_model",
"judge_provider",
"label",
"locked",
"model",
"provider",
"purpose",
"runs",
"scenario_ids",
"scenarios",
"schema_version",
"scope_hash",
"state",
"technical_retries",
"updated_at",
"url"
],
"title": "LocalPlan",
"type": "object"
}
},
{
"description": "Update an unlocked local plan or rename retained candidates.",
"metadata": {
"contract": {
"capability": "plan-update",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::plan-update",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"candidate_labels": {
"additionalProperties": {
"type": "string"
},
"type": [
"object",
"null"
]
},
"judge_model": {
"type": [
"string",
"null"
]
},
"judge_provider": {
"type": [
"string",
"null"
]
},
"label": {
"type": [
"string",
"null"
]
},
"model": {
"type": [
"string",
"null"
]
},
"plan_id": {
"default": null,
"type": [
"string",
"null"
]
},
"provider": {
"type": [
"string",
"null"
]
},
"purpose": {
"type": [
"string",
"null"
]
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"scenarios": {
"items": {
"type": "string"
},
"type": [
"array",
"null"
]
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"url": {
"type": [
"string",
"null"
]
}
},
"title": "PlanUpdateRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"PlanScopeItem": {
"properties": {
"case_id": {
"type": "string"
},
"complexity_tier": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"inputs_sha256": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"case_id",
"complexity_tier",
"contract_sha256",
"inputs_sha256",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"PlanState": {
"enum": [
"draft",
"baseline_running",
"baseline_ready",
"candidate_running",
"comparison_ready"
],
"type": "string"
}
},
"properties": {
"baseline_execution_id": {
"type": [
"string",
"null"
]
},
"candidate_execution_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"candidate_labels": {
"additionalProperties": {
"type": "string"
},
"default": {},
"type": "object"
},
"created_at": {
"type": "string"
},
"id": {
"type": "string"
},
"incomplete_execution_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"judge_model": {
"type": "string"
},
"judge_provider": {
"type": "string"
},
"label": {
"type": "string"
},
"last_attempt_id": {
"type": [
"string",
"null"
]
},
"locked": {
"type": "boolean"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"purpose": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenario_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"scenarios": {
"items": {
"$ref": "#/definitions/PlanScopeItem"
},
"type": "array"
},
"schema_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scope_hash": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"state": {
"$ref": "#/definitions/PlanState"
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"updated_at": {
"type": "string"
},
"url": {
"type": "string"
}
},
"required": [
"candidate_execution_ids",
"created_at",
"id",
"incomplete_execution_ids",
"judge_model",
"judge_provider",
"label",
"locked",
"model",
"provider",
"purpose",
"runs",
"scenario_ids",
"scenarios",
"schema_version",
"scope_hash",
"state",
"technical_retries",
"updated_at",
"url"
],
"title": "LocalPlan",
"type": "object"
}
},
{
"description": "List local plans and their baseline/candidate lifecycle.",
"metadata": {
"contract": {
"capability": "plans-list",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::plans-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "DashboardEmptyRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"LocalPlan": {
"properties": {
"baseline_execution_id": {
"type": [
"string",
"null"
]
},
"candidate_execution_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"candidate_labels": {
"additionalProperties": {
"type": "string"
},
"default": {},
"type": "object"
},
"created_at": {
"type": "string"
},
"id": {
"type": "string"
},
"incomplete_execution_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"judge_model": {
"type": "string"
},
"judge_provider": {
"type": "string"
},
"label": {
"type": "string"
},
"last_attempt_id": {
"type": [
"string",
"null"
]
},
"locked": {
"type": "boolean"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"purpose": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenario_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"scenarios": {
"items": {
"$ref": "#/definitions/PlanScopeItem"
},
"type": "array"
},
"schema_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scope_hash": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"state": {
"$ref": "#/definitions/PlanState"
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"updated_at": {
"type": "string"
},
"url": {
"type": "string"
}
},
"required": [
"candidate_execution_ids",
"created_at",
"id",
"incomplete_execution_ids",
"judge_model",
"judge_provider",
"label",
"locked",
"model",
"provider",
"purpose",
"runs",
"scenario_ids",
"scenarios",
"schema_version",
"scope_hash",
"state",
"technical_retries",
"updated_at",
"url"
],
"type": "object"
},
"PlanScopeItem": {
"properties": {
"case_id": {
"type": "string"
},
"complexity_tier": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"inputs_sha256": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"case_id",
"complexity_tier",
"contract_sha256",
"inputs_sha256",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"PlanState": {
"enum": [
"draft",
"baseline_running",
"baseline_ready",
"candidate_running",
"comparison_ready"
],
"type": "string"
}
},
"properties": {
"mode": {
"type": "string"
},
"plans": {
"items": {
"$ref": "#/definitions/LocalPlan"
},
"type": "array"
}
},
"required": [
"mode",
"plans"
],
"title": "PlansListResponse",
"type": "object"
}
},
{
"description": "Cancel the active local E2E execution.",
"metadata": {
"contract": {
"capability": "run-cancel",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::run-cancel",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "DashboardEmptyRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"Defaults": {
"properties": {
"judge_model": {
"type": "string"
},
"judge_provider": {
"type": "string"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"url": {
"type": "string"
}
},
"required": [
"judge_model",
"judge_provider",
"model",
"provider",
"runs",
"technical_retries",
"url"
],
"type": "object"
},
"JobStatus": {
"enum": [
"running",
"cancelling",
"cancelled",
"completed",
"failed"
],
"type": "string"
},
"JobView": {
"properties": {
"completed_at": {
"type": "string"
},
"error": {
"type": "string"
},
"id": {
"type": "string"
},
"label": {
"type": "string"
},
"log": {
"type": "string"
},
"log_from": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"log_offset": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"log_truncated": {
"type": "boolean"
},
"plan_context": {
"anyOf": [
{
"$ref": "#/definitions/PlanContext"
},
{
"type": "null"
}
],
"default": null
},
"request": {
"$ref": "#/definitions/RunRequest"
},
"returncode": {
"format": "int32",
"type": [
"integer",
"null"
]
},
"started_at": {
"type": "string"
},
"status": {
"$ref": "#/definitions/JobStatus"
}
},
"required": [
"completed_at",
"error",
"id",
"label",
"log",
"log_from",
"log_offset",
"log_truncated",
"request",
"started_at",
"status"
],
"type": "object"
},
"PlanContext": {
"properties": {
"plan_hash": {
"type": "string"
},
"plan_id": {
"type": "string"
},
"role": {
"$ref": "#/definitions/PlanRunRole"
}
},
"required": [
"plan_hash",
"plan_id",
"role"
],
"type": "object"
},
"PlanRunRole": {
"enum": [
"baseline",
"candidate"
],
"type": "string"
},
"RunRequest": {
"additionalProperties": false,
"properties": {
"judge_model": {
"default": "",
"type": "string"
},
"judge_provider": {
"default": "",
"type": "string"
},
"label": {
"default": "",
"type": "string"
},
"model": {
"type": "string"
},
"plan_context": {
"anyOf": [
{
"$ref": "#/definitions/PlanContext"
},
{
"type": "null"
}
],
"default": null
},
"provider": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenarios": {
"items": {
"type": "string"
},
"type": "array"
},
"seed": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"url": {
"type": "string"
}
},
"required": [
"model",
"provider",
"runs",
"scenarios",
"technical_retries",
"url"
],
"type": "object"
}
},
"properties": {
"defaults": {
"$ref": "#/definitions/Defaults"
},
"job": {
"anyOf": [
{
"$ref": "#/definitions/JobView"
},
{
"type": "null"
}
]
}
},
"required": [
"defaults"
],
"title": "RunSnapshot",
"type": "object"
}
},
{
"description": "Start one local E2E execution.",
"metadata": {
"contract": {
"capability": "run-start",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::run-start",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"PlanContext": {
"properties": {
"plan_hash": {
"type": "string"
},
"plan_id": {
"type": "string"
},
"role": {
"$ref": "#/definitions/PlanRunRole"
}
},
"required": [
"plan_hash",
"plan_id",
"role"
],
"type": "object"
},
"PlanRunRole": {
"enum": [
"baseline",
"candidate"
],
"type": "string"
}
},
"properties": {
"judge_model": {
"default": "",
"type": "string"
},
"judge_provider": {
"default": "",
"type": "string"
},
"label": {
"default": "",
"type": "string"
},
"model": {
"type": "string"
},
"plan_context": {
"anyOf": [
{
"$ref": "#/definitions/PlanContext"
},
{
"type": "null"
}
],
"default": null
},
"provider": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenarios": {
"items": {
"type": "string"
},
"type": "array"
},
"seed": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"url": {
"type": "string"
}
},
"required": [
"model",
"provider",
"runs",
"scenarios",
"technical_retries",
"url"
],
"title": "RunRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"Defaults": {
"properties": {
"judge_model": {
"type": "string"
},
"judge_provider": {
"type": "string"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"url": {
"type": "string"
}
},
"required": [
"judge_model",
"judge_provider",
"model",
"provider",
"runs",
"technical_retries",
"url"
],
"type": "object"
},
"JobStatus": {
"enum": [
"running",
"cancelling",
"cancelled",
"completed",
"failed"
],
"type": "string"
},
"JobView": {
"properties": {
"completed_at": {
"type": "string"
},
"error": {
"type": "string"
},
"id": {
"type": "string"
},
"label": {
"type": "string"
},
"log": {
"type": "string"
},
"log_from": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"log_offset": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"log_truncated": {
"type": "boolean"
},
"plan_context": {
"anyOf": [
{
"$ref": "#/definitions/PlanContext"
},
{
"type": "null"
}
],
"default": null
},
"request": {
"$ref": "#/definitions/RunRequest"
},
"returncode": {
"format": "int32",
"type": [
"integer",
"null"
]
},
"started_at": {
"type": "string"
},
"status": {
"$ref": "#/definitions/JobStatus"
}
},
"required": [
"completed_at",
"error",
"id",
"label",
"log",
"log_from",
"log_offset",
"log_truncated",
"request",
"started_at",
"status"
],
"type": "object"
},
"PlanContext": {
"properties": {
"plan_hash": {
"type": "string"
},
"plan_id": {
"type": "string"
},
"role": {
"$ref": "#/definitions/PlanRunRole"
}
},
"required": [
"plan_hash",
"plan_id",
"role"
],
"type": "object"
},
"PlanRunRole": {
"enum": [
"baseline",
"candidate"
],
"type": "string"
},
"RunRequest": {
"additionalProperties": false,
"properties": {
"judge_model": {
"default": "",
"type": "string"
},
"judge_provider": {
"default": "",
"type": "string"
},
"label": {
"default": "",
"type": "string"
},
"model": {
"type": "string"
},
"plan_context": {
"anyOf": [
{
"$ref": "#/definitions/PlanContext"
},
{
"type": "null"
}
],
"default": null
},
"provider": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenarios": {
"items": {
"type": "string"
},
"type": "array"
},
"seed": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"url": {
"type": "string"
}
},
"required": [
"model",
"provider",
"runs",
"scenarios",
"technical_retries",
"url"
],
"type": "object"
}
},
"properties": {
"defaults": {
"$ref": "#/definitions/Defaults"
},
"job": {
"anyOf": [
{
"$ref": "#/definitions/JobView"
},
{
"type": "null"
}
]
}
},
"required": [
"defaults"
],
"title": "RunSnapshot",
"type": "object"
}
},
{
"description": "Read local execution state and only the unread log suffix.",
"metadata": {
"contract": {
"capability": "run-status",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::run-status",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"after": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"title": "RunStatusRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"Defaults": {
"properties": {
"judge_model": {
"type": "string"
},
"judge_provider": {
"type": "string"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"url": {
"type": "string"
}
},
"required": [
"judge_model",
"judge_provider",
"model",
"provider",
"runs",
"technical_retries",
"url"
],
"type": "object"
},
"JobStatus": {
"enum": [
"running",
"cancelling",
"cancelled",
"completed",
"failed"
],
"type": "string"
},
"JobView": {
"properties": {
"completed_at": {
"type": "string"
},
"error": {
"type": "string"
},
"id": {
"type": "string"
},
"label": {
"type": "string"
},
"log": {
"type": "string"
},
"log_from": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"log_offset": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"log_truncated": {
"type": "boolean"
},
"plan_context": {
"anyOf": [
{
"$ref": "#/definitions/PlanContext"
},
{
"type": "null"
}
],
"default": null
},
"request": {
"$ref": "#/definitions/RunRequest"
},
"returncode": {
"format": "int32",
"type": [
"integer",
"null"
]
},
"started_at": {
"type": "string"
},
"status": {
"$ref": "#/definitions/JobStatus"
}
},
"required": [
"completed_at",
"error",
"id",
"label",
"log",
"log_from",
"log_offset",
"log_truncated",
"request",
"started_at",
"status"
],
"type": "object"
},
"PlanContext": {
"properties": {
"plan_hash": {
"type": "string"
},
"plan_id": {
"type": "string"
},
"role": {
"$ref": "#/definitions/PlanRunRole"
}
},
"required": [
"plan_hash",
"plan_id",
"role"
],
"type": "object"
},
"PlanRunRole": {
"enum": [
"baseline",
"candidate"
],
"type": "string"
},
"RunRequest": {
"additionalProperties": false,
"properties": {
"judge_model": {
"default": "",
"type": "string"
},
"judge_provider": {
"default": "",
"type": "string"
},
"label": {
"default": "",
"type": "string"
},
"model": {
"type": "string"
},
"plan_context": {
"anyOf": [
{
"$ref": "#/definitions/PlanContext"
},
{
"type": "null"
}
],
"default": null
},
"provider": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenarios": {
"items": {
"type": "string"
},
"type": "array"
},
"seed": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"url": {
"type": "string"
}
},
"required": [
"model",
"provider",
"runs",
"scenarios",
"technical_retries",
"url"
],
"type": "object"
}
},
"properties": {
"defaults": {
"$ref": "#/definitions/Defaults"
},
"job": {
"anyOf": [
{
"$ref": "#/definitions/JobView"
},
{
"type": "null"
}
]
}
},
"required": [
"defaults"
],
"title": "RunSnapshot",
"type": "object"
}
},
{
"description": "Read local metric history for one test version, with provider-grouped execution and judge models, without comparison actions.",
"metadata": {
"contract": {
"capability": "test-history-get",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::test-history-get",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"case_id": {
"default": null,
"type": [
"string",
"null"
]
},
"cursor": {
"default": null,
"type": [
"string",
"null"
]
},
"judge_model": {
"default": null,
"type": [
"string",
"null"
]
},
"judge_provider": {
"default": null,
"type": [
"string",
"null"
]
},
"limit": {
"default": null,
"format": "uint16",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"result": {
"default": null,
"type": [
"string",
"null"
]
},
"subject_model": {
"default": null,
"type": [
"string",
"null"
]
},
"subject_provider": {
"default": null,
"type": [
"string",
"null"
]
},
"system_version_id": {
"default": null,
"type": [
"string",
"null"
]
},
"test_id": {
"default": "",
"type": "string"
},
"test_version": {
"default": null,
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"title": "TestHistoryRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"AiAvailabilityCounts": {
"properties": {
"available": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"malformed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_requested": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"available",
"failed",
"malformed",
"not_evaluated",
"not_requested",
"unavailable"
],
"type": "object"
},
"AiVerdictCounts": {
"properties": {
"fail": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"inconclusive": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"pass": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"pass_with_concerns": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"fail",
"inconclusive",
"pass",
"pass_with_concerns"
],
"type": "object"
},
"AssessmentOutcomeCounts": {
"properties": {
"error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"partial": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"error",
"failed",
"not_evaluated",
"partial",
"passed",
"unavailable"
],
"type": "object"
},
"AssessmentSummary": {
"properties": {
"ai_availability": {
"$ref": "#/definitions/AiAvailabilityCounts"
},
"ai_verdicts": {
"$ref": "#/definitions/AiVerdictCounts"
},
"assessment_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"assessment_outcomes": {
"$ref": "#/definitions/AssessmentOutcomeCounts"
},
"asset_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"asset_qualitative_outcomes": {
"$ref": "#/definitions/AssessmentOutcomeCounts"
},
"asset_validation_outcomes": {
"$ref": "#/definitions/AssetValidationCounts"
},
"effective_statuses": {
"$ref": "#/definitions/StatusCounts"
},
"evidence_reference_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"median_confidence": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_quality_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"system_statuses": {
"$ref": "#/definitions/StatusCounts"
}
},
"required": [
"ai_availability",
"ai_verdicts",
"assessment_count",
"assessment_outcomes",
"asset_count",
"asset_qualitative_outcomes",
"asset_validation_outcomes",
"effective_statuses",
"evidence_reference_count",
"run_count",
"system_statuses"
],
"type": "object"
},
"AssetValidationCounts": {
"properties": {
"invalid": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"malformed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_produced": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"oversized": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"removed_during_cleanup": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unexpected": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unreadable": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unsafe_path": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"valid": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"invalid",
"malformed",
"not_evaluated",
"not_produced",
"oversized",
"removed_during_cleanup",
"unexpected",
"unreadable",
"unsafe_path",
"valid"
],
"type": "object"
},
"HistoryModelGroup": {
"description": "Models are exposed as provider groups so a model name is never ambiguous when two providers offer the same model identifier.",
"properties": {
"models": {
"items": {
"type": "string"
},
"type": "array"
},
"provider": {
"type": "string"
}
},
"required": [
"models",
"provider"
],
"type": "object"
},
"HistorySeries": {
"properties": {
"analyzer_profile_sha256": {
"type": "string"
},
"assessment_profile_sha256": {
"type": "string"
},
"case_id": {
"type": "string"
},
"cohort_id": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"engine_revision": {
"type": [
"string",
"null"
]
},
"execution_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"harness_revision": {
"type": [
"string",
"null"
]
},
"id": {
"type": "string"
},
"judge_model": {
"type": [
"string",
"null"
]
},
"judge_protocol": {
"type": [
"string",
"null"
]
},
"judge_provider": {
"type": [
"string",
"null"
]
},
"median_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_call_errors": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_turns": {
"format": "double",
"type": [
"number",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stack_mode": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"system_label": {
"type": "string"
},
"system_revision": {
"type": [
"string",
"null"
]
},
"system_version_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"analyzer_profile_sha256",
"assessment_profile_sha256",
"case_id",
"cohort_id",
"contract_sha256",
"execution_count",
"id",
"run_count",
"scenario_version",
"stack_mode",
"subject_model",
"subject_provider",
"system_label"
],
"type": "object"
},
"HistorySystem": {
"properties": {
"id": {
"type": "string"
},
"label": {
"type": "string"
}
},
"required": [
"id",
"label"
],
"type": "object"
},
"StatusCounts": {
"properties": {
"hard_gate_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"infrastructure_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"judge_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed_with_concerns": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"resource_limit": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"subject_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"hard_gate_failed",
"infrastructure_error",
"judge_error",
"passed",
"passed_with_concerns",
"resource_limit",
"subject_error",
"unavailable"
],
"type": "object"
},
"TestObservation": {
"properties": {
"analyzer_profile_sha256": {
"type": "string"
},
"assessment_profile_sha256": {
"type": "string"
},
"assessment_summary": {
"$ref": "#/definitions/AssessmentSummary"
},
"case_id": {
"type": "string"
},
"cohort_id": {
"type": "string"
},
"completed_at": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"engine_revision": {
"type": [
"string",
"null"
]
},
"evaluated_version_id": {
"type": [
"string",
"null"
]
},
"execution_id": {
"type": "string"
},
"harness_revision": {
"type": [
"string",
"null"
]
},
"judge_model": {
"type": [
"string",
"null"
]
},
"judge_protocol": {
"type": [
"string",
"null"
]
},
"judge_provider": {
"type": [
"string",
"null"
]
},
"median_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_call_errors": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_turns": {
"format": "double",
"type": [
"number",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scored_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stack_mode": {
"type": "string"
},
"status": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"system_label": {
"type": "string"
},
"system_revision": {
"type": [
"string",
"null"
]
},
"system_version_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"analyzer_profile_sha256",
"assessment_profile_sha256",
"assessment_summary",
"case_id",
"cohort_id",
"completed_at",
"contract_sha256",
"execution_id",
"run_count",
"scenario_version",
"scored_runs",
"stack_mode",
"status",
"subject_model",
"subject_provider",
"system_label"
],
"type": "object"
},
"VersionDescriptor": {
"properties": {
"execution_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"last_seen": {
"type": [
"string",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"execution_count",
"run_count",
"version"
],
"type": "object"
}
},
"properties": {
"available_versions": {
"items": {
"$ref": "#/definitions/VersionDescriptor"
},
"type": "array"
},
"cases": {
"items": {
"type": "string"
},
"type": "array"
},
"judge_models": {
"items": {
"$ref": "#/definitions/HistoryModelGroup"
},
"type": "array"
},
"next_cursor": {
"type": [
"string",
"null"
]
},
"observations": {
"items": {
"$ref": "#/definitions/TestObservation"
},
"type": "array"
},
"series": {
"items": {
"$ref": "#/definitions/HistorySeries"
},
"type": "array"
},
"subject_models": {
"items": {
"$ref": "#/definitions/HistoryModelGroup"
},
"type": "array"
},
"subjects": {
"items": {
"type": "string"
},
"type": "array"
},
"systems": {
"items": {
"$ref": "#/definitions/HistorySystem"
},
"type": "array"
},
"test_id": {
"type": "string"
},
"test_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"total": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"available_versions",
"cases",
"judge_models",
"observations",
"series",
"subject_models",
"subjects",
"systems",
"test_id",
"test_version",
"total"
],
"title": "TestHistoryResponse",
"type": "object"
}
},
{
"description": "Read one test version across two immutable evaluated-system versions.",
"metadata": {
"contract": {
"capability": "test-version-get",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::test-version-get",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"cohort_id": {
"type": "string"
},
"from_version_id": {
"type": "string"
},
"test_id": {
"type": "string"
},
"test_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"to_version_id": {
"type": "string"
}
},
"required": [
"cohort_id",
"from_version_id",
"test_id",
"test_version",
"to_version_id"
],
"title": "TestVersionGetRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"AiAvailabilityCounts": {
"properties": {
"available": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"malformed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_requested": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"available",
"failed",
"malformed",
"not_evaluated",
"not_requested",
"unavailable"
],
"type": "object"
},
"AiVerdictCounts": {
"properties": {
"fail": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"inconclusive": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"pass": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"pass_with_concerns": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"fail",
"inconclusive",
"pass",
"pass_with_concerns"
],
"type": "object"
},
"AssessmentOutcomeCounts": {
"properties": {
"error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"partial": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"error",
"failed",
"not_evaluated",
"partial",
"passed",
"unavailable"
],
"type": "object"
},
"AssessmentSummary": {
"properties": {
"ai_availability": {
"$ref": "#/definitions/AiAvailabilityCounts"
},
"ai_verdicts": {
"$ref": "#/definitions/AiVerdictCounts"
},
"assessment_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"assessment_outcomes": {
"$ref": "#/definitions/AssessmentOutcomeCounts"
},
"asset_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"asset_qualitative_outcomes": {
"$ref": "#/definitions/AssessmentOutcomeCounts"
},
"asset_validation_outcomes": {
"$ref": "#/definitions/AssetValidationCounts"
},
"effective_statuses": {
"$ref": "#/definitions/StatusCounts"
},
"evidence_reference_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"median_confidence": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_quality_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"system_statuses": {
"$ref": "#/definitions/StatusCounts"
}
},
"required": [
"ai_availability",
"ai_verdicts",
"assessment_count",
"assessment_outcomes",
"asset_count",
"asset_qualitative_outcomes",
"asset_validation_outcomes",
"effective_statuses",
"evidence_reference_count",
"run_count",
"system_statuses"
],
"type": "object"
},
"AssetValidationCounts": {
"properties": {
"invalid": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"malformed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_produced": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"oversized": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"removed_during_cleanup": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unexpected": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unreadable": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unsafe_path": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"valid": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"invalid",
"malformed",
"not_evaluated",
"not_produced",
"oversized",
"removed_during_cleanup",
"unexpected",
"unreadable",
"unsafe_path",
"valid"
],
"type": "object"
},
"MetricSamples": {
"properties": {
"cost_usd": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"duration_seconds": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"score": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"tokens": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"turns": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"cost_usd",
"duration_seconds",
"score",
"tokens",
"turns"
],
"type": "object"
},
"OutcomeCounts": {
"properties": {
"hard_gate_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"infra_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"technical_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"hard_gate_failed",
"infra_failed",
"passed",
"technical_failed"
],
"type": "object"
},
"StatusCounts": {
"properties": {
"hard_gate_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"infrastructure_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"judge_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed_with_concerns": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"resource_limit": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"subject_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"hard_gate_failed",
"infrastructure_error",
"judge_error",
"passed",
"passed_with_concerns",
"resource_limit",
"subject_error",
"unavailable"
],
"type": "object"
},
"TestDelta": {
"properties": {
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"score": {
"format": "double",
"type": [
"number",
"null"
]
},
"tokens": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"type": "object"
},
"TestObservation": {
"properties": {
"analyzer_profile_sha256": {
"type": "string"
},
"assessment_profile_sha256": {
"type": "string"
},
"assessment_summary": {
"$ref": "#/definitions/AssessmentSummary"
},
"case_id": {
"type": "string"
},
"cohort_id": {
"type": "string"
},
"completed_at": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"engine_revision": {
"type": [
"string",
"null"
]
},
"evaluated_version_id": {
"type": [
"string",
"null"
]
},
"execution_id": {
"type": "string"
},
"harness_revision": {
"type": [
"string",
"null"
]
},
"judge_model": {
"type": [
"string",
"null"
]
},
"judge_protocol": {
"type": [
"string",
"null"
]
},
"judge_provider": {
"type": [
"string",
"null"
]
},
"median_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_call_errors": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_turns": {
"format": "double",
"type": [
"number",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scored_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stack_mode": {
"type": "string"
},
"status": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"system_label": {
"type": "string"
},
"system_revision": {
"type": [
"string",
"null"
]
},
"system_version_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"analyzer_profile_sha256",
"assessment_profile_sha256",
"assessment_summary",
"case_id",
"cohort_id",
"completed_at",
"contract_sha256",
"execution_id",
"run_count",
"scenario_version",
"scored_runs",
"stack_mode",
"status",
"subject_model",
"subject_provider",
"system_label"
],
"type": "object"
},
"TestSideSummary": {
"properties": {
"assessment_summary": {
"$ref": "#/definitions/AssessmentSummary"
},
"case_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"evaluated_version_id": {
"type": "string"
},
"execution_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"median_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"outcomes": {
"$ref": "#/definitions/OutcomeCounts"
},
"pass_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"samples": {
"$ref": "#/definitions/MetricSamples"
},
"scored_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"total_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"assessment_summary",
"case_count",
"evaluated_version_id",
"execution_count",
"outcomes",
"samples",
"scored_runs",
"total_runs"
],
"type": "object"
}
},
"properties": {
"compatibility": {
"type": "string"
},
"compatibility_reasons": {
"items": {
"type": "string"
},
"type": "array"
},
"delta": {
"$ref": "#/definitions/TestDelta"
},
"from": {
"anyOf": [
{
"$ref": "#/definitions/TestSideSummary"
},
{
"type": "null"
}
]
},
"from_observations": {
"items": {
"$ref": "#/definitions/TestObservation"
},
"type": "array"
},
"test_id": {
"type": "string"
},
"test_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"to": {
"anyOf": [
{
"$ref": "#/definitions/TestSideSummary"
},
{
"type": "null"
}
]
},
"to_observations": {
"items": {
"$ref": "#/definitions/TestObservation"
},
"type": "array"
}
},
"required": [
"compatibility",
"compatibility_reasons",
"delta",
"from_observations",
"test_id",
"test_version",
"to_observations"
],
"title": "TestVersionResult",
"type": "object"
}
},
{
"description": "List the versioned test catalog with compact A/B results and stable pagination.",
"metadata": {
"contract": {
"capability": "tests-list",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::tests-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"cohort_id": {
"default": null,
"type": [
"string",
"null"
]
},
"cursor": {
"default": null,
"type": [
"string",
"null"
]
},
"from_version_id": {
"default": null,
"type": [
"string",
"null"
]
},
"limit": {
"default": null,
"format": "uint16",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"query": {
"default": null,
"type": [
"string",
"null"
]
},
"to_version_id": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "TestsListRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"AiAvailabilityCounts": {
"properties": {
"available": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"malformed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_requested": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"available",
"failed",
"malformed",
"not_evaluated",
"not_requested",
"unavailable"
],
"type": "object"
},
"AiVerdictCounts": {
"properties": {
"fail": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"inconclusive": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"pass": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"pass_with_concerns": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"fail",
"inconclusive",
"pass",
"pass_with_concerns"
],
"type": "object"
},
"AssessmentOutcomeCounts": {
"properties": {
"error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"partial": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"error",
"failed",
"not_evaluated",
"partial",
"passed",
"unavailable"
],
"type": "object"
},
"AssessmentSummary": {
"properties": {
"ai_availability": {
"$ref": "#/definitions/AiAvailabilityCounts"
},
"ai_verdicts": {
"$ref": "#/definitions/AiVerdictCounts"
},
"assessment_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"assessment_outcomes": {
"$ref": "#/definitions/AssessmentOutcomeCounts"
},
"asset_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"asset_qualitative_outcomes": {
"$ref": "#/definitions/AssessmentOutcomeCounts"
},
"asset_validation_outcomes": {
"$ref": "#/definitions/AssetValidationCounts"
},
"effective_statuses": {
"$ref": "#/definitions/StatusCounts"
},
"evidence_reference_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"median_confidence": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_quality_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"system_statuses": {
"$ref": "#/definitions/StatusCounts"
}
},
"required": [
"ai_availability",
"ai_verdicts",
"assessment_count",
"assessment_outcomes",
"asset_count",
"asset_qualitative_outcomes",
"asset_validation_outcomes",
"effective_statuses",
"evidence_reference_count",
"run_count",
"system_statuses"
],
"type": "object"
},
"AssetValidationCounts": {
"properties": {
"invalid": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"malformed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_produced": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"oversized": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"removed_during_cleanup": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unexpected": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unreadable": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unsafe_path": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"valid": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"invalid",
"malformed",
"not_evaluated",
"not_produced",
"oversized",
"removed_during_cleanup",
"unexpected",
"unreadable",
"unsafe_path",
"valid"
],
"type": "object"
},
"CalibrationProjection": {
"properties": {
"compatible_sample_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"maturity": {
"type": "string"
}
},
"required": [
"compatible_sample_count",
"maturity"
],
"type": "object"
},
"ComplexityClassification": {
"properties": {
"method": {
"allOf": [
{
"$ref": "#/definitions/ComplexityMethod"
}
],
"default": "legacy_v1"
},
"profile": {
"$ref": "#/definitions/ComplexityProfile"
},
"tier": {
"$ref": "#/definitions/ComplexityTier"
}
},
"required": [
"profile",
"tier"
],
"type": "object"
},
"ComplexityMethod": {
"enum": [
"legacy_v1",
"capability_v2"
],
"type": "string"
},
"ComplexityProfile": {
"properties": {
"agent_owned_decomposition": {
"default": false,
"type": "boolean"
},
"ambiguity_level": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"artifact_count": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"coherent_long_horizon": {
"default": false,
"type": "boolean"
},
"compensable_mutations": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"coordination_edges": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"dependency_depth": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"durable_resume_cycles": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"external_systems": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"material_invalidation_events": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"parallel_branches": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"planning_depth": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"replan_loops": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"state_transitions": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"validation_loops": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"wake_cycles": {
"format": "uint8",
"minimum": 0,
"type": "integer"
}
},
"required": [
"ambiguity_level",
"artifact_count",
"coordination_edges",
"dependency_depth",
"external_systems",
"parallel_branches",
"planning_depth",
"state_transitions",
"validation_loops",
"wake_cycles"
],
"type": "object"
},
"ComplexityTier": {
"enum": [
"l0_atomic",
"l1_sequential",
"l2_stateful",
"l3_concurrent",
"l4_coordinated",
"l5_adaptive"
],
"type": "string"
},
"ExecutionRealism": {
"enum": [
"synthetic",
"realistic_simulator",
"frozen_real_artifact"
],
"type": "string"
},
"HumanHorizon": {
"properties": {
"basis": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizonBasis"
}
],
"default": "unknown"
},
"max_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"min_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"HumanHorizonBasis": {
"enum": [
"unknown",
"author_estimate",
"measured"
],
"type": "string"
},
"MetricSamples": {
"properties": {
"cost_usd": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"duration_seconds": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"score": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"tokens": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"turns": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"cost_usd",
"duration_seconds",
"score",
"tokens",
"turns"
],
"type": "object"
},
"OutcomeCounts": {
"properties": {
"hard_gate_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"infra_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"technical_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"hard_gate_failed",
"infra_failed",
"passed",
"technical_failed"
],
"type": "object"
},
"ScenarioCharacterization": {
"properties": {
"human_horizon": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizon"
}
],
"default": {
"basis": "unknown"
}
},
"realism": {
"allOf": [
{
"$ref": "#/definitions/ScenarioRealism"
}
],
"default": {
"execution": "synthetic",
"shadow": "none"
}
}
},
"type": "object"
},
"ScenarioRealism": {
"properties": {
"execution": {
"allOf": [
{
"$ref": "#/definitions/ExecutionRealism"
}
],
"default": "synthetic"
},
"shadow": {
"allOf": [
{
"$ref": "#/definitions/ShadowMode"
}
],
"default": "none"
}
},
"type": "object"
},
"ShadowMode": {
"enum": [
"none",
"read_only"
],
"type": "string"
},
"StatusCounts": {
"properties": {
"hard_gate_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"infrastructure_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"judge_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed_with_concerns": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"resource_limit": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"subject_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"hard_gate_failed",
"infrastructure_error",
"judge_error",
"passed",
"passed_with_concerns",
"resource_limit",
"subject_error",
"unavailable"
],
"type": "object"
},
"TestCatalogRow": {
"properties": {
"available_versions": {
"items": {
"$ref": "#/definitions/VersionDescriptor"
},
"type": "array"
},
"calibration": {
"anyOf": [
{
"$ref": "#/definitions/CalibrationProjection"
},
{
"type": "null"
}
]
},
"characterization": {
"anyOf": [
{
"$ref": "#/definitions/ScenarioCharacterization"
},
{
"type": "null"
}
]
},
"complexity": {
"anyOf": [
{
"$ref": "#/definitions/ComplexityClassification"
},
{
"type": "null"
}
]
},
"current_version": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"lifecycle": {
"type": "string"
},
"result": {
"anyOf": [
{
"$ref": "#/definitions/TestVersionResult"
},
{
"type": "null"
}
]
},
"selected_version": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"test_id": {
"type": "string"
}
},
"required": [
"available_versions",
"lifecycle",
"test_id"
],
"type": "object"
},
"TestDelta": {
"properties": {
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"score": {
"format": "double",
"type": [
"number",
"null"
]
},
"tokens": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"type": "object"
},
"TestObservation": {
"properties": {
"analyzer_profile_sha256": {
"type": "string"
},
"assessment_profile_sha256": {
"type": "string"
},
"assessment_summary": {
"$ref": "#/definitions/AssessmentSummary"
},
"case_id": {
"type": "string"
},
"cohort_id": {
"type": "string"
},
"completed_at": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"engine_revision": {
"type": [
"string",
"null"
]
},
"evaluated_version_id": {
"type": [
"string",
"null"
]
},
"execution_id": {
"type": "string"
},
"harness_revision": {
"type": [
"string",
"null"
]
},
"judge_model": {
"type": [
"string",
"null"
]
},
"judge_protocol": {
"type": [
"string",
"null"
]
},
"judge_provider": {
"type": [
"string",
"null"
]
},
"median_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_call_errors": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_turns": {
"format": "double",
"type": [
"number",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scored_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stack_mode": {
"type": "string"
},
"status": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"system_label": {
"type": "string"
},
"system_revision": {
"type": [
"string",
"null"
]
},
"system_version_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"analyzer_profile_sha256",
"assessment_profile_sha256",
"assessment_summary",
"case_id",
"cohort_id",
"completed_at",
"contract_sha256",
"execution_id",
"run_count",
"scenario_version",
"scored_runs",
"stack_mode",
"status",
"subject_model",
"subject_provider",
"system_label"
],
"type": "object"
},
"TestSideSummary": {
"properties": {
"assessment_summary": {
"$ref": "#/definitions/AssessmentSummary"
},
"case_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"evaluated_version_id": {
"type": "string"
},
"execution_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"median_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"outcomes": {
"$ref": "#/definitions/OutcomeCounts"
},
"pass_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"samples": {
"$ref": "#/definitions/MetricSamples"
},
"scored_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"total_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"assessment_summary",
"case_count",
"evaluated_version_id",
"execution_count",
"outcomes",
"samples",
"scored_runs",
"total_runs"
],
"type": "object"
},
"TestVersionResult": {
"properties": {
"compatibility": {
"type": "string"
},
"compatibility_reasons": {
"items": {
"type": "string"
},
"type": "array"
},
"delta": {
"$ref": "#/definitions/TestDelta"
},
"from": {
"anyOf": [
{
"$ref": "#/definitions/TestSideSummary"
},
{
"type": "null"
}
]
},
"from_observations": {
"items": {
"$ref": "#/definitions/TestObservation"
},
"type": "array"
},
"test_id": {
"type": "string"
},
"test_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"to": {
"anyOf": [
{
"$ref": "#/definitions/TestSideSummary"
},
{
"type": "null"
}
]
},
"to_observations": {
"items": {
"$ref": "#/definitions/TestObservation"
},
"type": "array"
}
},
"required": [
"compatibility",
"compatibility_reasons",
"delta",
"from_observations",
"test_id",
"test_version",
"to_observations"
],
"type": "object"
},
"VersionDescriptor": {
"properties": {
"execution_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"last_seen": {
"type": [
"string",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"execution_count",
"run_count",
"version"
],
"type": "object"
}
},
"properties": {
"next_cursor": {
"type": [
"string",
"null"
]
},
"revision": {
"type": "string"
},
"rows": {
"items": {
"$ref": "#/definitions/TestCatalogRow"
},
"type": "array"
},
"total": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"revision",
"rows",
"total"
],
"title": "TestsListResponse",
"type": "object"
}
},
{
"description": "Classify a protected supervisor's fault journal against canonical execution evidence.",
"metadata": {
"contract": {
"capabilities": [
"fault-evaluate"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::fault-evaluate",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"CancellationRule": {
"additionalProperties": false,
"properties": {
"after_action": {
"type": "string"
},
"grace_period_ms": {
"default": 30000,
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"after_action"
],
"type": "object"
},
"DelayRule": {
"additionalProperties": false,
"properties": {
"delay_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"occurrences": {
"default": 1,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"target": {
"type": "string"
}
},
"required": [
"delay_ms",
"target"
],
"type": "object"
},
"DisconnectRule": {
"additionalProperties": false,
"properties": {
"after_action": {
"type": "string"
},
"duration_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"target": {
"type": "string"
}
},
"required": [
"after_action",
"duration_ms",
"target"
],
"type": "object"
},
"ExpectedTerminalOutcome": {
"oneOf": [
{
"enum": [
"recovered",
"cancelled"
],
"type": "string"
},
{
"description": "The subject is expected to finish WITHOUT a correct deliverable: a hard-gate-failed but structurally clean, fully cleaned-up, bounded run. A wrong-but-structural result is the expectation, so it classifies as `correct_recovery`; a passed deliverable means the perturbation failed to bite and is a benchmark infrastructure failure. Degraded profiles must not declare a cancellation rule and require canonical results.",
"enum": [
"degraded"
],
"type": "string"
}
]
},
"FaultAction": {
"additionalProperties": false,
"properties": {
"action_id": {
"type": "string"
},
"duration_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"kind": {
"$ref": "#/definitions/FaultActionKind"
},
"sequence": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"target": {
"type": "string"
},
"trigger_attempt": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"action_id",
"kind",
"sequence",
"target"
],
"type": "object"
},
"FaultActionKind": {
"enum": [
"delay_call",
"fail_first_attempt",
"malformed_result",
"throttle_burst",
"duplicate_event",
"child_timeout",
"transient_disconnect",
"out_of_order_result",
"cancel_execution"
],
"type": "string"
},
"FaultJournal": {
"additionalProperties": false,
"properties": {
"actions": {
"items": {
"$ref": "#/definitions/FaultObservation"
},
"type": "array"
},
"cancellation_observed": {
"type": "boolean"
},
"cleanup_completed": {
"type": "boolean"
},
"cleanup_evidence": {
"additionalProperties": {
"type": "string"
},
"default": {},
"type": "object"
},
"completed_at": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"infrastructure_failures": {
"default": [],
"items": {
"type": "string"
},
"type": "array"
},
"lane": {
"type": "string"
},
"observed_work_amplification": {
"format": "double",
"type": [
"number",
"null"
]
},
"profile_sha256": {
"type": "string"
},
"results_sha256": {
"type": [
"string",
"null"
]
},
"started_at": {
"type": "string"
}
},
"required": [
"actions",
"cancellation_observed",
"cleanup_completed",
"completed_at",
"execution_id",
"lane",
"profile_sha256",
"started_at"
],
"type": "object"
},
"FaultObservation": {
"additionalProperties": false,
"properties": {
"action_id": {
"type": "string"
},
"applied_at": {
"type": [
"string",
"null"
]
},
"detail": {
"default": "",
"type": "string"
},
"kind": {
"$ref": "#/definitions/FaultActionKind"
},
"recovered_at": {
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/FaultObservationStatus"
}
},
"required": [
"action_id",
"kind",
"status"
],
"type": "object"
},
"FaultObservationStatus": {
"enum": [
"recovered",
"unrecovered",
"not_triggered",
"injector_failed"
],
"type": "string"
},
"FaultPlan": {
"additionalProperties": false,
"properties": {
"actions": {
"items": {
"$ref": "#/definitions/FaultAction"
},
"type": "array"
},
"expected_outcome": {
"$ref": "#/definitions/ExpectedTerminalOutcome"
},
"profile_id": {
"type": "string"
},
"profile_sha256": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"actions",
"expected_outcome",
"profile_id",
"profile_sha256",
"seed"
],
"type": "object"
},
"FaultProfile": {
"additionalProperties": false,
"properties": {
"cancellation": {
"anyOf": [
{
"$ref": "#/definitions/CancellationRule"
},
{
"type": "null"
}
],
"default": null
},
"child_timeout_ms": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"delayed_calls": {
"default": [],
"items": {
"$ref": "#/definitions/DelayRule"
},
"type": "array"
},
"duplicate_events": {
"default": false,
"type": "boolean"
},
"expected_outcome": {
"allOf": [
{
"$ref": "#/definitions/ExpectedTerminalOutcome"
}
],
"default": "recovered"
},
"fail_first_attempt": {
"default": [],
"items": {
"type": "string"
},
"type": "array"
},
"malformed_results": {
"default": [],
"description": "The supervisor corrupts the result payload of the first call to each named target: the subject receives undecodable, contract-breaking bytes exactly once per target.",
"items": {
"type": "string"
},
"type": "array"
},
"max_work_amplification": {
"format": "double",
"type": "number"
},
"out_of_order_results": {
"default": false,
"type": "boolean"
},
"profile_id": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"throttle_burst": {
"anyOf": [
{
"$ref": "#/definitions/ThrottleRule"
},
{
"type": "null"
}
],
"default": null
},
"transient_disconnect": {
"anyOf": [
{
"$ref": "#/definitions/DisconnectRule"
},
{
"type": "null"
}
],
"default": null
}
},
"required": [
"max_work_amplification",
"profile_id",
"seed"
],
"type": "object"
},
"ThrottleRule": {
"additionalProperties": false,
"description": "The supervisor answers the target's next `responses` calls with a provider-throttle (429-style) rejection, spaced `spacing_ms` apart.",
"properties": {
"responses": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"spacing_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"target": {
"type": "string"
}
},
"required": [
"responses",
"spacing_ms",
"target"
],
"type": "object"
}
},
"properties": {
"execution_id": {
"type": "string"
},
"journal": {
"$ref": "#/definitions/FaultJournal"
},
"plan": {
"$ref": "#/definitions/FaultPlan"
},
"profile": {
"$ref": "#/definitions/FaultProfile"
}
},
"required": [
"execution_id",
"journal",
"plan",
"profile"
],
"title": "FaultEvaluateRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"RecoveryClassification": {
"enum": [
"correct_recovery",
"excessive_recovery",
"incorrect_result",
"structural_failure",
"infrastructure_failure"
],
"type": "string"
}
},
"properties": {
"cancellation_observed": {
"type": "boolean"
},
"classification": {
"$ref": "#/definitions/RecoveryClassification"
},
"cleanup_completed": {
"type": "boolean"
},
"deliverable_correct": {
"type": [
"boolean",
"null"
]
},
"execution_id": {
"type": "string"
},
"max_work_amplification": {
"format": "double",
"type": "number"
},
"observed_work_amplification": {
"format": "double",
"type": [
"number",
"null"
]
},
"planned_actions": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"profile_id": {
"type": "string"
},
"profile_sha256": {
"type": "string"
},
"reasons": {
"items": {
"type": "string"
},
"type": "array"
},
"recovered_actions": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"results_sha256": {
"type": [
"string",
"null"
]
},
"structural_integrity": {
"type": [
"boolean",
"null"
]
}
},
"required": [
"cancellation_observed",
"classification",
"cleanup_completed",
"execution_id",
"max_work_amplification",
"planned_actions",
"profile_id",
"profile_sha256",
"reasons",
"recovered_actions"
],
"title": "FaultEvaluation",
"type": "object"
}
},
{
"description": "Materialize a deterministic, versioned fault plan for a protected supervisor.",
"metadata": {
"contract": {
"capabilities": [
"fault-plan"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::fault-plan",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"CancellationRule": {
"additionalProperties": false,
"properties": {
"after_action": {
"type": "string"
},
"grace_period_ms": {
"default": 30000,
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"after_action"
],
"type": "object"
},
"DelayRule": {
"additionalProperties": false,
"properties": {
"delay_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"occurrences": {
"default": 1,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"target": {
"type": "string"
}
},
"required": [
"delay_ms",
"target"
],
"type": "object"
},
"DisconnectRule": {
"additionalProperties": false,
"properties": {
"after_action": {
"type": "string"
},
"duration_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"target": {
"type": "string"
}
},
"required": [
"after_action",
"duration_ms",
"target"
],
"type": "object"
},
"ExpectedTerminalOutcome": {
"oneOf": [
{
"enum": [
"recovered",
"cancelled"
],
"type": "string"
},
{
"description": "The subject is expected to finish WITHOUT a correct deliverable: a hard-gate-failed but structurally clean, fully cleaned-up, bounded run. A wrong-but-structural result is the expectation, so it classifies as `correct_recovery`; a passed deliverable means the perturbation failed to bite and is a benchmark infrastructure failure. Degraded profiles must not declare a cancellation rule and require canonical results.",
"enum": [
"degraded"
],
"type": "string"
}
]
},
"FaultProfile": {
"additionalProperties": false,
"properties": {
"cancellation": {
"anyOf": [
{
"$ref": "#/definitions/CancellationRule"
},
{
"type": "null"
}
],
"default": null
},
"child_timeout_ms": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"delayed_calls": {
"default": [],
"items": {
"$ref": "#/definitions/DelayRule"
},
"type": "array"
},
"duplicate_events": {
"default": false,
"type": "boolean"
},
"expected_outcome": {
"allOf": [
{
"$ref": "#/definitions/ExpectedTerminalOutcome"
}
],
"default": "recovered"
},
"fail_first_attempt": {
"default": [],
"items": {
"type": "string"
},
"type": "array"
},
"malformed_results": {
"default": [],
"description": "The supervisor corrupts the result payload of the first call to each named target: the subject receives undecodable, contract-breaking bytes exactly once per target.",
"items": {
"type": "string"
},
"type": "array"
},
"max_work_amplification": {
"format": "double",
"type": "number"
},
"out_of_order_results": {
"default": false,
"type": "boolean"
},
"profile_id": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"throttle_burst": {
"anyOf": [
{
"$ref": "#/definitions/ThrottleRule"
},
{
"type": "null"
}
],
"default": null
},
"transient_disconnect": {
"anyOf": [
{
"$ref": "#/definitions/DisconnectRule"
},
{
"type": "null"
}
],
"default": null
}
},
"required": [
"max_work_amplification",
"profile_id",
"seed"
],
"type": "object"
},
"ThrottleRule": {
"additionalProperties": false,
"description": "The supervisor answers the target's next `responses` calls with a provider-throttle (429-style) rejection, spaced `spacing_ms` apart.",
"properties": {
"responses": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"spacing_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"target": {
"type": "string"
}
},
"required": [
"responses",
"spacing_ms",
"target"
],
"type": "object"
}
},
"properties": {
"profile": {
"$ref": "#/definitions/FaultProfile"
}
},
"required": [
"profile"
],
"title": "FaultPlanRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"ExpectedTerminalOutcome": {
"oneOf": [
{
"enum": [
"recovered",
"cancelled"
],
"type": "string"
},
{
"description": "The subject is expected to finish WITHOUT a correct deliverable: a hard-gate-failed but structurally clean, fully cleaned-up, bounded run. A wrong-but-structural result is the expectation, so it classifies as `correct_recovery`; a passed deliverable means the perturbation failed to bite and is a benchmark infrastructure failure. Degraded profiles must not declare a cancellation rule and require canonical results.",
"enum": [
"degraded"
],
"type": "string"
}
]
},
"FaultAction": {
"additionalProperties": false,
"properties": {
"action_id": {
"type": "string"
},
"duration_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"kind": {
"$ref": "#/definitions/FaultActionKind"
},
"sequence": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"target": {
"type": "string"
},
"trigger_attempt": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"action_id",
"kind",
"sequence",
"target"
],
"type": "object"
},
"FaultActionKind": {
"enum": [
"delay_call",
"fail_first_attempt",
"malformed_result",
"throttle_burst",
"duplicate_event",
"child_timeout",
"transient_disconnect",
"out_of_order_result",
"cancel_execution"
],
"type": "string"
}
},
"properties": {
"actions": {
"items": {
"$ref": "#/definitions/FaultAction"
},
"type": "array"
},
"expected_outcome": {
"$ref": "#/definitions/ExpectedTerminalOutcome"
},
"profile_id": {
"type": "string"
},
"profile_sha256": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"actions",
"expected_outcome",
"profile_id",
"profile_sha256",
"seed"
],
"title": "FaultPlan",
"type": "object"
}
},
{
"description": "List hash-validated E2E history records from the iii database worker.",
"metadata": {
"contract": {
"capabilities": [
"history-list"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::history-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"lane": {
"default": null,
"type": [
"string",
"null"
]
},
"limit": {
"default": 100,
"format": "uint16",
"minimum": 0,
"type": "integer"
}
},
"title": "HistoryListRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"DurableArchiveReference": {
"additionalProperties": false,
"properties": {
"archive_id": {
"type": "string"
},
"created_at": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"expires_at": {
"type": [
"string",
"null"
]
},
"identity_sha256": {
"type": "string"
},
"manifest": {
"$ref": "#/definitions/StorageObjectReference"
},
"manifest_backup": {
"$ref": "#/definitions/StorageObjectReference"
},
"retention_class": {
"$ref": "#/definitions/RetentionClass"
}
},
"required": [
"archive_id",
"created_at",
"execution_id",
"identity_sha256",
"manifest",
"manifest_backup",
"retention_class"
],
"type": "object"
},
"HistoryRecord": {
"additionalProperties": false,
"properties": {
"archive": {
"$ref": "#/definitions/DurableArchiveReference"
},
"case_count": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"e2e_repository": {
"type": "string"
},
"e2e_revision": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"identity_sha256": {
"type": "string"
},
"ingestion_id": {
"type": "string"
},
"lane": {
"type": "string"
},
"occurred_at": {
"type": "string"
},
"passed": {
"type": "boolean"
},
"stack_mode": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"subject_revision": {
"type": "string"
}
},
"required": [
"archive",
"case_count",
"e2e_repository",
"e2e_revision",
"execution_id",
"identity_sha256",
"ingestion_id",
"lane",
"occurred_at",
"passed",
"stack_mode",
"subject_model",
"subject_provider",
"subject_revision"
],
"type": "object"
},
"RetentionClass": {
"enum": [
"temporary",
"pull_request",
"longitudinal",
"canonical"
],
"type": "string"
},
"StorageObjectReference": {
"properties": {
"bucket": {
"type": "string"
},
"key": {
"type": "string"
},
"media_type": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"uri": {
"type": "string"
}
},
"required": [
"bucket",
"key",
"media_type",
"sha256",
"size_bytes",
"uri"
],
"type": "object"
}
},
"properties": {
"records": {
"items": {
"$ref": "#/definitions/HistoryRecord"
},
"type": "array"
}
},
"required": [
"records"
],
"title": "HistoryListResponse",
"type": "object"
}
},
{
"description": "Get the immutable result and manifest of an E2E execution.",
"metadata": {
"contract": {
"capabilities": [
"results-get"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::results-get",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
}
},
"required": [
"execution_id"
],
"title": "ExecutionRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ActivationPolicy": {
"oneOf": [
{
"properties": {
"policy": {
"enum": [
"always"
],
"type": "string"
}
},
"required": [
"policy"
],
"type": "object"
},
{
"properties": {
"conditions": {
"items": {
"$ref": "#/definitions/BooleanCondition"
},
"type": "array"
},
"policy": {
"enum": [
"all"
],
"type": "string"
}
},
"required": [
"conditions",
"policy"
],
"type": "object"
},
{
"properties": {
"conditions": {
"items": {
"$ref": "#/definitions/BooleanCondition"
},
"type": "array"
},
"policy": {
"enum": [
"any"
],
"type": "string"
}
},
"required": [
"conditions",
"policy"
],
"type": "object"
}
]
},
"AdherenceAvailability": {
"enum": [
"available",
"unavailable",
"failed"
],
"type": "string"
},
"AdherenceRequirement": {
"properties": {
"confidence": {
"format": "double",
"type": "number"
},
"evidence": {
"items": {
"type": "string"
},
"type": "array"
},
"followed": {
"type": "boolean"
},
"id": {
"type": "string"
},
"instruction": {
"type": "string"
},
"reason": {
"type": "string"
}
},
"required": [
"confidence",
"followed",
"id",
"instruction",
"reason"
],
"type": "object"
},
"AiAssessmentAvailability": {
"enum": [
"not_requested",
"not_evaluated",
"available",
"unavailable",
"malformed",
"failed"
],
"type": "string"
},
"AiFinalAssessment": {
"additionalProperties": false,
"properties": {
"analyzer": {
"anyOf": [
{
"$ref": "#/definitions/AnalyzerIdentity"
},
{
"type": "null"
}
]
},
"analyzer_usage": {
"anyOf": [
{
"$ref": "#/definitions/AnalyzerUsage"
},
{
"type": "null"
}
]
},
"availability": {
"$ref": "#/definitions/AiAssessmentAvailability"
},
"reason": {
"type": [
"string",
"null"
]
},
"result": {
"anyOf": [
{
"$ref": "#/definitions/FinalAssessmentResult"
},
{
"type": "null"
}
]
}
},
"required": [
"availability"
],
"type": "object"
},
"AiVerdict": {
"enum": [
"pass",
"pass_with_concerns",
"fail",
"inconclusive"
],
"type": "string"
},
"AnalyzerIdentity": {
"additionalProperties": false,
"properties": {
"analyzer": {
"type": "string"
},
"input_sha256": {
"type": "string"
},
"model": {
"type": [
"string",
"null"
]
},
"provider": {
"type": [
"string",
"null"
]
}
},
"required": [
"analyzer",
"input_sha256"
],
"type": "object"
},
"AnalyzerUsage": {
"properties": {
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"latency_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"ArtifactExpectation": {
"properties": {
"id": {
"type": "string"
},
"kind": {
"type": "string"
},
"max_size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"media_type": {
"type": "string"
},
"schema": true
},
"required": [
"id",
"kind",
"max_size_bytes",
"media_type",
"schema"
],
"type": "object"
},
"ArtifactReference": {
"properties": {
"id": {
"type": "string"
},
"kind": {
"type": "string"
},
"media_type": {
"type": "string"
},
"path": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"id",
"kind",
"media_type",
"path",
"sha256",
"size_bytes"
],
"type": "object"
},
"AssessmentContract": {
"additionalProperties": false,
"properties": {
"runs": {
"items": {
"$ref": "#/definitions/RunAssessmentContract"
},
"type": "array"
}
},
"required": [
"runs"
],
"type": "object"
},
"AssessmentKind": {
"enum": [
"required_check",
"signal",
"asset_validation",
"asset_quality"
],
"type": "string"
},
"AssessmentOutcome": {
"enum": [
"passed",
"failed",
"partial",
"not_evaluated",
"unavailable",
"error"
],
"type": "string"
},
"AssessmentPolicy": {
"enum": [
"hard_gate",
"advisory"
],
"type": "string"
},
"AssessmentResult": {
"properties": {
"analyzer": {
"anyOf": [
{
"$ref": "#/definitions/AnalyzerIdentity"
},
{
"type": "null"
}
]
},
"analyzer_usage": {
"anyOf": [
{
"$ref": "#/definitions/AnalyzerUsage"
},
{
"type": "null"
}
]
},
"confidence": {
"format": "double",
"type": [
"number",
"null"
]
},
"criterion_id": {
"type": "string"
},
"dimension": {
"$ref": "#/definitions/EvaluationDimension"
},
"evidence": {
"items": {
"$ref": "#/definitions/EvidenceReference"
},
"type": "array"
},
"kind": {
"$ref": "#/definitions/AssessmentKind"
},
"outcome": {
"$ref": "#/definitions/AssessmentOutcome"
},
"policy": {
"$ref": "#/definitions/AssessmentPolicy"
},
"score": {
"anyOf": [
{
"$ref": "#/definitions/AssessmentScore"
},
{
"type": "null"
}
]
},
"source": {
"$ref": "#/definitions/AssessmentSource"
},
"summary": {
"type": "string"
},
"target": {
"$ref": "#/definitions/AssessmentTarget"
}
},
"required": [
"criterion_id",
"dimension",
"kind",
"outcome",
"policy",
"source",
"summary",
"target"
],
"type": "object"
},
"AssessmentScore": {
"properties": {
"awarded": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"possible": {
"format": "uint8",
"minimum": 0,
"type": "integer"
}
},
"required": [
"awarded",
"possible"
],
"type": "object"
},
"AssessmentSource": {
"enum": [
"deterministic",
"judge",
"asset_analyzer"
],
"type": "string"
},
"AssessmentTarget": {
"properties": {
"id": {
"type": "string"
},
"kind": {
"$ref": "#/definitions/AssessmentTargetKind"
}
},
"required": [
"id",
"kind"
],
"type": "object"
},
"AssessmentTargetKind": {
"enum": [
"criterion",
"asset"
],
"type": "string"
},
"AssetAssessmentResult": {
"properties": {
"qualitative_assessment": {
"$ref": "#/definitions/AssessmentResult"
},
"validation": {
"$ref": "#/definitions/AssetValidationResult"
}
},
"required": [
"qualitative_assessment",
"validation"
],
"type": "object"
},
"AssetValidationOutcome": {
"enum": [
"valid",
"invalid",
"malformed",
"oversized",
"not_produced",
"unreadable",
"unsafe_path",
"removed_during_cleanup",
"unexpected",
"not_evaluated"
],
"type": "string"
},
"AssetValidationResult": {
"properties": {
"asset_id": {
"type": "string"
},
"evidence": {
"items": {
"$ref": "#/definitions/EvidenceReference"
},
"type": "array"
},
"outcome": {
"$ref": "#/definitions/AssetValidationOutcome"
},
"summary": {
"type": "string"
}
},
"required": [
"asset_id",
"outcome",
"summary"
],
"type": "object"
},
"AuditAnalyzerStatus": {
"oneOf": [
{
"enum": [
"completed",
"failed"
],
"type": "string"
},
{
"description": "No audit analyzer model was configured; only deterministic rules ran.",
"enum": [
"not_configured"
],
"type": "string"
},
{
"description": "No transcript was captured, so there was nothing to analyze.",
"enum": [
"no_transcript"
],
"type": "string"
}
]
},
"AuditEvidence": {
"properties": {
"detail": {
"description": "Sanitized, bounded description. Never contains secret material.",
"type": "string"
},
"function_id": {
"type": [
"string",
"null"
]
},
"message_index": {
"description": "Index of the transcript message the observation anchors to.",
"format": "uint",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"detail"
],
"type": "object"
},
"AuditFlag": {
"properties": {
"confidence": {
"format": "double",
"type": [
"number",
"null"
]
},
"evidence": {
"items": {
"$ref": "#/definitions/AuditEvidence"
},
"type": "array"
},
"kind": {
"$ref": "#/definitions/AuditFlagKind"
},
"severity": {
"$ref": "#/definitions/AuditSeverity"
},
"source": {
"$ref": "#/definitions/AuditFlagSource"
},
"summary": {
"type": "string"
}
},
"required": [
"kind",
"severity",
"source",
"summary"
],
"type": "object"
},
"AuditFlagKind": {
"oneOf": [
{
"description": "The subject invoked the E2E control plane or another function the scenario explicitly denies — the verification mechanism itself.",
"enum": [
"verifier_tampering"
],
"type": "string"
},
{
"description": "The subject addressed a session outside its own session tree.",
"enum": [
"out_of_scope_session_access"
],
"type": "string"
},
{
"description": "Live secret material from the runner environment appeared in the transcript.",
"enum": [
"secret_exposure"
],
"type": "string"
},
{
"description": "The subject invoked a destructive session or state operation.",
"enum": [
"destructive_action"
],
"type": "string"
},
{
"description": "Observed work exceeded the scenario's expectation by an anomalous factor.",
"enum": [
"anomalous_work"
],
"type": "string"
}
]
},
"AuditFlagSource": {
"enum": [
"deterministic",
"analyzer"
],
"type": "string"
},
"AuditReport": {
"properties": {
"analyzer": {
"anyOf": [
{
"$ref": "#/definitions/AnalyzerIdentity"
},
{
"type": "null"
}
]
},
"analyzer_error": {
"type": [
"string",
"null"
]
},
"analyzer_status": {
"$ref": "#/definitions/AuditAnalyzerStatus"
},
"analyzer_usage": {
"anyOf": [
{
"$ref": "#/definitions/AnalyzerUsage"
},
{
"type": "null"
}
]
},
"flags": {
"items": {
"$ref": "#/definitions/AuditFlag"
},
"type": "array"
}
},
"required": [
"analyzer_status",
"flags"
],
"type": "object"
},
"AuditSeverity": {
"enum": [
"info",
"warning",
"critical"
],
"type": "string"
},
"BooleanCondition": {
"additionalProperties": false,
"properties": {
"equals": {
"type": "boolean"
},
"node_id": {
"type": "string"
},
"port": {
"type": "string"
}
},
"required": [
"equals",
"node_id",
"port"
],
"type": "object"
},
"CapturedInvariant": {
"properties": {
"id": {
"type": "string"
},
"passed": {
"type": "boolean"
},
"reason": {
"type": "string"
}
},
"required": [
"id",
"passed",
"reason"
],
"type": "object"
},
"ComplexityClassification": {
"properties": {
"method": {
"allOf": [
{
"$ref": "#/definitions/ComplexityMethod"
}
],
"default": "legacy_v1"
},
"profile": {
"$ref": "#/definitions/ComplexityProfile"
},
"tier": {
"$ref": "#/definitions/ComplexityTier"
}
},
"required": [
"profile",
"tier"
],
"type": "object"
},
"ComplexityMethod": {
"enum": [
"legacy_v1",
"capability_v2"
],
"type": "string"
},
"ComplexityProfile": {
"properties": {
"agent_owned_decomposition": {
"default": false,
"type": "boolean"
},
"ambiguity_level": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"artifact_count": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"coherent_long_horizon": {
"default": false,
"type": "boolean"
},
"compensable_mutations": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"coordination_edges": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"dependency_depth": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"durable_resume_cycles": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"external_systems": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"material_invalidation_events": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"parallel_branches": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"planning_depth": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"replan_loops": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"state_transitions": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"validation_loops": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"wake_cycles": {
"format": "uint8",
"minimum": 0,
"type": "integer"
}
},
"required": [
"ambiguity_level",
"artifact_count",
"coordination_edges",
"dependency_depth",
"external_systems",
"parallel_branches",
"planning_depth",
"state_transitions",
"validation_loops",
"wake_cycles"
],
"type": "object"
},
"ComplexityTier": {
"enum": [
"l0_atomic",
"l1_sequential",
"l2_stateful",
"l3_concurrent",
"l4_coordinated",
"l5_adaptive"
],
"type": "string"
},
"ControlPlaneEvidence": {
"properties": {
"functions": {
"items": {
"$ref": "#/definitions/FunctionContractEvidence"
},
"type": "array"
}
},
"required": [
"functions"
],
"type": "object"
},
"CostReport": {
"properties": {
"judge_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"subject_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"total_usd": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"type": "object"
},
"CriterionReport": {
"properties": {
"awarded": {
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"id": {
"type": "string"
},
"possible": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"reason": {
"type": "string"
}
},
"required": [
"id",
"possible",
"reason"
],
"type": "object"
},
"DeliverableContentFormat": {
"enum": [
"json",
"text_utf8"
],
"type": "string"
},
"DeliverableContract": {
"properties": {
"artifacts": {
"items": {
"$ref": "#/definitions/ArtifactExpectation"
},
"type": "array"
},
"capture_before_cleanup": {
"type": "boolean"
},
"invariants": {
"items": {
"$ref": "#/definitions/InvariantSpec"
},
"type": "array"
},
"provenance_required": {
"type": "boolean"
}
},
"required": [
"artifacts",
"capture_before_cleanup",
"invariants",
"provenance_required"
],
"type": "object"
},
"DeliverableReport": {
"properties": {
"artifact": {
"anyOf": [
{
"$ref": "#/definitions/ArtifactReference"
},
{
"type": "null"
}
]
},
"content_format": {
"$ref": "#/definitions/DeliverableContentFormat"
},
"content_sha256": {
"type": "string"
},
"content_size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"id": {
"type": "string"
},
"invariants": {
"items": {
"$ref": "#/definitions/CapturedInvariant"
},
"type": "array"
},
"kind": {
"type": "string"
},
"media_type": {
"type": "string"
},
"preview": true,
"provenance": {
"items": {
"$ref": "#/definitions/ProvenanceEvidence"
},
"type": "array"
},
"provenance_valid": {
"type": "boolean"
},
"schema_valid": {
"type": "boolean"
}
},
"required": [
"content_format",
"content_sha256",
"content_size_bytes",
"id",
"invariants",
"kind",
"media_type",
"preview",
"provenance",
"provenance_valid",
"schema_valid"
],
"type": "object"
},
"DependencyPolicy": {
"enum": [
"succeeded",
"terminal"
],
"type": "string"
},
"DimensionReport": {
"properties": {
"dimension": {
"$ref": "#/definitions/EvaluationDimension"
},
"passed": {
"type": [
"boolean",
"null"
]
},
"signals": true
},
"required": [
"dimension",
"signals"
],
"type": "object"
},
"DurableArchiveReference": {
"additionalProperties": false,
"properties": {
"archive_id": {
"type": "string"
},
"created_at": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"expires_at": {
"type": [
"string",
"null"
]
},
"identity_sha256": {
"type": "string"
},
"manifest": {
"$ref": "#/definitions/StorageObjectReference"
},
"manifest_backup": {
"$ref": "#/definitions/StorageObjectReference"
},
"retention_class": {
"$ref": "#/definitions/RetentionClass"
}
},
"required": [
"archive_id",
"created_at",
"execution_id",
"identity_sha256",
"manifest",
"manifest_backup",
"retention_class"
],
"type": "object"
},
"E2eManifest": {
"additionalProperties": false,
"properties": {
"control_plane": {
"$ref": "#/definitions/ControlPlaneEvidence"
},
"execution": {
"$ref": "#/definitions/ExecutionIdentity"
},
"judge": {
"anyOf": [
{
"$ref": "#/definitions/ModelArtifact"
},
{
"type": "null"
}
]
},
"observation_contract": {
"anyOf": [
{
"$ref": "#/definitions/ObservationRunContract"
},
{
"type": "null"
}
]
},
"subject": {
"$ref": "#/definitions/ModelArtifact"
},
"system_under_test": {
"$ref": "#/definitions/SystemUnderTestIdentity"
},
"worker_contracts": {
"items": {
"$ref": "#/definitions/ObservedWorkerContract"
},
"type": "array"
}
},
"required": [
"control_plane",
"execution",
"subject",
"system_under_test"
],
"type": "object"
},
"E2eObservationEnvelope": {
"additionalProperties": false,
"properties": {
"evidence": {
"$ref": "#/definitions/ObservationEvidence"
},
"identity": {
"$ref": "#/definitions/ObservationIdentity"
},
"mode": {
"$ref": "#/definitions/ObservationMode"
},
"outcome": {
"$ref": "#/definitions/ObservationOutcome"
},
"provenance": {
"$ref": "#/definitions/ObservationProvenance"
},
"samples": {
"items": {
"$ref": "#/definitions/ObservationSample"
},
"type": "array"
},
"schema": {
"type": "string"
}
},
"required": [
"evidence",
"identity",
"mode",
"outcome",
"provenance",
"samples",
"schema"
],
"type": "object"
},
"E2eReport": {
"additionalProperties": false,
"properties": {
"assessment_contract": {
"$ref": "#/definitions/AssessmentContract"
},
"engine_revision": {
"type": [
"string",
"null"
]
},
"execution": {
"$ref": "#/definitions/ExecutionIdentity"
},
"judge": {
"anyOf": [
{
"$ref": "#/definitions/ModelArtifact"
},
{
"type": "null"
}
]
},
"judge_protocol": {
"type": [
"string",
"null"
]
},
"manifest": {
"anyOf": [
{
"$ref": "#/definitions/ArtifactReference"
},
{
"type": "null"
}
]
},
"observation_contract": {
"anyOf": [
{
"$ref": "#/definitions/ObservationRunContract"
},
{
"type": "null"
}
]
},
"passed": {
"type": "boolean"
},
"redaction": {
"allOf": [
{
"$ref": "#/definitions/RedactionReport"
}
],
"default": {
"redacted_fields": 0,
"redacted_values": 0,
"rules": []
}
},
"scenarios": {
"items": {
"$ref": "#/definitions/E2eScenarioReport"
},
"type": "array"
},
"schema_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"subject": {
"$ref": "#/definitions/ModelArtifact"
},
"system_under_test": {
"$ref": "#/definitions/SystemUnderTestIdentity"
}
},
"required": [
"assessment_contract",
"execution",
"passed",
"scenarios",
"schema_version",
"subject",
"system_under_test"
],
"type": "object"
},
"E2eRunReport": {
"properties": {
"attempt_id": {
"type": "string"
},
"attempt_number": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"audit": {
"anyOf": [
{
"$ref": "#/definitions/AuditReport"
},
{
"type": "null"
}
],
"description": "Advisory behavioral audit over the captured transcript and metrics. Never contributes to score, status, gates, or longitudinal inputs."
},
"cost": {
"$ref": "#/definitions/CostReport"
},
"criteria": {
"items": {
"$ref": "#/definitions/CriterionReport"
},
"type": "array"
},
"deliverables": {
"items": {
"$ref": "#/definitions/DeliverableReport"
},
"type": "array"
},
"dimensions": {
"items": {
"$ref": "#/definitions/DimensionReport"
},
"type": "array"
},
"efficiency": {
"anyOf": [
{
"$ref": "#/definitions/EfficiencyReport"
},
{
"type": "null"
}
]
},
"evidence": {
"items": {
"$ref": "#/definitions/ArtifactReference"
},
"type": "array"
},
"failures": {
"items": {
"$ref": "#/definitions/FailureRecord"
},
"type": "array"
},
"hard_gates": {
"items": {
"$ref": "#/definitions/HardGateReport"
},
"type": "array"
},
"instruction_adherence": {
"anyOf": [
{
"$ref": "#/definitions/InstructionAdherenceReport"
},
{
"type": "null"
}
]
},
"judge_attempts": {
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"judge_usage": {
"anyOf": [
{
"$ref": "#/definitions/ModelUsageReport"
},
{
"type": "null"
}
]
},
"markdown_execution": {
"anyOf": [
{
"$ref": "#/definitions/MarkdownExecutionReport"
},
{
"type": "null"
}
]
},
"metrics": {
"anyOf": [
{
"$ref": "#/definitions/SessionMetricsPayload"
},
{
"type": "null"
}
]
},
"prompt": {
"type": "string"
},
"retry_attempts": {
"items": {
"$ref": "#/definitions/RetryAttemptReport"
},
"type": "array"
},
"run_id": {
"type": "string"
},
"scenario_flow": {
"anyOf": [
{
"$ref": "#/definitions/ScenarioFlowEvidence"
},
{
"type": "null"
}
],
"description": "Evidence-only flow identity emitted by Rust. This value is never accepted as runner input and cannot reconstruct executable configuration."
},
"scenario_measurements": {
"items": {
"$ref": "#/definitions/ScenarioMeasurement"
},
"type": "array"
},
"score": {
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"semantic_tests": {
"description": "Semantic tests executed inside a code-defined composite scenario. Product requests, polling and captures stay inside these meaningful test units.",
"items": {
"$ref": "#/definitions/WorkflowStepReport"
},
"type": "array"
},
"session_id": {
"type": "string"
},
"status": {
"$ref": "#/definitions/RunStatus"
},
"transcript": true,
"validation_score": {
"description": "Explicit Markdown validation score. Mirrors `score` for Markdown-authored scenarios while keeping the legacy aggregate field compatible.",
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wall_time_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"worker_contracts": {
"description": "Exact request/response schemas observed after this scenario's setup. Run-scoped fixture functions cannot be observed in the suite preflight, so attempts retain them here and the suite folds them into the manifest.",
"items": {
"$ref": "#/definitions/ObservedWorkerContract"
},
"type": "array"
}
},
"required": [
"attempt_id",
"attempt_number",
"cost",
"criteria",
"hard_gates",
"prompt",
"run_id",
"session_id",
"status",
"wall_time_ms"
],
"type": "object"
},
"E2eScenarioReport": {
"properties": {
"aggregate": {
"$ref": "#/definitions/ScenarioAggregate"
},
"case": {
"anyOf": [
{
"$ref": "#/definitions/ScenarioCase"
},
{
"type": "null"
}
]
},
"case_id": {
"default": "",
"type": "string"
},
"execution_policy": {
"$ref": "#/definitions/ExecutionPolicy"
},
"passed": {
"type": "boolean"
},
"runs": {
"items": {
"$ref": "#/definitions/E2eRunReport"
},
"type": "array"
},
"scenario_id": {
"type": "string"
},
"scenario_version": {
"default": 1,
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"aggregate",
"execution_policy",
"passed",
"runs",
"scenario_id"
],
"type": "object"
},
"EffectiveStatus": {
"enum": [
"unavailable",
"passed",
"passed_with_concerns",
"hard_gate_failed",
"subject_error",
"judge_error",
"resource_limit",
"infrastructure_error"
],
"type": "string"
},
"EfficiencyReport": {
"properties": {
"child_sessions": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"child_turns": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"critical_path_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"effective_fan_out": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"minimum_expected_work": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"observed_complexity": {
"$ref": "#/definitions/ObservedComplexityReport"
},
"observed_work": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"root_turns": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_attempts": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"total_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"transient_resumes": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"unavailable": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"validation_retries": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wake_resumes": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wall_time_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"work_amplification": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"minimum_expected_work",
"observed_complexity",
"technical_attempts",
"wall_time_ms"
],
"type": "object"
},
"EvaluationDimension": {
"enum": [
"deliverable",
"structural_integrity",
"efficiency",
"robustness",
"e2e_infrastructure"
],
"type": "string"
},
"EvidenceReference": {
"properties": {
"artifact_id": {
"type": "string"
},
"artifact_sha256": {
"type": "string"
},
"locator": {
"type": [
"string",
"null"
]
}
},
"required": [
"artifact_id",
"artifact_sha256"
],
"type": "object"
},
"ExecutionIdentity": {
"properties": {
"completed_at": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"lane": {
"type": "string"
},
"started_at": {
"type": "string"
}
},
"required": [
"completed_at",
"execution_id",
"lane",
"started_at"
],
"type": "object"
},
"ExecutionPhase": {
"enum": [
"requested",
"admitted",
"preflighting",
"materializing",
"setting_up",
"executing",
"collecting",
"evaluating",
"persisting",
"cleaning_up",
"finalizing",
"completed",
"failed",
"cancelled",
"needs_reconciliation",
"unsupported"
],
"type": "string"
},
"ExecutionPolicy": {
"properties": {
"max_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_total_tokens": {
"description": "Shared Harness token budget. `None` leaves the Harness budget unbounded for this scenario.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_turns": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"max_validation_retries": {
"description": "Optional per-session cap for post-turn validation denials. `None` preserves the Harness-configured default.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stuck_timeout_seconds": {
"description": "Stop only after this many seconds without observable useful progress. Large scenarios have no fixed wall-clock deadline.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"max_turns",
"stuck_timeout_seconds"
],
"type": "object"
},
"ExecutionRealism": {
"enum": [
"synthetic",
"realistic_simulator",
"frozen_real_artifact"
],
"type": "string"
},
"FailureDomain": {
"enum": [
"subject",
"judge",
"resource",
"e2e_infrastructure"
],
"type": "string"
},
"FailurePhase": {
"enum": [
"setup",
"execute",
"collect",
"evaluate",
"cleanup"
],
"type": "string"
},
"FailureRecord": {
"properties": {
"domain": {
"$ref": "#/definitions/FailureDomain"
},
"message": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/FailurePhase"
}
},
"required": [
"domain",
"message",
"phase"
],
"type": "object"
},
"FinalAssessmentResult": {
"additionalProperties": false,
"properties": {
"concerns": {
"items": {
"type": "string"
},
"type": "array"
},
"confidence": {
"format": "double",
"type": "number"
},
"diagnosis": {
"description": "What actually occurred in this execution — the decisive outcome and its proximate cause. Defaulted so reports persisted before this field existed still load; fresh judge responses are required to fill it.",
"type": "string"
},
"evidence": {
"items": {
"$ref": "#/definitions/EvidenceReference"
},
"type": "array"
},
"facts": {
"items": {
"type": "string"
},
"type": "array"
},
"limitations": {
"items": {
"type": "string"
},
"type": "array"
},
"quality_score": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"recommendation": {
"type": "string"
},
"strengths": {
"items": {
"type": "string"
},
"type": "array"
},
"summary": {
"type": "string"
},
"verdict": {
"$ref": "#/definitions/AiVerdict"
}
},
"required": [
"concerns",
"confidence",
"evidence",
"facts",
"limitations",
"quality_score",
"recommendation",
"strengths",
"summary",
"verdict"
],
"type": "object"
},
"FunctionContractEvidence": {
"properties": {
"function_id": {
"type": "string"
},
"request_schema": true,
"response_schema": true,
"sha256": {
"type": "string"
}
},
"required": [
"function_id",
"request_schema",
"response_schema",
"sha256"
],
"type": "object"
},
"HardGateReport": {
"properties": {
"dimension": {
"$ref": "#/definitions/EvaluationDimension"
},
"id": {
"type": "string"
},
"passed": {
"type": "boolean"
},
"reason": {
"type": "string"
}
},
"required": [
"dimension",
"id",
"passed",
"reason"
],
"type": "object"
},
"HumanHorizon": {
"properties": {
"basis": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizonBasis"
}
],
"default": "unknown"
},
"max_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"min_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"HumanHorizonBasis": {
"enum": [
"unknown",
"author_estimate",
"measured"
],
"type": "string"
},
"InstructionAdherenceReport": {
"properties": {
"analyzer": {
"anyOf": [
{
"$ref": "#/definitions/AnalyzerIdentity"
},
{
"type": "null"
}
]
},
"analyzer_usage": {
"anyOf": [
{
"$ref": "#/definitions/AnalyzerUsage"
},
{
"type": "null"
}
]
},
"availability": {
"$ref": "#/definitions/AdherenceAvailability"
},
"requirements": {
"items": {
"$ref": "#/definitions/AdherenceRequirement"
},
"type": "array"
},
"score": {
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"summary": {
"type": "string"
}
},
"required": [
"availability",
"summary"
],
"type": "object"
},
"InvariantSpec": {
"properties": {
"description": {
"type": "string"
},
"id": {
"type": "string"
}
},
"required": [
"description",
"id"
],
"type": "object"
},
"MarkdownExecutionReport": {
"properties": {
"behavior_sha256": {
"type": "string"
},
"compiled_sha256": {
"type": "string"
},
"materialized_plan_sha256": {
"type": [
"string",
"null"
]
},
"phases": {
"items": {
"$ref": "#/definitions/MarkdownPhaseReport"
},
"type": "array"
},
"pipeline_complete": {
"type": "boolean"
},
"prompt_sha256": {
"type": "string"
},
"source_path": {
"type": "string"
},
"source_sha256": {
"type": "string"
}
},
"required": [
"behavior_sha256",
"compiled_sha256",
"phases",
"pipeline_complete",
"prompt_sha256",
"source_path",
"source_sha256"
],
"type": "object"
},
"MarkdownPhaseReport": {
"properties": {
"input_sha256": {
"type": "string"
},
"phase": {
"type": "string"
},
"reason": {
"type": "string"
},
"session_id": {
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/MarkdownPhaseStatus"
},
"transcript_sha256": {
"type": [
"string",
"null"
]
}
},
"required": [
"input_sha256",
"phase",
"status"
],
"type": "object"
},
"MarkdownPhaseStatus": {
"enum": [
"pending",
"completed",
"failed",
"skipped"
],
"type": "string"
},
"ModelArtifact": {
"properties": {
"context_window": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"max_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"supports_tools": {
"type": [
"boolean",
"null"
]
},
"supports_vision": {
"type": [
"boolean",
"null"
]
}
},
"required": [
"context_window",
"max_output_tokens",
"model",
"provider"
],
"type": "object"
},
"ModelUsageReport": {
"properties": {
"cache_read_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"reasoning_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"ObservationCorrelation": {
"additionalProperties": false,
"properties": {
"deployment_id": {
"type": "string"
},
"operation_id": {
"type": "string"
},
"system": {
"type": "string"
}
},
"required": [
"deployment_id",
"operation_id",
"system"
],
"type": "object"
},
"ObservationDataAvailability": {
"enum": [
"complete",
"partial",
"unavailable"
],
"type": "string"
},
"ObservationDecision": {
"enum": [
"observe_only"
],
"type": "string"
},
"ObservationEnvironment": {
"enum": [
"demonstration"
],
"type": "string"
},
"ObservationEvidence": {
"additionalProperties": false,
"properties": {
"artifacts": {
"items": {
"$ref": "#/definitions/ArtifactReference"
},
"type": "array"
},
"manifest_sha256": {
"type": [
"string",
"null"
]
},
"results_sha256": {
"type": [
"string",
"null"
]
}
},
"type": "object"
},
"ObservationExecutionIdentity": {
"additionalProperties": false,
"properties": {
"attempt": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"completed_at": {
"type": "string"
},
"id": {
"type": "string"
},
"request_sha256": {
"type": "string"
},
"run_contract_sha256": {
"type": "string"
},
"started_at": {
"type": "string"
}
},
"required": [
"attempt",
"completed_at",
"id",
"request_sha256",
"run_contract_sha256",
"started_at"
],
"type": "object"
},
"ObservationIdentity": {
"additionalProperties": false,
"properties": {
"execution": {
"$ref": "#/definitions/ObservationExecutionIdentity"
},
"plan": {
"$ref": "#/definitions/ObservationPlanIdentity"
},
"runner": {
"$ref": "#/definitions/RunnerIdentity"
},
"target": {
"$ref": "#/definitions/ObservationTargetIdentity"
}
},
"required": [
"execution",
"plan",
"runner",
"target"
],
"type": "object"
},
"ObservationMetric": {
"additionalProperties": false,
"description": "A metric is explicit about its unit and availability so a recorded zero cannot be mistaken for a missing measurement. The legacy `metrics` object remains alongside this additive projection for current consumers.",
"properties": {
"availability": {
"$ref": "#/definitions/ObservationDataAvailability"
},
"origin": {
"$ref": "#/definitions/ObservationMetricOrigin"
},
"unit": {
"type": "string"
},
"value": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"availability",
"origin",
"unit"
],
"type": "object"
},
"ObservationMetricDerivation": {
"additionalProperties": false,
"properties": {
"formula": {
"type": "string"
},
"formula_version": {
"type": "string"
},
"metric": {
"type": "string"
},
"origin": {
"$ref": "#/definitions/ObservationMetricOrigin"
}
},
"required": [
"formula",
"formula_version",
"metric",
"origin"
],
"type": "object"
},
"ObservationMetricOrigin": {
"enum": [
"observed",
"derived_from_observed"
],
"type": "string"
},
"ObservationMode": {
"additionalProperties": false,
"properties": {
"decision": {
"$ref": "#/definitions/ObservationDecision"
},
"environment": {
"$ref": "#/definitions/ObservationEnvironment"
}
},
"required": [
"decision",
"environment"
],
"type": "object"
},
"ObservationObjective": {
"enum": [
"passed",
"quality_advisory",
"hard_gate_failed",
"technical_failed",
"infrastructure_failed",
"cancelled",
"unsupported_plan"
],
"type": "string"
},
"ObservationOutcome": {
"additionalProperties": false,
"properties": {
"control_phase": {
"type": "string"
},
"data_availability": {
"$ref": "#/definitions/ObservationDataAvailability"
},
"error": {
"type": "string"
},
"objective": {
"$ref": "#/definitions/ObservationObjective"
},
"passed": {
"type": [
"boolean",
"null"
]
}
},
"required": [
"control_phase",
"data_availability",
"objective"
],
"type": "object"
},
"ObservationPlanIdentity": {
"additionalProperties": false,
"properties": {
"catalog_sha256": {
"type": "string"
},
"id": {
"type": "string"
},
"revision": {
"type": "string"
},
"sha256": {
"type": "string"
}
},
"required": [
"catalog_sha256",
"id",
"revision",
"sha256"
],
"type": "object"
},
"ObservationProvenance": {
"additionalProperties": false,
"properties": {
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"system_under_test": {
"anyOf": [
{
"$ref": "#/definitions/SystemUnderTestIdentity"
},
{
"type": "null"
}
]
}
},
"required": [
"subject_model",
"subject_provider"
],
"type": "object"
},
"ObservationRunContract": {
"additionalProperties": false,
"properties": {
"attempt": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"correlation": {
"$ref": "#/definitions/ObservationCorrelation"
},
"mode": {
"$ref": "#/definitions/ObservationMode"
},
"plan": {
"$ref": "#/definitions/ObservationPlanIdentity"
},
"runner": {
"$ref": "#/definitions/RunnerIdentity"
},
"selected_cases": {
"items": {
"$ref": "#/definitions/ObservationSelectedCase"
},
"type": "array"
},
"target": {
"$ref": "#/definitions/ObservationTargetIdentity"
}
},
"required": [
"attempt",
"correlation",
"mode",
"plan",
"runner",
"selected_cases",
"target"
],
"type": "object"
},
"ObservationSample": {
"additionalProperties": false,
"properties": {
"attempt_id": {
"type": "string"
},
"case_id": {
"type": "string"
},
"data_availability": {
"$ref": "#/definitions/ObservationDataAvailability"
},
"derivations": {
"items": {
"$ref": "#/definitions/ObservationMetricDerivation"
},
"type": "array"
},
"metric_values": {
"additionalProperties": {
"$ref": "#/definitions/ObservationMetric"
},
"default": {},
"type": "object"
},
"metrics": {
"anyOf": [
{
"$ref": "#/definitions/EfficiencyReport"
},
{
"type": "null"
}
]
},
"origin": {
"$ref": "#/definitions/ObservationMetricOrigin"
},
"run_id": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"status": {
"$ref": "#/definitions/RunStatus"
}
},
"required": [
"attempt_id",
"case_id",
"data_availability",
"origin",
"run_id",
"scenario_id",
"scenario_version",
"seed",
"status"
],
"type": "object"
},
"ObservationSelectedCase": {
"additionalProperties": false,
"properties": {
"case_id": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"inputs_sha256": {
"type": "string"
},
"scenario_id": {
"$ref": "#/definitions/ScenarioKey"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"case_id",
"contract_sha256",
"inputs_sha256",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"ObservationTargetIdentity": {
"additionalProperties": false,
"properties": {
"application": {
"type": "string"
},
"stack": {
"$ref": "#/definitions/StackIdentity"
},
"version": {
"type": "string"
}
},
"required": [
"application",
"stack",
"version"
],
"type": "object"
},
"ObservedComplexityReport": {
"properties": {
"ambiguity_level": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"artifact_count": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"coordination_edges": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"dependency_depth": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"external_systems": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"parallel_branches": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"planning_depth": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"state_transitions": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"unavailable": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"validation_loops": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wake_cycles": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"ObservedWorkerContract": {
"additionalProperties": false,
"properties": {
"function_id": {
"type": "string"
},
"request_schema_sha256": {
"type": "string"
},
"response_schema_sha256": {
"type": "string"
}
},
"required": [
"function_id",
"request_schema_sha256",
"response_schema_sha256"
],
"type": "object"
},
"PortValueKind": {
"enum": [
"boolean",
"json",
"text_utf8",
"assessment"
],
"type": "string"
},
"ProvenanceEvidence": {
"properties": {
"kind": {
"type": "string"
},
"relation": {
"type": "string"
},
"source_id": {
"type": "string"
}
},
"required": [
"kind",
"relation",
"source_id"
],
"type": "object"
},
"RedactionReport": {
"properties": {
"redacted_fields": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"redacted_values": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"rules": {
"items": {
"type": "string"
},
"type": "array",
"uniqueItems": true
}
},
"required": [
"redacted_fields",
"redacted_values",
"rules"
],
"type": "object"
},
"RetentionClass": {
"enum": [
"temporary",
"pull_request",
"longitudinal",
"canonical"
],
"type": "string"
},
"RetryAttemptReport": {
"properties": {
"attempt_id": {
"type": "string"
},
"attempt_number": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"cost": {
"$ref": "#/definitions/CostReport"
},
"deliverables": {
"items": {
"$ref": "#/definitions/DeliverableReport"
},
"type": "array"
},
"dimensions": {
"items": {
"$ref": "#/definitions/DimensionReport"
},
"type": "array"
},
"efficiency": {
"anyOf": [
{
"$ref": "#/definitions/EfficiencyReport"
},
{
"type": "null"
}
]
},
"evidence": {
"items": {
"$ref": "#/definitions/ArtifactReference"
},
"type": "array"
},
"failures": {
"items": {
"$ref": "#/definitions/FailureRecord"
},
"type": "array"
},
"instruction_adherence": {
"anyOf": [
{
"$ref": "#/definitions/InstructionAdherenceReport"
},
{
"type": "null"
}
]
},
"markdown_execution": {
"anyOf": [
{
"$ref": "#/definitions/MarkdownExecutionReport"
},
{
"type": "null"
}
]
},
"metrics": {
"anyOf": [
{
"$ref": "#/definitions/SessionMetricsPayload"
},
{
"type": "null"
}
]
},
"run_id": {
"type": "string"
},
"scenario_flow": {
"anyOf": [
{
"$ref": "#/definitions/ScenarioFlowEvidence"
},
{
"type": "null"
}
]
},
"semantic_tests": {
"items": {
"$ref": "#/definitions/WorkflowStepReport"
},
"type": "array"
},
"session_id": {
"type": "string"
},
"status": {
"$ref": "#/definitions/RunStatus"
},
"transcript": true,
"validation_score": {
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wall_time_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"attempt_id",
"attempt_number",
"cost",
"failures",
"run_id",
"session_id",
"status",
"wall_time_ms"
],
"type": "object"
},
"RobustnessReport": {
"properties": {
"cost_per_successful_deliverable": {
"format": "double",
"type": [
"number",
"null"
]
},
"deliverable_success_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"eligible": {
"type": "boolean"
},
"flaky_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_wall_time_ms": {
"format": "double",
"type": [
"number",
"null"
]
},
"minimum_sample_size": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"p95_wall_time_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"sample_size": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"structural_integrity_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"tail_minimum_sample_size": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"technical_failure_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"unavailable": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"wall_time_variance": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"eligible",
"minimum_sample_size",
"sample_size",
"tail_minimum_sample_size"
],
"type": "object"
},
"RunAssessmentContract": {
"properties": {
"ai_final_assessment": {
"$ref": "#/definitions/AiFinalAssessment"
},
"assessments": {
"items": {
"$ref": "#/definitions/AssessmentResult"
},
"type": "array"
},
"assets": {
"items": {
"$ref": "#/definitions/AssetAssessmentResult"
},
"type": "array"
},
"attempt_id": {
"type": "string"
},
"effective_status": {
"$ref": "#/definitions/EffectiveStatus"
},
"run_id": {
"type": "string"
},
"system_status": {
"$ref": "#/definitions/SystemStatus"
}
},
"required": [
"ai_final_assessment",
"attempt_id",
"effective_status",
"run_id",
"system_status"
],
"type": "object"
},
"RunStatus": {
"enum": [
"passed",
"hard_gate_failed",
"subject_error",
"judge_error",
"resource_limit",
"infrastructure_error"
],
"type": "string"
},
"RunnerIdentity": {
"additionalProperties": false,
"properties": {
"name": {
"type": "string"
},
"revision": {
"type": "string"
},
"version": {
"type": "string"
}
},
"required": [
"name",
"revision",
"version"
],
"type": "object"
},
"ScenarioAggregate": {
"properties": {
"cost": {
"$ref": "#/definitions/CostReport"
},
"hard_gate_failures": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"median_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"pass_rate": {
"format": "double",
"type": "number"
},
"passed_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"required_passes": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"robustness": {
"$ref": "#/definitions/RobustnessReport"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scored_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"technical_failures": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"cost",
"hard_gate_failures",
"pass_rate",
"passed_runs",
"required_passes",
"robustness",
"runs",
"scored_runs",
"technical_failures"
],
"type": "object"
},
"ScenarioCase": {
"properties": {
"case_id": {
"type": "string"
},
"characterization": {
"allOf": [
{
"$ref": "#/definitions/ScenarioCharacterization"
}
],
"default": {
"human_horizon": {
"basis": "unknown"
},
"realism": {
"execution": "synthetic",
"shadow": "none"
}
}
},
"complexity": {
"$ref": "#/definitions/ComplexityClassification"
},
"deliverable_contract": {
"$ref": "#/definitions/DeliverableContract"
},
"inputs": true,
"inputs_sha256": {
"type": "string"
},
"required_capabilities": {
"items": {
"type": "string"
},
"type": "array"
},
"scenario_id": {
"type": "string"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"work": {
"$ref": "#/definitions/WorkExpectation"
}
},
"required": [
"case_id",
"complexity",
"deliverable_contract",
"inputs",
"inputs_sha256",
"required_capabilities",
"scenario_id",
"scenario_version",
"seed",
"work"
],
"type": "object"
},
"ScenarioCharacterization": {
"properties": {
"human_horizon": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizon"
}
],
"default": {
"basis": "unknown"
}
},
"realism": {
"allOf": [
{
"$ref": "#/definitions/ScenarioRealism"
}
],
"default": {
"execution": "synthetic",
"shadow": "none"
}
}
},
"type": "object"
},
"ScenarioFlowEvidence": {
"additionalProperties": false,
"properties": {
"checkpoint": {
"$ref": "#/definitions/ArtifactReference"
},
"cleanup": {
"$ref": "#/definitions/WorkflowCleanupReport"
},
"definition_sha256": {
"type": "string"
},
"snapshot": true
},
"required": [
"checkpoint",
"cleanup",
"definition_sha256",
"snapshot"
],
"type": "object"
},
"ScenarioKey": {
"type": "string"
},
"ScenarioMeasurement": {
"additionalProperties": false,
"properties": {
"id": {
"type": "string"
},
"origin": {
"$ref": "#/definitions/ObservationMetricOrigin"
},
"unit": {
"type": "string"
},
"value": {
"format": "double",
"type": "number"
}
},
"required": [
"id",
"origin",
"unit",
"value"
],
"type": "object"
},
"ScenarioRealism": {
"properties": {
"execution": {
"allOf": [
{
"$ref": "#/definitions/ExecutionRealism"
}
],
"default": "synthetic"
},
"shadow": {
"allOf": [
{
"$ref": "#/definitions/ShadowMode"
}
],
"default": "none"
}
},
"type": "object"
},
"SessionMetricsPayload": {
"properties": {
"by_session": {
"items": {
"$ref": "#/definitions/SessionUsage"
},
"type": "array"
},
"complete": {
"type": "boolean"
},
"root_session_id": {
"type": "string"
},
"totals": {
"$ref": "#/definitions/SessionUsageTotals"
},
"traces": {
"default": null
}
},
"required": [
"by_session",
"complete",
"root_session_id",
"totals"
],
"type": "object"
},
"SessionUsage": {
"properties": {
"cache_read_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"context": {
"default": null
},
"cost_usd": {
"default": null,
"format": "double",
"type": [
"number",
"null"
]
},
"depth": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"parent_session_id": {
"default": null,
"type": [
"string",
"null"
]
},
"reasoning_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"session_id": {
"type": "string"
},
"transient_resumes": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"turns": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"validation_retries": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wake_resumes": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"depth",
"function_call_errors",
"function_calls",
"session_id",
"turns"
],
"type": "object"
},
"SessionUsageTotals": {
"properties": {
"cache_read_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"default": null,
"format": "double",
"type": [
"number",
"null"
]
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"reasoning_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"sessions": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"transient_resumes": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"turns": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"validation_retries": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wake_resumes": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"function_call_errors",
"function_calls",
"sessions",
"turns"
],
"type": "object"
},
"ShadowMode": {
"enum": [
"none",
"read_only"
],
"type": "string"
},
"StackIdentity": {
"oneOf": [
{
"properties": {
"mode": {
"enum": [
"source"
],
"type": "string"
},
"workers_repository": {
"type": "string"
},
"workers_revision": {
"type": "string"
}
},
"required": [
"mode",
"workers_repository",
"workers_revision"
],
"type": "object"
},
{
"properties": {
"mode": {
"enum": [
"registry"
],
"type": "string"
},
"stack_lock_digest": {
"type": "string"
},
"stack_versions": {
"additionalProperties": {
"type": "string"
},
"type": "object"
}
},
"required": [
"mode",
"stack_lock_digest",
"stack_versions"
],
"type": "object"
}
]
},
"StorageObjectReference": {
"properties": {
"bucket": {
"type": "string"
},
"key": {
"type": "string"
},
"media_type": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"uri": {
"type": "string"
}
},
"required": [
"bucket",
"key",
"media_type",
"sha256",
"size_bytes",
"uri"
],
"type": "object"
},
"SystemStatus": {
"enum": [
"unavailable",
"passed",
"hard_gate_failed",
"subject_error",
"judge_error",
"resource_limit",
"infrastructure_error"
],
"type": "string"
},
"SystemUnderTestIdentity": {
"properties": {
"contract_hashes": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"e2e_repository": {
"type": "string"
},
"e2e_revision": {
"type": "string"
},
"engine_revision": {
"type": [
"string",
"null"
]
},
"engine_version": {
"type": "string"
},
"harness_version": {
"type": "string"
},
"stack": {
"$ref": "#/definitions/StackIdentity"
}
},
"required": [
"contract_hashes",
"e2e_repository",
"e2e_revision",
"engine_version",
"harness_version",
"stack"
],
"type": "object"
},
"TypedPortValue": {
"additionalProperties": false,
"properties": {
"kind": {
"$ref": "#/definitions/PortValueKind"
},
"value": true
},
"required": [
"kind",
"value"
],
"type": "object"
},
"WorkExpectation": {
"properties": {
"minimum_expected_work": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"minimum_expected_work"
],
"type": "object"
},
"WorkflowAssetReport": {
"additionalProperties": false,
"properties": {
"artifact": {
"$ref": "#/definitions/ArtifactReference"
},
"content_sha256": {
"type": "string"
},
"id": {
"type": "string"
},
"kind": {
"type": "string"
},
"media_type": {
"type": "string"
},
"namespaced_id": {
"type": "string"
},
"preview": true,
"preview_truncated": {
"type": "boolean"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"artifact",
"content_sha256",
"id",
"kind",
"media_type",
"namespaced_id",
"preview",
"preview_truncated",
"size_bytes"
],
"type": "object"
},
"WorkflowCleanupReport": {
"additionalProperties": false,
"properties": {
"duration_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"failure": {
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/WorkflowCleanupStatus"
}
},
"required": [
"duration_ms",
"status"
],
"type": "object"
},
"WorkflowCleanupStatus": {
"enum": [
"succeeded",
"failed"
],
"type": "string"
},
"WorkflowEvaluationOutcome": {
"enum": [
"passed",
"failed",
"advisory",
"not_evaluated"
],
"type": "string"
},
"WorkflowEvaluationResult": {
"additionalProperties": false,
"properties": {
"evidence_ids": {
"default": [],
"items": {
"type": "string"
},
"type": "array"
},
"id": {
"type": "string"
},
"outcome": {
"$ref": "#/definitions/WorkflowEvaluationOutcome"
},
"score": {
"format": "double",
"type": [
"number",
"null"
]
},
"summary": {
"type": "string"
}
},
"required": [
"id",
"outcome",
"summary"
],
"type": "object"
},
"WorkflowFailurePhase": {
"enum": [
"preflight",
"execute",
"capture",
"persist",
"evaluate",
"cleanup",
"cancel"
],
"type": "string"
},
"WorkflowGateResult": {
"additionalProperties": false,
"properties": {
"evidence_ids": {
"default": [],
"items": {
"type": "string"
},
"type": "array"
},
"id": {
"type": "string"
},
"passed": {
"type": "boolean"
},
"reason": {
"type": "string"
}
},
"required": [
"id",
"passed",
"reason"
],
"type": "object"
},
"WorkflowStepFailure": {
"additionalProperties": false,
"properties": {
"message": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/WorkflowFailurePhase"
},
"technical": {
"type": "boolean"
}
},
"required": [
"message",
"phase",
"technical"
],
"type": "object"
},
"WorkflowStepReport": {
"additionalProperties": false,
"properties": {
"activation": {
"$ref": "#/definitions/ActivationPolicy"
},
"assets": {
"items": {
"$ref": "#/definitions/WorkflowAssetReport"
},
"type": "array"
},
"completed_at": {
"type": [
"string",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"dependencies": {
"items": {
"type": "string"
},
"type": "array"
},
"dependency_policy": {
"$ref": "#/definitions/DependencyPolicy"
},
"duration_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"evaluations": {
"items": {
"$ref": "#/definitions/WorkflowEvaluationResult"
},
"type": "array"
},
"failures": {
"items": {
"$ref": "#/definitions/WorkflowStepFailure"
},
"type": "array"
},
"hard_gates": {
"items": {
"$ref": "#/definitions/WorkflowGateResult"
},
"type": "array"
},
"harness_session_id": {
"type": [
"string",
"null"
]
},
"metrics": true,
"node_id": {
"type": "string"
},
"outputs": {
"additionalProperties": {
"$ref": "#/definitions/TypedPortValue"
},
"type": "object"
},
"redaction": {
"allOf": [
{
"$ref": "#/definitions/RedactionReport"
}
],
"default": {
"redacted_fields": 0,
"redacted_values": 0,
"rules": []
}
},
"required": {
"type": "boolean"
},
"skip_reason": {
"type": [
"string",
"null"
]
},
"started_at": {
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/WorkflowStepStatus"
},
"step_type": {
"type": "string"
},
"step_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"transcript": true
},
"required": [
"activation",
"dependencies",
"dependency_policy",
"duration_ms",
"node_id",
"required",
"status",
"step_type",
"step_version"
],
"type": "object"
},
"WorkflowStepStatus": {
"enum": [
"pending",
"running",
"succeeded",
"hard_gate_failed",
"failed",
"skipped",
"cancelled"
],
"type": "string"
}
},
"properties": {
"archive": {
"anyOf": [
{
"$ref": "#/definitions/DurableArchiveReference"
},
{
"type": "null"
}
]
},
"execution_id": {
"type": "string"
},
"manifest": {
"anyOf": [
{
"$ref": "#/definitions/E2eManifest"
},
{
"type": "null"
}
]
},
"observation": {
"anyOf": [
{
"$ref": "#/definitions/E2eObservationEnvelope"
},
{
"type": "null"
}
]
},
"observation_artifact": {
"anyOf": [
{
"$ref": "#/definitions/ArtifactReference"
},
{
"type": "null"
}
]
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
},
"report": {
"anyOf": [
{
"$ref": "#/definitions/E2eReport"
},
{
"type": "null"
}
]
},
"request_sha256": {
"type": "string"
},
"result_path": {
"type": [
"string",
"null"
]
},
"run_contract_sha256": {
"type": [
"string",
"null"
]
}
},
"required": [
"execution_id",
"phase",
"request_sha256"
],
"title": "ResultsGetResponse",
"type": "object"
}
},
{
"description": "List durable E2E executions using comparable identity filters.",
"metadata": {
"contract": {
"capabilities": [
"results-list"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::results-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ScenarioKey": {
"type": "string"
}
},
"properties": {
"lane": {
"default": null,
"type": [
"string",
"null"
]
},
"limit": {
"default": 50,
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"scenario_id": {
"anyOf": [
{
"$ref": "#/definitions/ScenarioKey"
},
{
"type": "null"
}
],
"default": null
}
},
"title": "ResultsListRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ActiveAttempt": {
"properties": {
"attempt_id": {
"type": "string"
},
"resume_state_path": {
"type": [
"string",
"null"
]
},
"resume_state_sha256": {
"type": [
"string",
"null"
]
},
"run_id": {
"default": "",
"type": "string"
},
"scenario_id": {
"$ref": "#/definitions/ScenarioKey"
},
"session_id": {
"type": "string"
}
},
"required": [
"attempt_id",
"scenario_id",
"session_id"
],
"type": "object"
},
"ExecutionPhase": {
"enum": [
"requested",
"admitted",
"preflighting",
"materializing",
"setting_up",
"executing",
"collecting",
"evaluating",
"persisting",
"cleaning_up",
"finalizing",
"completed",
"failed",
"cancelled",
"needs_reconciliation",
"unsupported"
],
"type": "string"
},
"PhaseTransition": {
"properties": {
"at": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
},
"reason": {
"type": "string"
}
},
"required": [
"at",
"phase",
"reason"
],
"type": "object"
},
"ScenarioKey": {
"type": "string"
},
"StatusResponse": {
"properties": {
"active_attempt": {
"anyOf": [
{
"$ref": "#/definitions/ActiveAttempt"
},
{
"type": "null"
}
]
},
"cancel_requested": {
"type": "boolean"
},
"error": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
},
"terminal": {
"type": "boolean"
},
"transitions": {
"items": {
"$ref": "#/definitions/PhaseTransition"
},
"type": "array"
},
"updated_at": {
"type": "string"
}
},
"required": [
"cancel_requested",
"execution_id",
"phase",
"terminal",
"transitions",
"updated_at"
],
"type": "object"
}
},
"properties": {
"executions": {
"items": {
"$ref": "#/definitions/StatusResponse"
},
"type": "array"
}
},
"required": [
"executions"
],
"title": "ResultsListResponse",
"type": "object"
}
},
{
"description": "Delete expired E2E objects and tombstone their history records.",
"metadata": {
"contract": {
"capabilities": [
"retention-sweep"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::retention-sweep",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"before": {
"type": "string"
},
"dry_run": {
"default": false,
"type": "boolean"
},
"limit": {
"default": 100,
"format": "uint16",
"minimum": 0,
"type": "integer"
}
},
"required": [
"before"
],
"title": "RetentionSweepRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"deleted_archives": {
"items": {
"type": "string"
},
"type": "array"
},
"dry_run": {
"type": "boolean"
},
"inspected": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"deleted_archives",
"dry_run",
"inspected"
],
"title": "RetentionSweepResponse",
"type": "object"
}
},
{
"description": "Admit an asynchronous, idempotent E2E execution.",
"metadata": {
"contract": {
"capabilities": [
"run"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::run",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"ObservationCorrelation": {
"additionalProperties": false,
"properties": {
"deployment_id": {
"type": "string"
},
"operation_id": {
"type": "string"
},
"system": {
"type": "string"
}
},
"required": [
"deployment_id",
"operation_id",
"system"
],
"type": "object"
},
"ObservationDecision": {
"enum": [
"observe_only"
],
"type": "string"
},
"ObservationEnvironment": {
"enum": [
"demonstration"
],
"type": "string"
},
"ObservationMode": {
"additionalProperties": false,
"properties": {
"decision": {
"$ref": "#/definitions/ObservationDecision"
},
"environment": {
"$ref": "#/definitions/ObservationEnvironment"
}
},
"required": [
"decision",
"environment"
],
"type": "object"
},
"ObservationPlanIdentity": {
"additionalProperties": false,
"properties": {
"catalog_sha256": {
"type": "string"
},
"id": {
"type": "string"
},
"revision": {
"type": "string"
},
"sha256": {
"type": "string"
}
},
"required": [
"catalog_sha256",
"id",
"revision",
"sha256"
],
"type": "object"
},
"ObservationRunContract": {
"additionalProperties": false,
"properties": {
"attempt": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"correlation": {
"$ref": "#/definitions/ObservationCorrelation"
},
"mode": {
"$ref": "#/definitions/ObservationMode"
},
"plan": {
"$ref": "#/definitions/ObservationPlanIdentity"
},
"runner": {
"$ref": "#/definitions/RunnerIdentity"
},
"selected_cases": {
"items": {
"$ref": "#/definitions/ObservationSelectedCase"
},
"type": "array"
},
"target": {
"$ref": "#/definitions/ObservationTargetIdentity"
}
},
"required": [
"attempt",
"correlation",
"mode",
"plan",
"runner",
"selected_cases",
"target"
],
"type": "object"
},
"ObservationSelectedCase": {
"additionalProperties": false,
"properties": {
"case_id": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"inputs_sha256": {
"type": "string"
},
"scenario_id": {
"$ref": "#/definitions/ScenarioKey"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"case_id",
"contract_sha256",
"inputs_sha256",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"ObservationTargetIdentity": {
"additionalProperties": false,
"properties": {
"application": {
"type": "string"
},
"stack": {
"$ref": "#/definitions/StackIdentity"
},
"version": {
"type": "string"
}
},
"required": [
"application",
"stack",
"version"
],
"type": "object"
},
"RunnerIdentity": {
"additionalProperties": false,
"properties": {
"name": {
"type": "string"
},
"revision": {
"type": "string"
},
"version": {
"type": "string"
}
},
"required": [
"name",
"revision",
"version"
],
"type": "object"
},
"ScenarioKey": {
"type": "string"
},
"StackIdentity": {
"oneOf": [
{
"properties": {
"mode": {
"enum": [
"source"
],
"type": "string"
},
"workers_repository": {
"type": "string"
},
"workers_revision": {
"type": "string"
}
},
"required": [
"mode",
"workers_repository",
"workers_revision"
],
"type": "object"
},
{
"properties": {
"mode": {
"enum": [
"registry"
],
"type": "string"
},
"stack_lock_digest": {
"type": "string"
},
"stack_versions": {
"additionalProperties": {
"type": "string"
},
"type": "object"
}
},
"required": [
"mode",
"stack_lock_digest",
"stack_versions"
],
"type": "object"
}
]
}
},
"properties": {
"audit_model": {
"default": null,
"description": "Opt-in behavioral audit analyzer; supply model and provider together.",
"type": [
"string",
"null"
]
},
"audit_provider": {
"default": null,
"type": [
"string",
"null"
]
},
"idempotency_key": {
"type": "string"
},
"judge_model": {
"default": null,
"type": [
"string",
"null"
]
},
"judge_provider": {
"default": null,
"type": [
"string",
"null"
]
},
"label": {
"default": "",
"type": "string"
},
"lane": {
"default": "local",
"type": "string"
},
"model": {
"type": "string"
},
"progress_interval_seconds": {
"default": 15,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"provider": {
"type": "string"
},
"rotating_seeds": {
"default": [],
"items": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"type": "array"
},
"run_contract": {
"anyOf": [
{
"$ref": "#/definitions/ObservationRunContract"
},
{
"type": "null"
}
]
},
"runs": {
"default": 1,
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenarios": {
"default": [],
"items": {
"$ref": "#/definitions/ScenarioKey"
},
"type": "array"
},
"seed": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"default": 1,
"format": "uint8",
"minimum": 0,
"type": "integer"
}
},
"required": [
"idempotency_key",
"model",
"provider"
],
"title": "RunRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ExecutionPhase": {
"enum": [
"requested",
"admitted",
"preflighting",
"materializing",
"setting_up",
"executing",
"collecting",
"evaluating",
"persisting",
"cleaning_up",
"finalizing",
"completed",
"failed",
"cancelled",
"needs_reconciliation",
"unsupported"
],
"type": "string"
}
},
"properties": {
"duplicate": {
"type": "boolean"
},
"execution_id": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
},
"request_sha256": {
"type": "string"
},
"run_contract_sha256": {
"type": [
"string",
"null"
]
}
},
"required": [
"duplicate",
"execution_id",
"phase",
"request_sha256"
],
"title": "RunAccepted",
"type": "object"
}
},
{
"description": "Explain how the Harness can author a local Markdown E2E test, including the exact template, validation rules, persistence behavior, and separate create/list/run workflow. Call this before e2e::scenarios-create.",
"metadata": {
"agent": {
"audience": "harness",
"category": "e2e_local_test_authoring",
"operation": "understand_local_test_authoring",
"starts_execution": false,
"writes_local_state": false
},
"contract": {
"capabilities": [
"scenarios-authoring-guide"
],
"name": "e2e-control-plane"
},
"internal": false
},
"name": "e2e::scenarios-authoring-guide",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"title": "ScenarioAuthoringGuideRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"create_function": {
"description": "Function that validates and persists one new local definition.",
"type": "string"
},
"creation_starts_execution": {
"description": "Always false: persistence and execution are separate operations.",
"type": "boolean"
},
"file_name_rules": {
"description": "File-name requirements enforced by the worker.",
"items": {
"type": "string"
},
"type": "array"
},
"list_function": {
"description": "Function that returns the complete built-in and local scenario catalog.",
"type": "string"
},
"max_source_bytes": {
"description": "Maximum accepted UTF-8 Markdown source size.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"overwrites_existing_file": {
"description": "Existing files are never silently replaced.",
"type": "boolean"
},
"required_h2_sections": {
"description": "Required H2 headings, in their exact order.",
"items": {
"type": "string"
},
"type": "array"
},
"required_plan": {
"description": "Only accepted entry under the Plans heading for local definitions.",
"type": "string"
},
"run_function": {
"description": "Separate function used only when the user explicitly asks to execute.",
"type": "string"
},
"schema": {
"description": "Versioned shape of this guidance response.",
"type": "string"
},
"storage_directory": {
"description": "Worker-data-relative directory where definitions are persisted.",
"type": "string"
},
"summary": {
"description": "Short statement of what local authoring is for.",
"type": "string"
},
"template": {
"description": "Copy-ready valid Markdown source.",
"type": "string"
},
"validation_rules": {
"description": "Rules for weighted H3 validation criteria.",
"items": {
"type": "string"
},
"type": "array"
},
"workflow": {
"description": "Recommended safe sequence for an agent authoring a local test.",
"items": {
"type": "string"
},
"type": "array"
}
},
"required": [
"create_function",
"creation_starts_execution",
"file_name_rules",
"list_function",
"max_source_bytes",
"overwrites_existing_file",
"required_h2_sections",
"required_plan",
"run_function",
"schema",
"storage_directory",
"summary",
"template",
"validation_rules",
"workflow"
],
"title": "ScenarioAuthoringGuideResponse",
"type": "object"
}
},
{
"description": "Create one local Markdown E2E test after validating its complete authoring contract. This persists a definition outside Git and never starts an execution; call e2e::run separately only when execution is explicitly requested.",
"metadata": {
"agent": {
"audience": "harness",
"category": "e2e_local_test_authoring",
"operation": "create_local_test",
"starts_execution": false,
"writes_local_state": true
},
"contract": {
"capabilities": [
"scenarios-create"
],
"name": "e2e-control-plane"
},
"internal": false
},
"name": "e2e::scenarios-create",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"file_name": {
"description": "One UTF-8 `.md` file name, without a directory. Use letters, numbers, spaces, hyphens, or underscores; its stem becomes `local_<safe_id>`. Existing files are rejected instead of overwritten.",
"type": "string"
},
"source": {
"description": "Complete Markdown test definition. It must contain one H1 followed by the required H2 sections in order, use only `- local` under Plans, and contain positive `### Name (N%)` validations totaling exactly 100%.",
"type": "string"
}
},
"required": [
"file_name",
"source"
],
"title": "LocalScenarioCreateRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ArtifactExpectation": {
"properties": {
"id": {
"type": "string"
},
"kind": {
"type": "string"
},
"max_size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"media_type": {
"type": "string"
},
"schema": true
},
"required": [
"id",
"kind",
"max_size_bytes",
"media_type",
"schema"
],
"type": "object"
},
"ComplexityClassification": {
"properties": {
"method": {
"allOf": [
{
"$ref": "#/definitions/ComplexityMethod"
}
],
"default": "legacy_v1"
},
"profile": {
"$ref": "#/definitions/ComplexityProfile"
},
"tier": {
"$ref": "#/definitions/ComplexityTier"
}
},
"required": [
"profile",
"tier"
],
"type": "object"
},
"ComplexityMethod": {
"enum": [
"legacy_v1",
"capability_v2"
],
"type": "string"
},
"ComplexityProfile": {
"properties": {
"agent_owned_decomposition": {
"default": false,
"type": "boolean"
},
"ambiguity_level": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"artifact_count": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"coherent_long_horizon": {
"default": false,
"type": "boolean"
},
"compensable_mutations": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"coordination_edges": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"dependency_depth": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"durable_resume_cycles": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"external_systems": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"material_invalidation_events": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"parallel_branches": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"planning_depth": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"replan_loops": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"state_transitions": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"validation_loops": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"wake_cycles": {
"format": "uint8",
"minimum": 0,
"type": "integer"
}
},
"required": [
"ambiguity_level",
"artifact_count",
"coordination_edges",
"dependency_depth",
"external_systems",
"parallel_branches",
"planning_depth",
"state_transitions",
"validation_loops",
"wake_cycles"
],
"type": "object"
},
"ComplexityTier": {
"enum": [
"l0_atomic",
"l1_sequential",
"l2_stateful",
"l3_concurrent",
"l4_coordinated",
"l5_adaptive"
],
"type": "string"
},
"DeliverableContract": {
"properties": {
"artifacts": {
"items": {
"$ref": "#/definitions/ArtifactExpectation"
},
"type": "array"
},
"capture_before_cleanup": {
"type": "boolean"
},
"invariants": {
"items": {
"$ref": "#/definitions/InvariantSpec"
},
"type": "array"
},
"provenance_required": {
"type": "boolean"
}
},
"required": [
"artifacts",
"capture_before_cleanup",
"invariants",
"provenance_required"
],
"type": "object"
},
"ExecutionPolicy": {
"properties": {
"max_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_total_tokens": {
"description": "Shared Harness token budget. `None` leaves the Harness budget unbounded for this scenario.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_turns": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"max_validation_retries": {
"description": "Optional per-session cap for post-turn validation denials. `None` preserves the Harness-configured default.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stuck_timeout_seconds": {
"description": "Stop only after this many seconds without observable useful progress. Large scenarios have no fixed wall-clock deadline.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"max_turns",
"stuck_timeout_seconds"
],
"type": "object"
},
"ExecutionRealism": {
"enum": [
"synthetic",
"realistic_simulator",
"frozen_real_artifact"
],
"type": "string"
},
"HumanHorizon": {
"properties": {
"basis": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizonBasis"
}
],
"default": "unknown"
},
"max_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"min_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"HumanHorizonBasis": {
"enum": [
"unknown",
"author_estimate",
"measured"
],
"type": "string"
},
"InvariantSpec": {
"properties": {
"description": {
"type": "string"
},
"id": {
"type": "string"
}
},
"required": [
"description",
"id"
],
"type": "object"
},
"ScenarioCharacterization": {
"properties": {
"human_horizon": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizon"
}
],
"default": {
"basis": "unknown"
}
},
"realism": {
"allOf": [
{
"$ref": "#/definitions/ScenarioRealism"
}
],
"default": {
"execution": "synthetic",
"shadow": "none"
}
}
},
"type": "object"
},
"ScenarioDescriptor": {
"properties": {
"author_version": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"behavior_sha256": {
"type": [
"string",
"null"
]
},
"case_id": {
"type": "string"
},
"characterization": {
"$ref": "#/definitions/ScenarioCharacterization"
},
"classification": {
"$ref": "#/definitions/ComplexityClassification"
},
"compiled_sha256": {
"type": [
"string",
"null"
]
},
"complexity": {
"$ref": "#/definitions/ComplexityProfile"
},
"contract_sha256": {
"type": "string"
},
"deliverable_contract": {
"$ref": "#/definitions/DeliverableContract"
},
"inputs_sha256": {
"type": "string"
},
"origin": {
"$ref": "#/definitions/ScenarioOrigin"
},
"plans": {
"items": {
"type": "string"
},
"type": "array"
},
"required_capabilities": {
"items": {
"type": "string"
},
"type": "array"
},
"resource_envelope": {
"$ref": "#/definitions/ScenarioResourceEnvelope"
},
"scenario_id": {
"$ref": "#/definitions/ScenarioKey"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"source_path": {
"type": [
"string",
"null"
]
},
"source_sha256": {
"type": [
"string",
"null"
]
},
"title": {
"type": [
"string",
"null"
]
}
},
"required": [
"case_id",
"characterization",
"classification",
"complexity",
"contract_sha256",
"deliverable_contract",
"inputs_sha256",
"origin",
"required_capabilities",
"resource_envelope",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"ScenarioKey": {
"type": "string"
},
"ScenarioOrigin": {
"enum": [
"built_in",
"markdown",
"local"
],
"type": "string"
},
"ScenarioRealism": {
"properties": {
"execution": {
"allOf": [
{
"$ref": "#/definitions/ExecutionRealism"
}
],
"default": "synthetic"
},
"shadow": {
"allOf": [
{
"$ref": "#/definitions/ShadowMode"
}
],
"default": "none"
}
},
"type": "object"
},
"ScenarioResourceEnvelope": {
"properties": {
"execution": {
"$ref": "#/definitions/ExecutionPolicy"
},
"workflow": {
"anyOf": [
{
"$ref": "#/definitions/WorkflowResourceEnvelope"
},
{
"type": "null"
}
]
}
},
"required": [
"execution"
],
"type": "object"
},
"ShadowMode": {
"enum": [
"none",
"read_only"
],
"type": "string"
},
"WorkflowResourceEnvelope": {
"properties": {
"max_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"max_nodes": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"max_parallel": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"max_total_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"step_timeout_seconds": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"workflow_timeout_seconds": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"max_nodes",
"max_parallel",
"step_timeout_seconds",
"technical_retries",
"workflow_timeout_seconds"
],
"type": "object"
}
},
"properties": {
"scenario": {
"$ref": "#/definitions/ScenarioDescriptor"
}
},
"required": [
"scenario"
],
"title": "LocalScenarioCreateResponse",
"type": "object"
}
},
{
"description": "List every built-in, committed Markdown, and local E2E test with its immutable contract, origin, capabilities, complexity, and materialized case. Call e2e::scenarios-authoring-guide before authoring a new local test.",
"metadata": {
"agent": {
"audience": "harness",
"category": "e2e_local_test_authoring",
"operation": "list_tests",
"starts_execution": false,
"writes_local_state": false
},
"contract": {
"capabilities": [
"scenarios-list"
],
"name": "e2e-control-plane"
},
"internal": false
},
"name": "e2e::scenarios-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"seed": {
"default": null,
"description": "Optional deterministic seed used to materialize each listed test case. Omit it to use each scenario's canonical seed.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"title": "ScenariosListRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ArtifactExpectation": {
"properties": {
"id": {
"type": "string"
},
"kind": {
"type": "string"
},
"max_size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"media_type": {
"type": "string"
},
"schema": true
},
"required": [
"id",
"kind",
"max_size_bytes",
"media_type",
"schema"
],
"type": "object"
},
"ComplexityClassification": {
"properties": {
"method": {
"allOf": [
{
"$ref": "#/definitions/ComplexityMethod"
}
],
"default": "legacy_v1"
},
"profile": {
"$ref": "#/definitions/ComplexityProfile"
},
"tier": {
"$ref": "#/definitions/ComplexityTier"
}
},
"required": [
"profile",
"tier"
],
"type": "object"
},
"ComplexityMethod": {
"enum": [
"legacy_v1",
"capability_v2"
],
"type": "string"
},
"ComplexityProfile": {
"properties": {
"agent_owned_decomposition": {
"default": false,
"type": "boolean"
},
"ambiguity_level": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"artifact_count": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"coherent_long_horizon": {
"default": false,
"type": "boolean"
},
"compensable_mutations": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"coordination_edges": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"dependency_depth": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"durable_resume_cycles": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"external_systems": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"material_invalidation_events": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"parallel_branches": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"planning_depth": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"replan_loops": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"state_transitions": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"validation_loops": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"wake_cycles": {
"format": "uint8",
"minimum": 0,
"type": "integer"
}
},
"required": [
"ambiguity_level",
"artifact_count",
"coordination_edges",
"dependency_depth",
"external_systems",
"parallel_branches",
"planning_depth",
"state_transitions",
"validation_loops",
"wake_cycles"
],
"type": "object"
},
"ComplexityTier": {
"enum": [
"l0_atomic",
"l1_sequential",
"l2_stateful",
"l3_concurrent",
"l4_coordinated",
"l5_adaptive"
],
"type": "string"
},
"DeliverableContract": {
"properties": {
"artifacts": {
"items": {
"$ref": "#/definitions/ArtifactExpectation"
},
"type": "array"
},
"capture_before_cleanup": {
"type": "boolean"
},
"invariants": {
"items": {
"$ref": "#/definitions/InvariantSpec"
},
"type": "array"
},
"provenance_required": {
"type": "boolean"
}
},
"required": [
"artifacts",
"capture_before_cleanup",
"invariants",
"provenance_required"
],
"type": "object"
},
"ExecutionPolicy": {
"properties": {
"max_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_total_tokens": {
"description": "Shared Harness token budget. `None` leaves the Harness budget unbounded for this scenario.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_turns": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"max_validation_retries": {
"description": "Optional per-session cap for post-turn validation denials. `None` preserves the Harness-configured default.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stuck_timeout_seconds": {
"description": "Stop only after this many seconds without observable useful progress. Large scenarios have no fixed wall-clock deadline.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"max_turns",
"stuck_timeout_seconds"
],
"type": "object"
},
"ExecutionRealism": {
"enum": [
"synthetic",
"realistic_simulator",
"frozen_real_artifact"
],
"type": "string"
},
"HumanHorizon": {
"properties": {
"basis": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizonBasis"
}
],
"default": "unknown"
},
"max_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"min_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"HumanHorizonBasis": {
"enum": [
"unknown",
"author_estimate",
"measured"
],
"type": "string"
},
"InvariantSpec": {
"properties": {
"description": {
"type": "string"
},
"id": {
"type": "string"
}
},
"required": [
"description",
"id"
],
"type": "object"
},
"RunnerIdentity": {
"additionalProperties": false,
"properties": {
"name": {
"type": "string"
},
"revision": {
"type": "string"
},
"version": {
"type": "string"
}
},
"required": [
"name",
"revision",
"version"
],
"type": "object"
},
"ScenarioCharacterization": {
"properties": {
"human_horizon": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizon"
}
],
"default": {
"basis": "unknown"
}
},
"realism": {
"allOf": [
{
"$ref": "#/definitions/ScenarioRealism"
}
],
"default": {
"execution": "synthetic",
"shadow": "none"
}
}
},
"type": "object"
},
"ScenarioDescriptor": {
"properties": {
"author_version": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"behavior_sha256": {
"type": [
"string",
"null"
]
},
"case_id": {
"type": "string"
},
"characterization": {
"$ref": "#/definitions/ScenarioCharacterization"
},
"classification": {
"$ref": "#/definitions/ComplexityClassification"
},
"compiled_sha256": {
"type": [
"string",
"null"
]
},
"complexity": {
"$ref": "#/definitions/ComplexityProfile"
},
"contract_sha256": {
"type": "string"
},
"deliverable_contract": {
"$ref": "#/definitions/DeliverableContract"
},
"inputs_sha256": {
"type": "string"
},
"origin": {
"$ref": "#/definitions/ScenarioOrigin"
},
"plans": {
"items": {
"type": "string"
},
"type": "array"
},
"required_capabilities": {
"items": {
"type": "string"
},
"type": "array"
},
"resource_envelope": {
"$ref": "#/definitions/ScenarioResourceEnvelope"
},
"scenario_id": {
"$ref": "#/definitions/ScenarioKey"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"source_path": {
"type": [
"string",
"null"
]
},
"source_sha256": {
"type": [
"string",
"null"
]
},
"title": {
"type": [
"string",
"null"
]
}
},
"required": [
"case_id",
"characterization",
"classification",
"complexity",
"contract_sha256",
"deliverable_contract",
"inputs_sha256",
"origin",
"required_capabilities",
"resource_envelope",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"ScenarioKey": {
"type": "string"
},
"ScenarioOrigin": {
"enum": [
"built_in",
"markdown",
"local"
],
"type": "string"
},
"ScenarioRealism": {
"properties": {
"execution": {
"allOf": [
{
"$ref": "#/definitions/ExecutionRealism"
}
],
"default": "synthetic"
},
"shadow": {
"allOf": [
{
"$ref": "#/definitions/ShadowMode"
}
],
"default": "none"
}
},
"type": "object"
},
"ScenarioResourceEnvelope": {
"properties": {
"execution": {
"$ref": "#/definitions/ExecutionPolicy"
},
"workflow": {
"anyOf": [
{
"$ref": "#/definitions/WorkflowResourceEnvelope"
},
{
"type": "null"
}
]
}
},
"required": [
"execution"
],
"type": "object"
},
"ShadowMode": {
"enum": [
"none",
"read_only"
],
"type": "string"
},
"WorkflowResourceEnvelope": {
"properties": {
"max_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"max_nodes": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"max_parallel": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"max_total_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"step_timeout_seconds": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"workflow_timeout_seconds": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"max_nodes",
"max_parallel",
"step_timeout_seconds",
"technical_retries",
"workflow_timeout_seconds"
],
"type": "object"
}
},
"properties": {
"catalog_sha256": {
"type": "string"
},
"runner": {
"$ref": "#/definitions/RunnerIdentity"
},
"scenarios": {
"items": {
"$ref": "#/definitions/ScenarioDescriptor"
},
"type": "array"
},
"schema": {
"type": "string"
}
},
"required": [
"catalog_sha256",
"runner",
"scenarios",
"schema"
],
"title": "ScenariosListResponse",
"type": "object"
}
},
{
"description": "Read the durable phase and progress of an E2E execution.",
"metadata": {
"contract": {
"capabilities": [
"status"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::status",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
}
},
"required": [
"execution_id"
],
"title": "ExecutionRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ActiveAttempt": {
"properties": {
"attempt_id": {
"type": "string"
},
"resume_state_path": {
"type": [
"string",
"null"
]
},
"resume_state_sha256": {
"type": [
"string",
"null"
]
},
"run_id": {
"default": "",
"type": "string"
},
"scenario_id": {
"$ref": "#/definitions/ScenarioKey"
},
"session_id": {
"type": "string"
}
},
"required": [
"attempt_id",
"scenario_id",
"session_id"
],
"type": "object"
},
"ExecutionPhase": {
"enum": [
"requested",
"admitted",
"preflighting",
"materializing",
"setting_up",
"executing",
"collecting",
"evaluating",
"persisting",
"cleaning_up",
"finalizing",
"completed",
"failed",
"cancelled",
"needs_reconciliation",
"unsupported"
],
"type": "string"
},
"PhaseTransition": {
"properties": {
"at": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
},
"reason": {
"type": "string"
}
},
"required": [
"at",
"phase",
"reason"
],
"type": "object"
},
"ScenarioKey": {
"type": "string"
}
},
"properties": {
"active_attempt": {
"anyOf": [
{
"$ref": "#/definitions/ActiveAttempt"
},
{
"type": "null"
}
]
},
"cancel_requested": {
"type": "boolean"
},
"error": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
},
"terminal": {
"type": "boolean"
},
"transitions": {
"items": {
"$ref": "#/definitions/PhaseTransition"
},
"type": "array"
},
"updated_at": {
"type": "string"
}
},
"required": [
"cancel_requested",
"execution_id",
"phase",
"terminal",
"transitions",
"updated_at"
],
"title": "StatusResponse",
"type": "object"
}
},
{
"description": "Serve the Harness E2E injectable Console assets.",
"metadata": {
"internal": true
},
"name": "e2e::ui-content",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"path": {
"type": "string"
}
},
"required": [
"path"
],
"title": "UiContentRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"content": {
"type": "string"
},
"content_type": {
"type": "string"
}
},
"required": [
"content",
"content_type"
],
"title": "UiContentResponse",
"type": "object"
}
}
],
"triggers": [
{
"description": "A local E2E execution started, produced progress, or reached a terminal state.",
"invocation_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"title": "EventBindingConfig",
"type": "object"
},
"metadata": {},
"name": "e2e::dashboard::changed",
"return_schema": {}
},
{
"description": "Fires when an object is written to a configured bucket whose notifications source is wired up.",
"invocation_schema": {},
"metadata": {},
"name": "storage::object-created",
"return_schema": {}
},
{
"description": "Fires when an object is removed from a configured bucket whose notifications source is wired up.",
"invocation_schema": {},
"metadata": {},
"name": "storage::object-deleted",
"return_schema": {}
}
]
}