$worker
harness-e2e
v0.9.9-experimentalMeasure Harness capability, retain reproducible evidence, and expose the E2E dashboard.
iiiverifiedinstall
$iii trigger compose::add worker=harness-e2e@0.9.9-experimental
- macOS: arm64
- Linux: arm64 · armv7 · x64
exact versions are immutable; binary and bundle artifacts are digest-pinned.
agent-ready brief for v0.9.9-experimental
install + config + dependencies + readme + api reference, all in one place. fetch as agent-context.md for an llm to consume.
full markdown
the same content rendered as discrete blocks below is exposed as a single markdown document at
/workers/harness-e2e.md?version=0.9.9-experimental. paste it into an llm prompt or pipe it through curl from a worker.install
install
$iii trigger compose::add worker=harness-e2e@0.9.9-experimental
configuration
iii-config.yaml
- data_dir: ~/.iii/data/harness-e2edependencies
readme
README.md
Harness E2E
harness-e2e measures which complexity levels a Harness stack can execute
with correct deliverables, structural integrity, bounded work, and repeatable
outcomes.
Objective scores preserve measured criterion points independently of completion or resource limits. Criteria do not veto the score or approve a run. Completion, technical validity, artifact evidence, and runtime controls are reported separately; infrastructure and execution failures still fail the CLI.
The repository is intentionally independent from the workers source tree.
Runtime discovery, execution, observation, state access, and cleanup all happen
through functions registered in iii. The only product input is an immutable
subject artifact or an already-running iii stack.
The SWE service suite provides eight isolated engineering
tasks and a continuous eight-ticket journey over the same Python service, with
optional delegation, immutable checkpoints, isolated verification, and a trusted
GitHub handoff.
SWE execution requires Linux with /usr/bin/bwrap and enabled unprivileged user
namespaces. CI installs the distribution AppArmor profile needed by Bubblewrap.
Commands and file operations run inside the attempt workspace; controller files
remain outside that boundary.
Binaries
harness-e2eis started by Compose and registers the asynchronouse2e::*control plane plus the injectable Console dashboard. Explicit subcommands keep direct scenario execution, report inspection, and the standalone dashboard available from the same binary.
Build and validate the repository:
pnpm --dir dashboard install --frozen-lockfile
pnpm --dir dashboard typecheck
pnpm --dir dashboard lint
pnpm --dir dashboard test
pnpm --dir dashboard build
cargo test --locked --all-targets
cargo clippy --locked --all-targets -- -D warnings
node --test tests/dashboard/*.test.cjs
HARNESS_E2E_BIN="$PWD/target/debug/harness-e2e" python3 -m unittest discover -s tests/python -p 'test_*.py'List the materialized scenarios and their scenario versions:
cargo run --locked --bin harness-e2e -- list
cargo run --locked --bin harness-e2e -- catalog
cargo run --locked --bin harness-e2e -- validate-scenariosThe four Registry scenarios
use the regular scenario catalog, execution flow, and scores: registry_planning,
registry_implementation, registry_environment, and registry_verification.
Each has its own atomic validations.
The Linkly tutorial scenario, linkly_tutorial,
runs the seven chapters of the agentic Linkly tutorial plus a project-restart guard as one
scripted dialogue on one Harness session, against the linkly-agentic scaffold's own Compose
stack, and scores twenty-two deterministic checks. scripts/linkly_stack.py prepares that stack.
The trending topics build scenario uses an isolated per-attempt Git remote and independent Playwright acceptance against the delivered SHA. Its runtime and controls require Linux amd64, Docker, Git, Python 3, Node and access to the pinned fixture. Design is free; screenshots are evidence, not an aesthetic score.
Native criteria preserve known awards when dependent checks cannot run. Those
checks have no award and remain not_evaluated; an incomplete criterion set has
no total score. Product failures stay technically valid, while infrastructure
failures invalidate the run without erasing prior criterion observations.
New declarative scenarios are authored only as scenarios/*.md. The compiler
embeds the exact source, validates the canonical English section structure,
and exposes the resulting file-stem id through the CLI, worker catalog,
campaign runner, dashboard, and canonical result artifacts. Required sections are
Version, Before Test, Prompt and Validations. Plans select their scenarios explicitly.
Replay an archived input only through its immutable plan (the runner rejects any scenario, model, policy, budget, stack, runner, run-count, or retry drift):
cargo run --locked -- replay-materialized \
target/e2e/evidence/<run-id>/<attempt-id>/materialized-plan.jsonRun against an existing stack:
cargo run --locked --bin harness-e2e -- run \
--url ws://127.0.0.1:49134 \
--model codex/gpt-5.6-luna \
--provider openai-codex \
--scenario todo_worker_simpleValidate one of the checked-in canonical campaign assets:
python3 scripts/run_e2e_campaign.py config/campaigns/endurance.json --e2e-bin target/debug/harness-e2e --validate-only
python3 scripts/run_e2e_campaign.py config/campaigns/endurance.json --e2e-bin target/debug/harness-e2e --dry-runOperational campaign execution is dispatched only by Release Control through
.github/workflows/exact-stack-e2e.yml. The repository no longer
publishes independent daily, weekly, post-deploy, or fault-stress dispatch
workflows.
Campaign manifests never select or rotate seeds. They separate replay-safe
turns from scripted dialogue and composite flows, persist a summary for every
group, and are advisory by default while their longitudinal history is being
calibrated. Release Control owns scheduling and dispatch; the executor keeps
the result advisory and archives each materialized group through the
environment-owned durable archiver.
The code-focused campaigns use protected disposable checkouts of
iii-hq/e2e-fixture. The engineering handoff uses its dedicated pinned
revision, while shell_coder_sandbox, chess_engine_build, and trend_blog
share a second pinned revision through HARNESS_E2E_FIXTURE_PATH. The protected
launcher enforces the fixture and cleanup boundary.
typescript_chat_service carries its own frozen skeleton in the repository and
needs no checkout, but it does require Node 22.6 or newer on the runner host: the
subject's TypeScript application is executed directly through Node type
stripping, both by the public suite and by the runner-owned behavioral probe.
config/test-plan.json defines the executable profiles: smoke, regression, capability, evolution,
resilience, endurance, and software-engineering. In the dashboard these profiles are starting templates
for the same plan form and baseline/candidate visualization used by existing plans.
Choose New plan, optionally select a template, edit the scope, and select the
execution model, plus the judge model when the scope includes a Markdown test.
Save draft, Save and run, and Duplicate plan use one shared lifecycle
and retain native evidence. Fault-injection plans export
to the protected executor. See executable profile plans.
cargo run --locked -- test-plan listThe software-engineering profile selects the seven incremental Kanban cases,
four Registry cases, the trending-topics blog build and the Linkly tutorial,
once each with no technical retries: 13 cases and 13 planned runs. Its twelve
execution groups keep Registry implementation and verification together, in
that order, so verification receives the implementation delivery. Registry
planning requires an explicit auxiliary judge model. Trending topics runs in its
own case-trending-topics-build group using the existing pinned fixture workflow.
Linkly runs its eight exchanges in one case-linkly-tutorial group. The executor
creates a fresh pinned linkly-agentic scaffold as that group's Compose project,
with baseline worker versions taken from the resolved stack contract.
cargo run --locked -- test-plan materialize --profile software-engineeringTemplates and execution rules are materialized directly by Rust from the source and native contracts. There are no generated catalogs to synchronize.
Release Control dispatches .github/workflows/exact-stack-e2e.yml directly in
this repository with five inputs and no decisions of its own: execution_id,
the plan naming one profile of config/test-plan.json, a stack policy
({"policy":"latest"} or exact versions), the executor commit runner_sha,
and the cli_version to install.
Everything else is resolved here, from the commit pinned by runner_sha:
harness-e2e test-plan materialize --profileexpands the profile into its campaigns, groups and cases, with aprofile_sha256over the result.scripts/resolve_stack_lock.pyturns the stack policy into one exactrc-e2e/v2contract per campaign — every Registry version resolved,latestnever surviving into a contract — whichscripts/exact_stack_campaign.pyvalidates as before.- Each group runs in an isolated ephemeral stack; fault groups route to the protected runner; one root bundle is produced without rebuilding the native Harness artifacts.
scripts/report_execution.py posts what was observed to Release Control's run
ledger over OIDC: materialized before anything runs, one shard per campaign
group whatever that group did, and a summary whatever the finalizer did. Runs
come from results.json, or from the journal checkpoints when a group died
before writing one; a group that produced neither still reports, saying so. No
execution is silently lost.
workers supplies versioned components of the stack under test; it does not
orchestrate campaigns.
Dashboard
Build and start the dashboard from the repository root:
cargo build --locked --bin harness-e2e
target/debug/harness-e2e dashboardThe Rust build follows the same embedded-SPA contract as workers/console: it
builds the React bundle with pnpm when dashboard/dist/ is missing or stale,
then embeds the Vite output in the binary. Node and pnpm must be available on
PATH. For frontend development with HMR, use pnpm --dir dashboard dev; the
Vite server proxies runtime data, the scoped iii WebSocket, and local-run APIs
to the Rust dashboard on port 4173.
Rust-defined composite scenarios, including the multi-test security_review
example, use the current shared result schema and read-only execution projection.
The running Harness must publish request and response schemas compatible with the current typed surface. Missing or incompatible fields fail preflight; no payload-version compatibility mode is available.
The server listens on 0.0.0.0:4173 by default. Open
http://localhost:4173/#/overview on the same machine, or replace localhost
with the machine's address when accessing it remotely. Use --listen 0.0.0.0:PORT to select another port, III_URL to select the running Harness
stack, and --runs-dir to select another local history directory.
Local mode loads data incrementally through iii: 25 compact summaries on the
first overview page, one complete report when an execution is opened, only the
selected pair for comparison, and the model/scenario catalog when the run dialog
opens. Server-side filtering and cursor pagination keep history growth out of
the initial payload. Static published and --view-only presentations preserve
the generated-file fallback.
Local mode exposes controls that can start and cancel E2E runs, so expose the
port only on a trusted network. Use --listen 127.0.0.1:4173 when access should
remain local. See dashboard/README.md for view-only mode
and the complete dashboard behavior.
Compare a local change with Release Control
The Console's Plans page offers Reference: Release Control to browse RC history through the authenticated Release Control browser bridge. Keep the RC tab open, enable its local Harness connection, and connect it to the same personal Engine as the Console. The bridge needs the E2E read functions from the companion Release Control change. No GitHub token or artifact synchronization is needed.
Open a plan to see remote and local executions together with their origin. Select a reference and a local result to compare their measurements. The scenario links open the existing A → B comparison with both executions selected. Missing reports and metrics remain visible as unavailable; reading history creates no local plan. The comparison runs locally and sends no local results to Release Control.
Choose run locally on a remote reference to save its materialized test parameters as a local plan and run them against your current Harness. Repeating that action creates a new local plan using the current scenario contracts, while preserving earlier plans and results. The reference's scenarios, rounds, repetitions and retry settings come from the execution's materialization, not from the current profile with the same name. The current local scenario implementations and Harness are used deliberately: this is a personal experiment, not an exact-stack certification. No build/Git tracking or matching remote stack is required. Fault-injection groups still require the protected executor; they are not silently omitted. References without shard seeds for every scenario cannot be reproduced. Differences in local scenario version or case identity are shown as advisory information.
Results stay in the local plan store. The RC execution remains a reference, never a locally recreated official execution. Native result validation remains strict; the remote data is read through the RC API rather than installed as a native report. Full remote evidence is available through the execution's GitHub link, subject to its retention; this flow does not download an evidence archive.
Compose lifecycle
Release Control names the exact project roots. This repository writes the
root configuration and passes those worker@version references to
compose::add; iii resolves the Registry graph, writes the project topology,
and reconciles its containers. Every execution starts an empty Engine and a
dedicated Compose daemon, then runs compose::add, compose::up,
compose::status, and compose::down. Each execution uses one isolated
namespace for both Compose and the project functions it starts.
Compose supplies III_URL, III_NAMESPACE, III_WORKER_NAME, and III_CONFIG
to the harness-e2e process. All four values are mandatory. The referenced
configuration file contains the execution-specific data_dir; there is no
local fallback, command-line override, or runtime self-registration.
Publication validates the locally built binary through a path:// Compose
container before the package is uploaded. Published campaigns use only exact
Registry package versions. Provider secrets are written to temporary
permission-restricted env_file files and are never included in contract,
Compose, evidence, or archive artifacts. Those files are attached to the
providers and to the exact llm-router dependency pinned in the contract:
the router resolves provider credentials and needs them in its own isolated
environment to report the providers as configured. The scaffold predeclares that
dependency's configuration without changing the contract's roots or versions.
Engine registration is not model readiness: provider registration and model
discovery continue asynchronously after Compose reports a worker ready. Before
materializing a group, the runner waits up to 180 seconds for the exact subject
in router::models::get (HARNESS_E2E_MODEL_WAIT_SECONDS overrides this bound).
An unavailable model fails in model_readiness, preserves the last catalog
response and provider status under stack/, and never falls back to another model.
The worker exposes e2e::run, e2e::status, e2e::cancel,
e2e::results-get, e2e::results-list, e2e::compare,
e2e::scenarios-list, e2e::scenarios-create,
e2e::scenarios-authoring-guide, e2e::archive, e2e::archive-head,
e2e::archive-restore,
e2e::history-list, and e2e::retention-sweep.
Fault supervisors use e2e::fault-plan and e2e::fault-evaluate so plan
materialization and recovery classification stay on the same iii control plane.
Subject policies deny e2e::*.
Durable artifacts are chunked through storage::*, while longitudinal series
are ingested through database::*. The runner has no S3, GCS, R2, SQL-driver,
or Harness dependency.
Weekly Stress materializes deterministic fault plans and evaluates journals from a
protected supervisor.
Lane promotion is governed by
config/policies/cutover.json.
Repository boundaries
src/owns the runner, local wire adapters, scenarios, evaluation, longitudinal comparison, and the E2E control worker.config/owns reviewed comparison and cutover policies, fault profiles, and standalone stack configuration.tests/owns test-only fixtures, golden wire schemas, and the Node/Python validation suites.schemas/contains the public contracts for generated E2E artifacts.dashboard/contains the React, TypeScript, Vite, and Tailwind dashboard embedded in the Rust binary.- generated reports, transcripts, logs, and deliverables stay outside Git.
The crate may depend on the iii SDK and generic libraries. It must not declare
a path or Git dependency on workers, Harness, or another product crate.
Contract compatibility is established at runtime from
engine::functions::list and engine::functions::info; the checked-in schemas
are parity fixtures, not a linked product API.
The deterministic assessment boundary has one current payload shape, written
only to results.json; scenario contracts are the only versioned domain. The
judge model is auxiliary: Markdown tests use it for their validators and
instruction adherence, and only those tests require it.
Deterministic, pre-cleanup asset capture applies explicit safety limits and
writes an unversioned sidecar containing the canonical deterministic validation
portion, which is aggregated into results.json.
Observation
The runner waits for a session tree to finish by binding
harness::turn-completed to an internal sink (e2e::on-turn-completed) before
harness::send. That sink is not a control-plane verb: it is not registered
with e2e::run / e2e::status / e2e::cancel, and it does not appear in
e2e::scenarios-list. Subject policies already deny e2e::*.
A 15s watchdog samples harness::metrics and one root harness::status for
stuck detection, heartbeat logs, and e2e::cancel. If the trigger type is
missing from engine::triggers::list, the run is unsupported infrastructure —
there is no silent fallback to polling harness::status or harness::metrics.
After the tree completes, the runner still collects terminal status, metrics,
transcripts, and deliverables.
Subject artifacts
Cross-repository executions accept a subject manifest matching
schemas/subject-artifact.json. The archive and every declared file are
verified before use. Mutable URLs, shortened Git revisions, unexpected archive
paths, and digest mismatches are rejected.
Untrusted subject artifacts are never given provider, storage, or GitHub credentials in their environment. Provider workers and the trusted E2E worker are started separately. PR execution remains non-blocking shadow evidence until the source repository, revision, E2E ref, and credential boundary are approved.
Comparison
Every completed execution records the subject and E2E revisions, observed wire
contracts, scenario version, materialized inputs, seed, policies, artifacts,
and raw structural evidence. e2e::compare accepts two distinct completed
execution ids (from_execution_id and to_execution_id) and writes a unique
comparisons/ plus e2e-summary.md. Numeric
deltas remain disabled when the case set or canonical contract differs.
Deliverable, structural, technical, cost, latency, turns, retries, and work amplification deltas remain independent. Cost and wall-time are reported as observed metrics and compared only within a compatible baseline/candidate cohort. amplification deltas remain independent. A tier is repeatable after five local runs satisfy the deliverable, structural, and technical thresholds. Cost and wall-time are reported as observed metrics and compared only within a compatible baseline/candidate cohort.
Runtime-only package boundary
This repository executes exact-stack Test Plans and never publishes itself as
a Registry worker. Release Control supplies a stack policy and an immutable
executor SHA to exact-stack-e2e.yml; the contract this repository assembles
from them pins every Registry version to an exact one, including historical
candidates, because a campaign has to be able to say afterwards what it ran.
The root iii.worker.yaml remains the public manifest for local iii worker
development and package compatibility. The root worker-compose.yaml remains
a normal public Compose document. Release Control and post-prepare workflow
phases deliberately read neither source contract.
api reference (json)
agent-api-reference.json
{
"functions": [
{
"description": "Persist a completed execution through private iii storage and idempotent history.",
"metadata": {
"contract": {
"capabilities": [
"archive"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::archive",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"RetentionClass": {
"enum": [
"temporary",
"pull_request",
"longitudinal",
"canonical"
],
"type": "string"
}
},
"properties": {
"execution_id": {
"type": "string"
},
"retention_class": {
"$ref": "#/definitions/RetentionClass"
}
},
"required": [
"execution_id",
"retention_class"
],
"title": "ArchiveExecutionRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"DurableArchiveReference": {
"additionalProperties": false,
"properties": {
"archive_id": {
"type": "string"
},
"created_at": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"expires_at": {
"type": [
"string",
"null"
]
},
"identity_sha256": {
"type": "string"
},
"manifest": {
"$ref": "#/definitions/StorageObjectReference"
},
"manifest_backup": {
"$ref": "#/definitions/StorageObjectReference"
},
"retention_class": {
"$ref": "#/definitions/RetentionClass"
}
},
"required": [
"archive_id",
"created_at",
"execution_id",
"identity_sha256",
"manifest",
"manifest_backup",
"retention_class"
],
"type": "object"
},
"HistoryRecord": {
"additionalProperties": false,
"properties": {
"archive": {
"$ref": "#/definitions/DurableArchiveReference"
},
"baseline_comparable": {
"type": "boolean"
},
"case_count": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"e2e_repository": {
"type": "string"
},
"e2e_revision": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"identity_sha256": {
"type": "string"
},
"ingestion_id": {
"type": "string"
},
"lane": {
"type": "string"
},
"occurred_at": {
"type": "string"
},
"passed": {
"type": "boolean"
},
"result_contract_sha256": {
"type": "string"
},
"scoring_profile_sha256": {
"type": "string"
},
"stack_mode": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"subject_revision": {
"type": "string"
}
},
"required": [
"archive",
"baseline_comparable",
"case_count",
"e2e_repository",
"e2e_revision",
"execution_id",
"identity_sha256",
"ingestion_id",
"lane",
"occurred_at",
"passed",
"result_contract_sha256",
"scoring_profile_sha256",
"stack_mode",
"subject_model",
"subject_provider",
"subject_revision"
],
"type": "object"
},
"RetentionClass": {
"enum": [
"temporary",
"pull_request",
"longitudinal",
"canonical"
],
"type": "string"
},
"StorageObjectReference": {
"properties": {
"bucket": {
"type": "string"
},
"key": {
"type": "string"
},
"media_type": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"uri": {
"type": "string"
}
},
"required": [
"bucket",
"key",
"media_type",
"sha256",
"size_bytes",
"uri"
],
"type": "object"
}
},
"properties": {
"archive": {
"$ref": "#/definitions/DurableArchiveReference"
},
"duplicate_ingestion": {
"type": "boolean"
},
"history": {
"$ref": "#/definitions/HistoryRecord"
}
},
"required": [
"archive",
"duplicate_ingestion",
"history"
],
"title": "ArchiveResponse",
"type": "object"
}
},
{
"description": "Verify availability and immutable metadata for an archived execution.",
"metadata": {
"contract": {
"capabilities": [
"archive-head"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::archive-head",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
}
},
"required": [
"execution_id"
],
"title": "ExecutionRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ArchiveAvailability": {
"enum": [
"available",
"expired"
],
"type": "string"
},
"DurableArchiveReference": {
"additionalProperties": false,
"properties": {
"archive_id": {
"type": "string"
},
"created_at": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"expires_at": {
"type": [
"string",
"null"
]
},
"identity_sha256": {
"type": "string"
},
"manifest": {
"$ref": "#/definitions/StorageObjectReference"
},
"manifest_backup": {
"$ref": "#/definitions/StorageObjectReference"
},
"retention_class": {
"$ref": "#/definitions/RetentionClass"
}
},
"required": [
"archive_id",
"created_at",
"execution_id",
"identity_sha256",
"manifest",
"manifest_backup",
"retention_class"
],
"type": "object"
},
"RetentionClass": {
"enum": [
"temporary",
"pull_request",
"longitudinal",
"canonical"
],
"type": "string"
},
"StorageObjectReference": {
"properties": {
"bucket": {
"type": "string"
},
"key": {
"type": "string"
},
"media_type": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"uri": {
"type": "string"
}
},
"required": [
"bucket",
"key",
"media_type",
"sha256",
"size_bytes",
"uri"
],
"type": "object"
}
},
"properties": {
"archive": {
"$ref": "#/definitions/DurableArchiveReference"
},
"availability": {
"$ref": "#/definitions/ArchiveAvailability"
},
"verified_at": {
"type": "string"
}
},
"required": [
"archive",
"availability",
"verified_at"
],
"title": "ArchiveHeadResponse",
"type": "object"
}
},
{
"description": "Restore and verify an archived execution from its immutable manifest.",
"metadata": {
"contract": {
"capabilities": [
"archive-restore"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::archive-restore",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
}
},
"required": [
"execution_id"
],
"title": "ExecutionRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ArchiveAvailability": {
"enum": [
"available",
"expired"
],
"type": "string"
},
"DurableArchiveReference": {
"additionalProperties": false,
"properties": {
"archive_id": {
"type": "string"
},
"created_at": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"expires_at": {
"type": [
"string",
"null"
]
},
"identity_sha256": {
"type": "string"
},
"manifest": {
"$ref": "#/definitions/StorageObjectReference"
},
"manifest_backup": {
"$ref": "#/definitions/StorageObjectReference"
},
"retention_class": {
"$ref": "#/definitions/RetentionClass"
}
},
"required": [
"archive_id",
"created_at",
"execution_id",
"identity_sha256",
"manifest",
"manifest_backup",
"retention_class"
],
"type": "object"
},
"RetentionClass": {
"enum": [
"temporary",
"pull_request",
"longitudinal",
"canonical"
],
"type": "string"
},
"StorageObjectReference": {
"properties": {
"bucket": {
"type": "string"
},
"key": {
"type": "string"
},
"media_type": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"uri": {
"type": "string"
}
},
"required": [
"bucket",
"key",
"media_type",
"sha256",
"size_bytes",
"uri"
],
"type": "object"
}
},
"properties": {
"archive": {
"$ref": "#/definitions/DurableArchiveReference"
},
"availability": {
"$ref": "#/definitions/ArchiveAvailability"
},
"restored_files": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"restored_from_backup": {
"type": "boolean"
},
"restored_root": {
"type": [
"string",
"null"
]
}
},
"required": [
"archive",
"availability",
"restored_files",
"restored_from_backup"
],
"title": "ArchiveRestoreResponse",
"type": "object"
}
},
{
"description": "Request idempotent cancellation and cleanup of an E2E execution.",
"metadata": {
"contract": {
"capabilities": [
"cancel"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::cancel",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
}
},
"required": [
"execution_id"
],
"title": "ExecutionRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ExecutionPhase": {
"enum": [
"requested",
"admitted",
"preflighting",
"materializing",
"setting_up",
"executing",
"collecting",
"evaluating",
"persisting",
"cleaning_up",
"finalizing",
"completed",
"failed",
"cancelled",
"needs_reconciliation",
"unsupported"
],
"type": "string"
}
},
"properties": {
"accepted": {
"type": "boolean"
},
"execution_id": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
}
},
"required": [
"accepted",
"execution_id",
"phase"
],
"title": "CancelResponse",
"type": "object"
}
},
{
"description": "Compare two completed executions when their identities are eligible.",
"metadata": {
"contract": {
"capabilities": [
"compare"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::compare",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ComparisonPolicy": {
"properties": {
"baseline_id": {
"description": "Identifier of the reviewed baseline these thresholds came from, when one was loaded. Absent when the code-default thresholds are in effect.",
"type": [
"string",
"null"
]
},
"regression": {
"allOf": [
{
"$ref": "#/definitions/RegressionThresholds"
}
],
"default": {
"cost_increase_ratio": 0.2,
"deliverable_success_drop": 0.03,
"p95_turns_increase_ratio": 0.25,
"structural_integrity_drop": 0.02,
"technical_failure_increase": 0.02,
"wall_time_increase_ratio": 0.2
}
},
"scenario_overrides": {
"additionalProperties": {
"$ref": "#/definitions/ThresholdOverrides"
},
"description": "Per-scenario partial threshold overrides carried from the reviewed baseline.",
"type": "object"
}
},
"type": "object"
},
"RegressionThresholds": {
"properties": {
"cost_increase_ratio": {
"format": "double",
"type": "number"
},
"deliverable_success_drop": {
"format": "double",
"type": "number"
},
"p95_turns_increase_ratio": {
"format": "double",
"type": "number"
},
"structural_integrity_drop": {
"format": "double",
"type": "number"
},
"technical_failure_increase": {
"format": "double",
"type": "number"
},
"wall_time_increase_ratio": {
"format": "double",
"type": "number"
}
},
"required": [
"cost_increase_ratio",
"deliverable_success_drop",
"p95_turns_increase_ratio",
"structural_integrity_drop",
"technical_failure_increase",
"wall_time_increase_ratio"
],
"type": "object"
},
"ThresholdOverrides": {
"additionalProperties": false,
"description": "Partial per-scenario replacement for [`RegressionThresholds`]: a named field replaces the baseline value, an omitted field inherits it.",
"properties": {
"cost_increase_ratio": {
"format": "double",
"type": [
"number",
"null"
]
},
"deliverable_success_drop": {
"format": "double",
"type": [
"number",
"null"
]
},
"p95_turns_increase_ratio": {
"format": "double",
"type": [
"number",
"null"
]
},
"structural_integrity_drop": {
"format": "double",
"type": [
"number",
"null"
]
},
"technical_failure_increase": {
"format": "double",
"type": [
"number",
"null"
]
},
"wall_time_increase_ratio": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"type": "object"
}
},
"properties": {
"baseline": {
"default": null,
"description": "Reviewed baseline file to load thresholds from. Omit to use the checked-in `config/baselines/default.json` when it exists, falling back to the code-default thresholds. Ignored when `policy` is set.",
"type": [
"string",
"null"
]
},
"from_execution_id": {
"type": "string"
},
"policy": {
"anyOf": [
{
"$ref": "#/definitions/ComparisonPolicy"
},
{
"type": "null"
}
],
"default": null,
"description": "Explicit policy override. Omit to gate on the reviewed baseline."
},
"to_execution_id": {
"type": "string"
}
},
"required": [
"from_execution_id",
"to_execution_id"
],
"title": "CompareRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ArtifactReference": {
"properties": {
"id": {
"type": "string"
},
"kind": {
"type": "string"
},
"media_type": {
"type": "string"
},
"path": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"id",
"kind",
"media_type",
"path",
"sha256",
"size_bytes"
],
"type": "object"
},
"BenchmarkDelta": {
"properties": {
"consumption": {
"additionalProperties": {
"$ref": "#/definitions/DeltaValue"
},
"default": {},
"description": "Descriptive only; these deltas never create a regression gate.",
"type": "object"
},
"deliverable_success_rate": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"flaky_rate": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"median_score": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p50_cost_usd": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p50_turns": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p50_wall_time_ms": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p50_work_amplification": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p95_cost_usd": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p95_turns": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p95_wall_time_ms": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p95_work_amplification": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"retry_rate": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"structural_integrity_rate": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"technical_failure_rate": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"unavailable": {
"additionalProperties": {
"type": "string"
},
"type": "object"
}
},
"type": "object"
},
"CaseCohortKey": {
"properties": {
"case_id": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"execution": {
"$ref": "#/definitions/ExecutionCohortIdentity"
},
"inputs_sha256": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"case_id",
"contract_sha256",
"execution",
"inputs_sha256",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"CaseComparison": {
"properties": {
"complexity_tier": {
"$ref": "#/definitions/ComplexityTier"
},
"delta": {
"$ref": "#/definitions/BenchmarkDelta"
},
"from": {
"$ref": "#/definitions/CaseMetrics"
},
"from_run_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"key": {
"$ref": "#/definitions/CaseCohortKey"
},
"regressions": {
"items": {
"$ref": "#/definitions/RegressionSignal"
},
"type": "array"
},
"to": {
"$ref": "#/definitions/CaseMetrics"
},
"to_run_ids": {
"items": {
"type": "string"
},
"type": "array"
}
},
"required": [
"complexity_tier",
"delta",
"from",
"from_run_ids",
"key",
"regressions",
"to",
"to_run_ids"
],
"type": "object"
},
"CaseMetrics": {
"properties": {
"consumption": {
"allOf": [
{
"$ref": "#/definitions/ConsumptionMetrics"
}
],
"default": {
"completed_runs": 0,
"observed_runs": 0,
"p50_function_calls": null,
"p50_total_tokens": null,
"p95_function_calls": null,
"p95_total_tokens": null,
"tokens_per_completion": null,
"total_tokens_consumed": null,
"unavailable": {}
},
"description": "Includes every observed run, even those excluded from quality comparison."
},
"deliverable_success": {
"anyOf": [
{
"$ref": "#/definitions/RateEstimate"
},
{
"type": "null"
}
]
},
"excluded_infrastructure_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"flaky_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"included_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"median_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"p50_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"p50_turns": {
"format": "double",
"type": [
"number",
"null"
]
},
"p50_wall_time_ms": {
"format": "double",
"type": [
"number",
"null"
]
},
"p50_work_amplification": {
"format": "double",
"type": [
"number",
"null"
]
},
"p95_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"p95_turns": {
"format": "double",
"type": [
"number",
"null"
]
},
"p95_wall_time_ms": {
"format": "double",
"type": [
"number",
"null"
]
},
"p95_work_amplification": {
"format": "double",
"type": [
"number",
"null"
]
},
"retry_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"structural_integrity": {
"anyOf": [
{
"$ref": "#/definitions/RateEstimate"
},
{
"type": "null"
}
]
},
"technical_failure": {
"anyOf": [
{
"$ref": "#/definitions/RateEstimate"
},
{
"type": "null"
}
]
},
"unavailable": {
"additionalProperties": {
"type": "string"
},
"type": "object"
}
},
"required": [
"excluded_infrastructure_runs",
"included_runs"
],
"type": "object"
},
"CohortAudit": {
"properties": {
"excluded_cases": {
"items": {
"$ref": "#/definitions/ExcludedCase"
},
"type": "array"
},
"excluded_runs": {
"items": {
"$ref": "#/definitions/ExcludedRun"
},
"type": "array"
},
"included_case_ids": {
"items": {
"type": "string"
},
"type": "array"
}
},
"required": [
"excluded_cases",
"excluded_runs",
"included_case_ids"
],
"type": "object"
},
"ComparisonSide": {
"enum": [
"from",
"to"
],
"type": "string"
},
"ComparisonSummary": {
"properties": {
"baseline_id": {
"description": "Reviewed baseline the regression thresholds came from; absent when the code-default thresholds gated this comparison.",
"type": [
"string",
"null"
]
},
"cases": {
"items": {
"$ref": "#/definitions/CaseComparison"
},
"type": "array"
},
"cohort": {
"$ref": "#/definitions/CohortAudit"
},
"comparable": {
"type": "boolean"
},
"comparison_id": {
"type": "string"
},
"from_execution_id": {
"type": "string"
},
"from_revision": {
"type": [
"string",
"null"
]
},
"gate_passed": {
"type": "boolean"
},
"reasons": {
"items": {
"type": "string"
},
"type": "array"
},
"regressions": {
"items": {
"$ref": "#/definitions/RegressionSignal"
},
"type": "array"
},
"to_execution_id": {
"type": "string"
},
"to_revision": {
"type": [
"string",
"null"
]
}
},
"required": [
"cases",
"cohort",
"comparable",
"comparison_id",
"from_execution_id",
"gate_passed",
"reasons",
"regressions",
"to_execution_id"
],
"type": "object"
},
"ComplexityTier": {
"enum": [
"l0_atomic",
"l1_sequential",
"l2_stateful",
"l3_concurrent",
"l4_coordinated",
"l5_adaptive"
],
"type": "string"
},
"ConsumptionMetrics": {
"description": "Subject-side usage across all observed attempts.",
"properties": {
"completed_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"observed_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"p50_function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"p50_total_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"p95_function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"p95_total_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"tokens_per_completion": {
"format": "double",
"type": [
"number",
"null"
]
},
"total_tokens_consumed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"unavailable": {
"additionalProperties": {
"type": "string"
},
"type": "object"
}
},
"required": [
"completed_runs",
"observed_runs",
"unavailable"
],
"type": "object"
},
"DeltaValue": {
"properties": {
"absolute": {
"format": "double",
"type": "number"
},
"relative_ratio": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"absolute"
],
"type": "object"
},
"ExcludedCase": {
"properties": {
"case_id": {
"type": "string"
},
"reasons": {
"items": {
"type": "string"
},
"type": "array"
},
"scenario_id": {
"type": "string"
},
"side": {
"$ref": "#/definitions/ComparisonSide"
}
},
"required": [
"case_id",
"reasons",
"scenario_id",
"side"
],
"type": "object"
},
"ExcludedRun": {
"properties": {
"attempt_id": {
"type": "string"
},
"case_id": {
"type": "string"
},
"reason": {
"type": "string"
},
"run_id": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"side": {
"$ref": "#/definitions/ComparisonSide"
}
},
"required": [
"attempt_id",
"case_id",
"reason",
"run_id",
"scenario_id",
"side"
],
"type": "object"
},
"ExecutionCohortIdentity": {
"properties": {
"e2e_repository": {
"type": [
"string",
"null"
]
},
"judge_model": {
"type": [
"string",
"null"
]
},
"judge_provider": {
"type": [
"string",
"null"
]
},
"lane": {
"type": "string"
},
"stack_mode": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
}
},
"required": [
"lane",
"stack_mode",
"subject_model",
"subject_provider"
],
"type": "object"
},
"RateEstimate": {
"properties": {
"ci95_lower": {
"format": "double",
"type": "number"
},
"ci95_upper": {
"format": "double",
"type": "number"
},
"rate": {
"format": "double",
"type": "number"
},
"sample_size": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"successes": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"ci95_lower",
"ci95_upper",
"rate",
"sample_size",
"successes"
],
"type": "object"
},
"RegressionDimension": {
"enum": [
"deliverable",
"structural_integrity",
"technical_reliability",
"cost",
"wall_time",
"turns"
],
"type": "string"
},
"RegressionSignal": {
"properties": {
"blocking": {
"type": "boolean"
},
"case_id": {
"type": "string"
},
"dimension": {
"$ref": "#/definitions/RegressionDimension"
},
"maturity": {
"default": "",
"type": "string"
},
"n_from": {
"default": 0,
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"n_to": {
"default": 0,
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"observed_delta": {
"format": "double",
"type": "number"
},
"reason": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"statistic": {
"default": "",
"type": "string"
},
"threshold": {
"format": "double",
"type": "number"
}
},
"required": [
"blocking",
"case_id",
"dimension",
"observed_delta",
"reason",
"scenario_id",
"threshold"
],
"type": "object"
}
},
"properties": {
"artifacts": {
"items": {
"$ref": "#/definitions/ArtifactReference"
},
"type": "array"
},
"comparison": {
"$ref": "#/definitions/ComparisonSummary"
}
},
"required": [
"artifacts",
"comparison"
],
"title": "ComparisonResponse",
"type": "object"
}
},
{
"description": "Read models and scenarios when the execution dialog opens.",
"metadata": {
"contract": {
"capability": "catalog-get",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::catalog-get",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"url": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "CatalogRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"CatalogModel": {
"properties": {
"id": {
"type": "string"
},
"provider": {
"type": "string"
}
},
"required": [
"id",
"provider"
],
"type": "object"
},
"LocalScenarioSummary": {
"properties": {
"id": {
"type": "string"
},
"source_path": {
"type": "string"
},
"source_sha256": {
"type": "string"
},
"title": {
"type": "string"
},
"version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"id",
"source_path",
"source_sha256",
"title",
"version"
],
"type": "object"
}
},
"properties": {
"local_scenarios": {
"items": {
"$ref": "#/definitions/LocalScenarioSummary"
},
"type": "array"
},
"models": {
"items": {
"$ref": "#/definitions/CatalogModel"
},
"type": "array"
},
"scenarios": {
"items": {
"type": "string"
},
"type": "array"
},
"url": {
"type": "string"
}
},
"required": [
"local_scenarios",
"models",
"scenarios",
"url"
],
"title": "CatalogResponse",
"type": "object"
}
},
{
"description": "List immutable evaluated-system versions and their exact evaluation cohorts.",
"metadata": {
"contract": {
"capability": "evaluated-versions-list",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::evaluated-versions-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"cohort_id": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "EvaluatedVersionsRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"CohortDescriptor": {
"properties": {
"id": {
"type": "string"
},
"judge_model": {
"type": [
"string",
"null"
]
},
"judge_provider": {
"type": [
"string",
"null"
]
},
"lane": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
}
},
"required": [
"id",
"lane",
"subject_model",
"subject_provider"
],
"type": "object"
},
"EvaluatedVersionDescriptor": {
"properties": {
"cohort_id": {
"type": "string"
},
"completed_at": {
"type": "string"
},
"execution_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"id": {
"type": "string"
},
"label": {
"type": "string"
},
"stack_mode": {
"type": "string"
}
},
"required": [
"cohort_id",
"completed_at",
"execution_count",
"id",
"label",
"stack_mode"
],
"type": "object"
}
},
"properties": {
"cohorts": {
"items": {
"$ref": "#/definitions/CohortDescriptor"
},
"type": "array"
},
"revision": {
"type": "string"
},
"versions": {
"items": {
"$ref": "#/definitions/EvaluatedVersionDescriptor"
},
"type": "array"
}
},
"required": [
"cohorts",
"revision",
"versions"
],
"title": "EvaluatedVersionsResponse",
"type": "object"
}
},
{
"description": "Read one retained execution summary and its diagnostic report.",
"metadata": {
"contract": {
"capability": "execution-get",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::execution-get",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
}
},
"required": [
"execution_id"
],
"title": "ExecutionGetRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ExecutionListResponse": {
"properties": {
"executions": {
"items": true,
"type": "array"
},
"last_update": {
"type": "string"
},
"mode": {
"type": "string"
},
"next_cursor": {
"type": [
"string",
"null"
]
},
"repo_url": {
"type": "string"
},
"retention": {
"$ref": "#/definitions/Retention"
},
"total": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"executions",
"last_update",
"mode",
"repo_url",
"retention",
"total"
],
"type": "object"
},
"Retention": {
"properties": {
"details": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"summaries": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"details",
"summaries"
],
"type": "object"
}
},
"properties": {
"detail": true,
"manifest": {
"$ref": "#/definitions/ExecutionListResponse"
}
},
"required": [
"detail",
"manifest"
],
"title": "ExecutionBundle",
"type": "object"
}
},
{
"description": "List compact retained execution summaries with server-side filters and cursor pagination.",
"metadata": {
"contract": {
"capability": "executions-list",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::executions-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"cursor": {
"default": null,
"type": [
"string",
"null"
]
},
"event": {
"default": null,
"type": [
"string",
"null"
]
},
"ids": {
"default": [],
"items": {
"type": "string"
},
"type": "array"
},
"ids_csv": {
"default": null,
"type": [
"string",
"null"
]
},
"limit": {
"default": null,
"format": "uint16",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"query": {
"default": null,
"type": [
"string",
"null"
]
},
"status": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "ExecutionListRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"Retention": {
"properties": {
"details": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"summaries": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"details",
"summaries"
],
"type": "object"
}
},
"properties": {
"executions": {
"items": true,
"type": "array"
},
"last_update": {
"type": "string"
},
"mode": {
"type": "string"
},
"next_cursor": {
"type": [
"string",
"null"
]
},
"repo_url": {
"type": "string"
},
"retention": {
"$ref": "#/definitions/Retention"
},
"total": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"executions",
"last_update",
"mode",
"repo_url",
"retention",
"total"
],
"title": "ExecutionListResponse",
"type": "object"
}
},
{
"description": "Validate and save one local-only Markdown scenario, then return its compiled identity.",
"metadata": {
"contract": {
"capability": "local-scenario-create",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::local-scenario-create",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"file_name": {
"description": "One UTF-8 `.md` file name, without a directory. Use letters, numbers, spaces, hyphens, or underscores; its stem becomes `local_<safe_id>`. Existing files are rejected instead of overwritten.",
"type": "string"
},
"source": {
"description": "Complete Markdown test definition. It must contain one H1 followed by the required H2 sections in order and contain positive `### Name (N%)` validations totaling exactly 100%.",
"type": "string"
}
},
"required": [
"file_name",
"source"
],
"title": "LocalScenarioCreateRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ArtifactExpectation": {
"properties": {
"id": {
"type": "string"
},
"kind": {
"type": "string"
},
"max_size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"media_type": {
"type": "string"
},
"schema": true
},
"required": [
"id",
"kind",
"max_size_bytes",
"media_type",
"schema"
],
"type": "object"
},
"ComplexityClassification": {
"properties": {
"method": {
"$ref": "#/definitions/ComplexityMethod"
},
"profile": {
"$ref": "#/definitions/ComplexityProfile"
},
"tier": {
"$ref": "#/definitions/ComplexityTier"
}
},
"required": [
"method",
"profile",
"tier"
],
"type": "object"
},
"ComplexityMethod": {
"enum": [
"capability_v2"
],
"type": "string"
},
"ComplexityProfile": {
"properties": {
"agent_owned_decomposition": {
"default": false,
"type": "boolean"
},
"ambiguity_level": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"artifact_count": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"coherent_long_horizon": {
"default": false,
"type": "boolean"
},
"compensable_mutations": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"coordination_edges": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"dependency_depth": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"durable_resume_cycles": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"external_systems": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"material_invalidation_events": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"parallel_branches": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"planning_depth": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"replan_loops": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"state_transitions": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"validation_loops": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"wake_cycles": {
"format": "uint8",
"minimum": 0,
"type": "integer"
}
},
"required": [
"ambiguity_level",
"artifact_count",
"coordination_edges",
"dependency_depth",
"external_systems",
"parallel_branches",
"planning_depth",
"state_transitions",
"validation_loops",
"wake_cycles"
],
"type": "object"
},
"ComplexityTier": {
"enum": [
"l0_atomic",
"l1_sequential",
"l2_stateful",
"l3_concurrent",
"l4_coordinated",
"l5_adaptive"
],
"type": "string"
},
"DeliverableContract": {
"properties": {
"artifacts": {
"items": {
"$ref": "#/definitions/ArtifactExpectation"
},
"type": "array"
},
"capture_before_cleanup": {
"type": "boolean"
},
"invariants": {
"items": {
"$ref": "#/definitions/InvariantSpec"
},
"type": "array"
},
"provenance_required": {
"type": "boolean"
}
},
"required": [
"artifacts",
"capture_before_cleanup",
"invariants",
"provenance_required"
],
"type": "object"
},
"ExecutionPolicy": {
"properties": {
"max_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_total_tokens": {
"description": "Shared Harness token budget. `None` leaves the Harness budget unbounded for this scenario.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_turns": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"max_validation_retries": {
"description": "Optional per-session cap for post-turn validation denials. `None` preserves the Harness-configured default.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stuck_timeout_seconds": {
"description": "Stop only after this many seconds without observable useful progress. Large scenarios have no fixed wall-clock deadline.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"max_turns",
"stuck_timeout_seconds"
],
"type": "object"
},
"ExecutionRealism": {
"enum": [
"synthetic",
"realistic_simulator",
"frozen_real_artifact"
],
"type": "string"
},
"HumanHorizon": {
"properties": {
"basis": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizonBasis"
}
],
"default": "unknown"
},
"max_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"min_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"HumanHorizonBasis": {
"enum": [
"unknown",
"author_estimate",
"measured"
],
"type": "string"
},
"InvariantSpec": {
"properties": {
"description": {
"type": "string"
},
"id": {
"type": "string"
}
},
"required": [
"description",
"id"
],
"type": "object"
},
"ScenarioCharacterization": {
"properties": {
"human_horizon": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizon"
}
],
"default": {
"basis": "unknown"
}
},
"realism": {
"allOf": [
{
"$ref": "#/definitions/ScenarioRealism"
}
],
"default": {
"execution": "synthetic",
"shadow": "none"
}
}
},
"type": "object"
},
"ScenarioDescriptor": {
"properties": {
"author_version": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"behavior_sha256": {
"type": [
"string",
"null"
]
},
"case_id": {
"type": "string"
},
"characterization": {
"$ref": "#/definitions/ScenarioCharacterization"
},
"classification": {
"$ref": "#/definitions/ComplexityClassification"
},
"compiled_sha256": {
"type": [
"string",
"null"
]
},
"complexity": {
"$ref": "#/definitions/ComplexityProfile"
},
"contract_sha256": {
"type": "string"
},
"deliverable_contract": {
"$ref": "#/definitions/DeliverableContract"
},
"inputs_sha256": {
"type": "string"
},
"origin": {
"$ref": "#/definitions/ScenarioOrigin"
},
"required_capabilities": {
"items": {
"type": "string"
},
"type": "array"
},
"resource_envelope": {
"$ref": "#/definitions/ScenarioResourceEnvelope"
},
"scenario_id": {
"$ref": "#/definitions/ScenarioKey"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"source_path": {
"type": [
"string",
"null"
]
},
"source_sha256": {
"type": [
"string",
"null"
]
},
"title": {
"type": [
"string",
"null"
]
}
},
"required": [
"case_id",
"characterization",
"classification",
"complexity",
"contract_sha256",
"deliverable_contract",
"inputs_sha256",
"origin",
"required_capabilities",
"resource_envelope",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"ScenarioKey": {
"type": "string"
},
"ScenarioOrigin": {
"enum": [
"built_in",
"markdown",
"local"
],
"type": "string"
},
"ScenarioRealism": {
"properties": {
"execution": {
"allOf": [
{
"$ref": "#/definitions/ExecutionRealism"
}
],
"default": "synthetic"
},
"shadow": {
"allOf": [
{
"$ref": "#/definitions/ShadowMode"
}
],
"default": "none"
}
},
"type": "object"
},
"ScenarioResourceEnvelope": {
"properties": {
"execution": {
"$ref": "#/definitions/ExecutionPolicy"
},
"workflow": {
"anyOf": [
{
"$ref": "#/definitions/WorkflowResourceEnvelope"
},
{
"type": "null"
}
]
}
},
"required": [
"execution"
],
"type": "object"
},
"ShadowMode": {
"enum": [
"none",
"read_only"
],
"type": "string"
},
"WorkflowResourceEnvelope": {
"properties": {
"max_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"max_nodes": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"max_parallel": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"max_total_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"step_timeout_seconds": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"workflow_timeout_seconds": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"max_nodes",
"max_parallel",
"step_timeout_seconds",
"technical_retries",
"workflow_timeout_seconds"
],
"type": "object"
}
},
"properties": {
"scenario": {
"$ref": "#/definitions/ScenarioDescriptor"
}
},
"required": [
"scenario"
],
"title": "LocalScenarioCreateResponse",
"type": "object"
}
},
{
"description": "Configure, export and execute saved plans, and inspect or cancel their composed executions.",
"metadata": {
"contract": {
"capability": "plan-control",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::plan-control",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ReferenceModel": {
"additionalProperties": false,
"properties": {
"model": {
"type": "string"
},
"provider": {
"type": "string"
}
},
"required": [
"model",
"provider"
],
"type": "object"
}
},
"oneOf": [
{
"properties": {
"action": {
"enum": [
"import_reference"
],
"type": "string"
},
"judge": {
"anyOf": [
{
"$ref": "#/definitions/ReferenceModel"
},
{
"type": "null"
}
]
},
"label": {
"default": "",
"type": "string"
},
"materialized": true,
"reference_execution_id": {
"type": "string"
},
"shards": {
"default": [],
"items": true,
"type": "array"
},
"subject": {
"$ref": "#/definitions/ReferenceModel"
}
},
"required": [
"action",
"materialized",
"reference_execution_id",
"subject"
],
"type": "object"
},
{
"properties": {
"action": {
"enum": [
"requirements"
],
"type": "string"
},
"plan_id": {
"type": "string"
}
},
"required": [
"action",
"plan_id"
],
"type": "object"
},
{
"properties": {
"action": {
"enum": [
"export"
],
"type": "string"
},
"plan_id": {
"type": "string"
}
},
"required": [
"action",
"plan_id"
],
"type": "object"
},
{
"properties": {
"action": {
"enum": [
"execution"
],
"type": "string"
},
"execution_id": {
"type": "string"
}
},
"required": [
"action",
"execution_id"
],
"type": "object"
},
{
"properties": {
"action": {
"enum": [
"cancel"
],
"type": "string"
},
"execution_id": {
"type": "string"
}
},
"required": [
"action",
"execution_id"
],
"type": "object"
}
],
"title": "Request"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": true,
"title": "Map_of_AnyValue",
"type": "object"
}
},
{
"description": "Create a draft local plan with an explicit small test scope.",
"metadata": {
"contract": {
"capability": "plan-create",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::plan-create",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"duplicate_of": {
"default": null,
"type": [
"string",
"null"
]
},
"judge_model": {
"default": "",
"type": "string"
},
"judge_provider": {
"default": "",
"type": "string"
},
"label": {
"default": "",
"type": "string"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"purpose": {
"default": "",
"type": "string"
},
"runs": {
"default": 1,
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenarios": {
"items": {
"type": "string"
},
"type": "array"
},
"seed": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"default": 1,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"template_id": {
"default": null,
"type": [
"string",
"null"
]
},
"url": {
"type": "string"
}
},
"required": [
"model",
"provider",
"scenarios",
"url"
],
"title": "PlanCreateRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"PlanScopeItem": {
"properties": {
"case_id": {
"type": "string"
},
"complexity_tier": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"inputs_sha256": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"case_id",
"complexity_tier",
"contract_sha256",
"inputs_sha256",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"PlanState": {
"enum": [
"draft",
"baseline_running",
"baseline_ready",
"candidate_running",
"comparison_ready"
],
"type": "string"
}
},
"properties": {
"baseline_execution_id": {
"type": [
"string",
"null"
]
},
"candidate_execution_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"candidate_labels": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"compatible": {
"type": "boolean"
},
"created_at": {
"type": "string"
},
"id": {
"type": "string"
},
"incomplete_execution_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"judge_model": {
"type": "string"
},
"judge_provider": {
"type": "string"
},
"label": {
"type": "string"
},
"last_attempt_id": {
"type": [
"string",
"null"
]
},
"locked": {
"type": "boolean"
},
"model": {
"type": "string"
},
"protected_executor_required": {
"type": "boolean"
},
"provider": {
"type": "string"
},
"purpose": {
"type": "string"
},
"reference_differences": {
"items": {
"type": "string"
},
"type": "array"
},
"reference_execution_id": {
"type": [
"string",
"null"
]
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenario_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"scenarios": {
"items": {
"$ref": "#/definitions/PlanScopeItem"
},
"type": "array"
},
"schema_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scope_hash": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"state": {
"$ref": "#/definitions/PlanState"
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"template_id": {
"type": [
"string",
"null"
]
},
"updated_at": {
"type": "string"
},
"url": {
"type": "string"
}
},
"required": [
"candidate_execution_ids",
"candidate_labels",
"compatible",
"created_at",
"id",
"incomplete_execution_ids",
"judge_model",
"judge_provider",
"label",
"locked",
"model",
"protected_executor_required",
"provider",
"purpose",
"runs",
"scenario_ids",
"scenarios",
"schema_version",
"scope_hash",
"state",
"technical_retries",
"updated_at",
"url"
],
"title": "LocalPlan",
"type": "object"
}
},
{
"description": "Read one local plan.",
"metadata": {
"contract": {
"capability": "plan-get",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::plan-get",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"plan_id": {
"type": "string"
}
},
"required": [
"plan_id"
],
"title": "PlanGetRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"PlanScopeItem": {
"properties": {
"case_id": {
"type": "string"
},
"complexity_tier": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"inputs_sha256": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"case_id",
"complexity_tier",
"contract_sha256",
"inputs_sha256",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"PlanState": {
"enum": [
"draft",
"baseline_running",
"baseline_ready",
"candidate_running",
"comparison_ready"
],
"type": "string"
}
},
"properties": {
"baseline_execution_id": {
"type": [
"string",
"null"
]
},
"candidate_execution_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"candidate_labels": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"compatible": {
"type": "boolean"
},
"created_at": {
"type": "string"
},
"id": {
"type": "string"
},
"incomplete_execution_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"judge_model": {
"type": "string"
},
"judge_provider": {
"type": "string"
},
"label": {
"type": "string"
},
"last_attempt_id": {
"type": [
"string",
"null"
]
},
"locked": {
"type": "boolean"
},
"model": {
"type": "string"
},
"protected_executor_required": {
"type": "boolean"
},
"provider": {
"type": "string"
},
"purpose": {
"type": "string"
},
"reference_differences": {
"items": {
"type": "string"
},
"type": "array"
},
"reference_execution_id": {
"type": [
"string",
"null"
]
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenario_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"scenarios": {
"items": {
"$ref": "#/definitions/PlanScopeItem"
},
"type": "array"
},
"schema_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scope_hash": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"state": {
"$ref": "#/definitions/PlanState"
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"template_id": {
"type": [
"string",
"null"
]
},
"updated_at": {
"type": "string"
},
"url": {
"type": "string"
}
},
"required": [
"candidate_execution_ids",
"candidate_labels",
"compatible",
"created_at",
"id",
"incomplete_execution_ids",
"judge_model",
"judge_provider",
"label",
"locked",
"model",
"protected_executor_required",
"provider",
"purpose",
"runs",
"scenario_ids",
"scenarios",
"schema_version",
"scope_hash",
"state",
"technical_retries",
"updated_at",
"url"
],
"title": "LocalPlan",
"type": "object"
}
},
{
"description": "Start a baseline or candidate from a locked local plan.",
"metadata": {
"contract": {
"capability": "plan-run-start",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::plan-run-start",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"PlanRunRole": {
"enum": [
"baseline",
"candidate"
],
"type": "string"
}
},
"properties": {
"idempotency_key": {
"type": "string"
},
"plan_id": {
"default": null,
"type": [
"string",
"null"
]
},
"role": {
"$ref": "#/definitions/PlanRunRole"
}
},
"required": [
"idempotency_key",
"role"
],
"title": "PlanRunRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"PlanScopeItem": {
"properties": {
"case_id": {
"type": "string"
},
"complexity_tier": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"inputs_sha256": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"case_id",
"complexity_tier",
"contract_sha256",
"inputs_sha256",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"PlanState": {
"enum": [
"draft",
"baseline_running",
"baseline_ready",
"candidate_running",
"comparison_ready"
],
"type": "string"
}
},
"properties": {
"baseline_execution_id": {
"type": [
"string",
"null"
]
},
"candidate_execution_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"candidate_labels": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"compatible": {
"type": "boolean"
},
"created_at": {
"type": "string"
},
"id": {
"type": "string"
},
"incomplete_execution_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"judge_model": {
"type": "string"
},
"judge_provider": {
"type": "string"
},
"label": {
"type": "string"
},
"last_attempt_id": {
"type": [
"string",
"null"
]
},
"locked": {
"type": "boolean"
},
"model": {
"type": "string"
},
"protected_executor_required": {
"type": "boolean"
},
"provider": {
"type": "string"
},
"purpose": {
"type": "string"
},
"reference_differences": {
"items": {
"type": "string"
},
"type": "array"
},
"reference_execution_id": {
"type": [
"string",
"null"
]
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenario_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"scenarios": {
"items": {
"$ref": "#/definitions/PlanScopeItem"
},
"type": "array"
},
"schema_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scope_hash": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"state": {
"$ref": "#/definitions/PlanState"
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"template_id": {
"type": [
"string",
"null"
]
},
"updated_at": {
"type": "string"
},
"url": {
"type": "string"
}
},
"required": [
"candidate_execution_ids",
"candidate_labels",
"compatible",
"created_at",
"id",
"incomplete_execution_ids",
"judge_model",
"judge_provider",
"label",
"locked",
"model",
"protected_executor_required",
"provider",
"purpose",
"runs",
"scenario_ids",
"scenarios",
"schema_version",
"scope_hash",
"state",
"technical_retries",
"updated_at",
"url"
],
"title": "LocalPlan",
"type": "object"
}
},
{
"description": "Update an unlocked local plan or rename retained candidates.",
"metadata": {
"contract": {
"capability": "plan-update",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::plan-update",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"candidate_labels": {
"additionalProperties": {
"type": "string"
},
"type": [
"object",
"null"
]
},
"judge_model": {
"type": [
"string",
"null"
]
},
"judge_provider": {
"type": [
"string",
"null"
]
},
"label": {
"type": [
"string",
"null"
]
},
"model": {
"type": [
"string",
"null"
]
},
"plan_id": {
"default": null,
"type": [
"string",
"null"
]
},
"provider": {
"type": [
"string",
"null"
]
},
"purpose": {
"type": [
"string",
"null"
]
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"scenarios": {
"items": {
"type": "string"
},
"type": [
"array",
"null"
]
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"url": {
"type": [
"string",
"null"
]
}
},
"title": "PlanUpdateRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"PlanScopeItem": {
"properties": {
"case_id": {
"type": "string"
},
"complexity_tier": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"inputs_sha256": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"case_id",
"complexity_tier",
"contract_sha256",
"inputs_sha256",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"PlanState": {
"enum": [
"draft",
"baseline_running",
"baseline_ready",
"candidate_running",
"comparison_ready"
],
"type": "string"
}
},
"properties": {
"baseline_execution_id": {
"type": [
"string",
"null"
]
},
"candidate_execution_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"candidate_labels": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"compatible": {
"type": "boolean"
},
"created_at": {
"type": "string"
},
"id": {
"type": "string"
},
"incomplete_execution_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"judge_model": {
"type": "string"
},
"judge_provider": {
"type": "string"
},
"label": {
"type": "string"
},
"last_attempt_id": {
"type": [
"string",
"null"
]
},
"locked": {
"type": "boolean"
},
"model": {
"type": "string"
},
"protected_executor_required": {
"type": "boolean"
},
"provider": {
"type": "string"
},
"purpose": {
"type": "string"
},
"reference_differences": {
"items": {
"type": "string"
},
"type": "array"
},
"reference_execution_id": {
"type": [
"string",
"null"
]
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenario_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"scenarios": {
"items": {
"$ref": "#/definitions/PlanScopeItem"
},
"type": "array"
},
"schema_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scope_hash": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"state": {
"$ref": "#/definitions/PlanState"
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"template_id": {
"type": [
"string",
"null"
]
},
"updated_at": {
"type": "string"
},
"url": {
"type": "string"
}
},
"required": [
"candidate_execution_ids",
"candidate_labels",
"compatible",
"created_at",
"id",
"incomplete_execution_ids",
"judge_model",
"judge_provider",
"label",
"locked",
"model",
"protected_executor_required",
"provider",
"purpose",
"runs",
"scenario_ids",
"scenarios",
"schema_version",
"scope_hash",
"state",
"technical_retries",
"updated_at",
"url"
],
"title": "LocalPlan",
"type": "object"
}
},
{
"description": "List local plans and their baseline/candidate lifecycle.",
"metadata": {
"contract": {
"capability": "plans-list",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::plans-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "DashboardEmptyRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"LocalPlan": {
"properties": {
"baseline_execution_id": {
"type": [
"string",
"null"
]
},
"candidate_execution_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"candidate_labels": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"compatible": {
"type": "boolean"
},
"created_at": {
"type": "string"
},
"id": {
"type": "string"
},
"incomplete_execution_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"judge_model": {
"type": "string"
},
"judge_provider": {
"type": "string"
},
"label": {
"type": "string"
},
"last_attempt_id": {
"type": [
"string",
"null"
]
},
"locked": {
"type": "boolean"
},
"model": {
"type": "string"
},
"protected_executor_required": {
"type": "boolean"
},
"provider": {
"type": "string"
},
"purpose": {
"type": "string"
},
"reference_differences": {
"items": {
"type": "string"
},
"type": "array"
},
"reference_execution_id": {
"type": [
"string",
"null"
]
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenario_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"scenarios": {
"items": {
"$ref": "#/definitions/PlanScopeItem"
},
"type": "array"
},
"schema_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scope_hash": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"state": {
"$ref": "#/definitions/PlanState"
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"template_id": {
"type": [
"string",
"null"
]
},
"updated_at": {
"type": "string"
},
"url": {
"type": "string"
}
},
"required": [
"candidate_execution_ids",
"candidate_labels",
"compatible",
"created_at",
"id",
"incomplete_execution_ids",
"judge_model",
"judge_provider",
"label",
"locked",
"model",
"protected_executor_required",
"provider",
"purpose",
"runs",
"scenario_ids",
"scenarios",
"schema_version",
"scope_hash",
"state",
"technical_retries",
"updated_at",
"url"
],
"type": "object"
},
"PlanScopeItem": {
"properties": {
"case_id": {
"type": "string"
},
"complexity_tier": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"inputs_sha256": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"case_id",
"complexity_tier",
"contract_sha256",
"inputs_sha256",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"PlanState": {
"enum": [
"draft",
"baseline_running",
"baseline_ready",
"candidate_running",
"comparison_ready"
],
"type": "string"
}
},
"properties": {
"master_plan": true,
"mode": {
"type": "string"
},
"plans": {
"items": {
"$ref": "#/definitions/LocalPlan"
},
"type": "array"
}
},
"required": [
"master_plan",
"mode",
"plans"
],
"title": "PlansListResponse",
"type": "object"
}
},
{
"description": "Cancel the active local E2E execution.",
"metadata": {
"contract": {
"capability": "run-cancel",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::run-cancel",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "DashboardEmptyRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"Defaults": {
"properties": {
"judge_model": {
"type": "string"
},
"judge_provider": {
"type": "string"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"url": {
"type": "string"
}
},
"required": [
"judge_model",
"judge_provider",
"model",
"provider",
"runs",
"technical_retries",
"url"
],
"type": "object"
},
"JobStatus": {
"enum": [
"running",
"cancelling",
"cancelled",
"completed",
"failed"
],
"type": "string"
},
"JobView": {
"properties": {
"completed_at": {
"type": "string"
},
"error": {
"type": "string"
},
"id": {
"type": "string"
},
"label": {
"type": "string"
},
"log": {
"type": "string"
},
"log_from": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"log_offset": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"log_truncated": {
"type": "boolean"
},
"request": {
"$ref": "#/definitions/RunRequest"
},
"returncode": {
"format": "int32",
"type": [
"integer",
"null"
]
},
"started_at": {
"type": "string"
},
"status": {
"$ref": "#/definitions/JobStatus"
}
},
"required": [
"completed_at",
"error",
"id",
"label",
"log",
"log_from",
"log_offset",
"log_truncated",
"request",
"started_at",
"status"
],
"type": "object"
},
"RunRequest": {
"additionalProperties": false,
"properties": {
"judge_model": {
"default": "",
"type": "string"
},
"judge_provider": {
"default": "",
"type": "string"
},
"label": {
"default": "",
"type": "string"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenarios": {
"items": {
"type": "string"
},
"type": "array"
},
"seed": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"url": {
"type": "string"
}
},
"required": [
"model",
"provider",
"runs",
"scenarios",
"technical_retries",
"url"
],
"type": "object"
}
},
"properties": {
"defaults": {
"$ref": "#/definitions/Defaults"
},
"job": {
"anyOf": [
{
"$ref": "#/definitions/JobView"
},
{
"type": "null"
}
]
}
},
"required": [
"defaults"
],
"title": "RunSnapshot",
"type": "object"
}
},
{
"description": "Start one local E2E execution.",
"metadata": {
"contract": {
"capability": "run-start",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::run-start",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"judge_model": {
"default": "",
"type": "string"
},
"judge_provider": {
"default": "",
"type": "string"
},
"label": {
"default": "",
"type": "string"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenarios": {
"items": {
"type": "string"
},
"type": "array"
},
"seed": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"url": {
"type": "string"
}
},
"required": [
"model",
"provider",
"runs",
"scenarios",
"technical_retries",
"url"
],
"title": "RunRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"Defaults": {
"properties": {
"judge_model": {
"type": "string"
},
"judge_provider": {
"type": "string"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"url": {
"type": "string"
}
},
"required": [
"judge_model",
"judge_provider",
"model",
"provider",
"runs",
"technical_retries",
"url"
],
"type": "object"
},
"JobStatus": {
"enum": [
"running",
"cancelling",
"cancelled",
"completed",
"failed"
],
"type": "string"
},
"JobView": {
"properties": {
"completed_at": {
"type": "string"
},
"error": {
"type": "string"
},
"id": {
"type": "string"
},
"label": {
"type": "string"
},
"log": {
"type": "string"
},
"log_from": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"log_offset": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"log_truncated": {
"type": "boolean"
},
"request": {
"$ref": "#/definitions/RunRequest"
},
"returncode": {
"format": "int32",
"type": [
"integer",
"null"
]
},
"started_at": {
"type": "string"
},
"status": {
"$ref": "#/definitions/JobStatus"
}
},
"required": [
"completed_at",
"error",
"id",
"label",
"log",
"log_from",
"log_offset",
"log_truncated",
"request",
"started_at",
"status"
],
"type": "object"
},
"RunRequest": {
"additionalProperties": false,
"properties": {
"judge_model": {
"default": "",
"type": "string"
},
"judge_provider": {
"default": "",
"type": "string"
},
"label": {
"default": "",
"type": "string"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenarios": {
"items": {
"type": "string"
},
"type": "array"
},
"seed": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"url": {
"type": "string"
}
},
"required": [
"model",
"provider",
"runs",
"scenarios",
"technical_retries",
"url"
],
"type": "object"
}
},
"properties": {
"defaults": {
"$ref": "#/definitions/Defaults"
},
"job": {
"anyOf": [
{
"$ref": "#/definitions/JobView"
},
{
"type": "null"
}
]
}
},
"required": [
"defaults"
],
"title": "RunSnapshot",
"type": "object"
}
},
{
"description": "Read local execution state and only the unread log suffix.",
"metadata": {
"contract": {
"capability": "run-status",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::run-status",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"after": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"title": "RunStatusRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"Defaults": {
"properties": {
"judge_model": {
"type": "string"
},
"judge_provider": {
"type": "string"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"url": {
"type": "string"
}
},
"required": [
"judge_model",
"judge_provider",
"model",
"provider",
"runs",
"technical_retries",
"url"
],
"type": "object"
},
"JobStatus": {
"enum": [
"running",
"cancelling",
"cancelled",
"completed",
"failed"
],
"type": "string"
},
"JobView": {
"properties": {
"completed_at": {
"type": "string"
},
"error": {
"type": "string"
},
"id": {
"type": "string"
},
"label": {
"type": "string"
},
"log": {
"type": "string"
},
"log_from": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"log_offset": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"log_truncated": {
"type": "boolean"
},
"request": {
"$ref": "#/definitions/RunRequest"
},
"returncode": {
"format": "int32",
"type": [
"integer",
"null"
]
},
"started_at": {
"type": "string"
},
"status": {
"$ref": "#/definitions/JobStatus"
}
},
"required": [
"completed_at",
"error",
"id",
"label",
"log",
"log_from",
"log_offset",
"log_truncated",
"request",
"started_at",
"status"
],
"type": "object"
},
"RunRequest": {
"additionalProperties": false,
"properties": {
"judge_model": {
"default": "",
"type": "string"
},
"judge_provider": {
"default": "",
"type": "string"
},
"label": {
"default": "",
"type": "string"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenarios": {
"items": {
"type": "string"
},
"type": "array"
},
"seed": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"url": {
"type": "string"
}
},
"required": [
"model",
"provider",
"runs",
"scenarios",
"technical_retries",
"url"
],
"type": "object"
}
},
"properties": {
"defaults": {
"$ref": "#/definitions/Defaults"
},
"job": {
"anyOf": [
{
"$ref": "#/definitions/JobView"
},
{
"type": "null"
}
]
}
},
"required": [
"defaults"
],
"title": "RunSnapshot",
"type": "object"
}
},
{
"description": "Read local metric history for one test version, with provider-grouped execution and judge models, without comparison actions.",
"metadata": {
"contract": {
"capability": "test-history-get",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::test-history-get",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"case_id": {
"default": null,
"type": [
"string",
"null"
]
},
"cursor": {
"default": null,
"type": [
"string",
"null"
]
},
"judge_model": {
"default": null,
"type": [
"string",
"null"
]
},
"judge_provider": {
"default": null,
"type": [
"string",
"null"
]
},
"limit": {
"default": null,
"format": "uint16",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"result": {
"default": null,
"type": [
"string",
"null"
]
},
"subject_model": {
"default": null,
"type": [
"string",
"null"
]
},
"subject_provider": {
"default": null,
"type": [
"string",
"null"
]
},
"system_version_id": {
"default": null,
"type": [
"string",
"null"
]
},
"test_id": {
"default": "",
"type": "string"
},
"test_version": {
"default": null,
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"title": "TestHistoryRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"AssessmentOutcomeCounts": {
"properties": {
"error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"partial": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"error",
"failed",
"not_evaluated",
"partial",
"passed",
"unavailable"
],
"type": "object"
},
"AssessmentSummary": {
"description": "Objective assessment totals for a set of runs. Every count is derived from deterministic evidence: hard gates, deterministic criteria, and asset validation.",
"properties": {
"assessment_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"assessment_outcomes": {
"$ref": "#/definitions/AssessmentOutcomeCounts"
},
"asset_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"asset_validation_outcomes": {
"$ref": "#/definitions/AssetValidationCounts"
},
"evidence_reference_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"system_statuses": {
"$ref": "#/definitions/StatusCounts"
}
},
"required": [
"assessment_count",
"assessment_outcomes",
"asset_count",
"asset_validation_outcomes",
"evidence_reference_count",
"run_count",
"system_statuses"
],
"type": "object"
},
"AssetValidationCounts": {
"properties": {
"invalid": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"malformed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_produced": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"oversized": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"removed_during_cleanup": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unexpected": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unreadable": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unsafe_path": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"valid": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"invalid",
"malformed",
"not_evaluated",
"not_produced",
"oversized",
"removed_during_cleanup",
"unexpected",
"unreadable",
"unsafe_path",
"valid"
],
"type": "object"
},
"HistoryModelGroup": {
"description": "Models are exposed as provider groups so a model name is never ambiguous when two providers offer the same model identifier.",
"properties": {
"models": {
"items": {
"type": "string"
},
"type": "array"
},
"provider": {
"type": "string"
}
},
"required": [
"models",
"provider"
],
"type": "object"
},
"HistorySeries": {
"properties": {
"assessment_profile_sha256": {
"type": "string"
},
"case_id": {
"type": "string"
},
"cohort_id": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"engine_revision": {
"type": [
"string",
"null"
]
},
"execution_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"harness_revision": {
"type": [
"string",
"null"
]
},
"id": {
"type": "string"
},
"judge_model": {
"type": [
"string",
"null"
]
},
"judge_provider": {
"type": [
"string",
"null"
]
},
"median_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_call_errors": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_turns": {
"format": "double",
"type": [
"number",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stack_mode": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"system_label": {
"type": "string"
},
"system_revision": {
"type": [
"string",
"null"
]
},
"system_version_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"assessment_profile_sha256",
"case_id",
"cohort_id",
"contract_sha256",
"execution_count",
"id",
"run_count",
"scenario_version",
"stack_mode",
"subject_model",
"subject_provider",
"system_label"
],
"type": "object"
},
"HistorySystem": {
"properties": {
"id": {
"type": "string"
},
"label": {
"type": "string"
}
},
"required": [
"id",
"label"
],
"type": "object"
},
"StatusCounts": {
"properties": {
"hard_gate_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"infrastructure_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"judge_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"resource_limit": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"subject_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"hard_gate_failed",
"infrastructure_error",
"judge_error",
"passed",
"resource_limit",
"subject_error",
"unavailable"
],
"type": "object"
},
"TestObservation": {
"properties": {
"assessment_profile_sha256": {
"type": "string"
},
"assessment_summary": {
"$ref": "#/definitions/AssessmentSummary"
},
"case_id": {
"type": "string"
},
"cohort_id": {
"type": "string"
},
"completed_at": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"engine_revision": {
"type": [
"string",
"null"
]
},
"evaluated_version_id": {
"type": [
"string",
"null"
]
},
"execution_id": {
"type": "string"
},
"harness_revision": {
"type": [
"string",
"null"
]
},
"judge_model": {
"type": [
"string",
"null"
]
},
"judge_provider": {
"type": [
"string",
"null"
]
},
"median_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_call_errors": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_turns": {
"format": "double",
"type": [
"number",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scored_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stack_mode": {
"type": "string"
},
"status": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"system_label": {
"type": "string"
},
"system_revision": {
"type": [
"string",
"null"
]
},
"system_version_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"assessment_profile_sha256",
"assessment_summary",
"case_id",
"cohort_id",
"completed_at",
"contract_sha256",
"execution_id",
"run_count",
"scenario_version",
"scored_runs",
"stack_mode",
"status",
"subject_model",
"subject_provider",
"system_label"
],
"type": "object"
},
"VersionDescriptor": {
"properties": {
"execution_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"last_seen": {
"type": [
"string",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"execution_count",
"run_count",
"version"
],
"type": "object"
}
},
"properties": {
"available_versions": {
"items": {
"$ref": "#/definitions/VersionDescriptor"
},
"type": "array"
},
"cases": {
"items": {
"type": "string"
},
"type": "array"
},
"current_version": {
"description": "The contract's current version, which may have no executions yet.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"judge_models": {
"items": {
"$ref": "#/definitions/HistoryModelGroup"
},
"type": "array"
},
"next_cursor": {
"type": [
"string",
"null"
]
},
"observations": {
"items": {
"$ref": "#/definitions/TestObservation"
},
"type": "array"
},
"series": {
"items": {
"$ref": "#/definitions/HistorySeries"
},
"type": "array"
},
"subject_models": {
"items": {
"$ref": "#/definitions/HistoryModelGroup"
},
"type": "array"
},
"subjects": {
"items": {
"type": "string"
},
"type": "array"
},
"systems": {
"items": {
"$ref": "#/definitions/HistorySystem"
},
"type": "array"
},
"test_id": {
"type": "string"
},
"test_version": {
"description": "The version whose executions are shown (the latest with evidence by default).",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"total": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"available_versions",
"cases",
"judge_models",
"observations",
"series",
"subject_models",
"subjects",
"systems",
"test_id",
"test_version",
"total"
],
"title": "TestHistoryResponse",
"type": "object"
}
},
{
"description": "Read one test version across two immutable evaluated-system versions.",
"metadata": {
"contract": {
"capability": "test-version-get",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::test-version-get",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"cohort_id": {
"type": "string"
},
"from_version_id": {
"type": "string"
},
"test_id": {
"type": "string"
},
"test_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"to_version_id": {
"type": "string"
}
},
"required": [
"cohort_id",
"from_version_id",
"test_id",
"test_version",
"to_version_id"
],
"title": "TestVersionGetRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"AssessmentOutcomeCounts": {
"properties": {
"error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"partial": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"error",
"failed",
"not_evaluated",
"partial",
"passed",
"unavailable"
],
"type": "object"
},
"AssessmentSummary": {
"description": "Objective assessment totals for a set of runs. Every count is derived from deterministic evidence: hard gates, deterministic criteria, and asset validation.",
"properties": {
"assessment_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"assessment_outcomes": {
"$ref": "#/definitions/AssessmentOutcomeCounts"
},
"asset_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"asset_validation_outcomes": {
"$ref": "#/definitions/AssetValidationCounts"
},
"evidence_reference_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"system_statuses": {
"$ref": "#/definitions/StatusCounts"
}
},
"required": [
"assessment_count",
"assessment_outcomes",
"asset_count",
"asset_validation_outcomes",
"evidence_reference_count",
"run_count",
"system_statuses"
],
"type": "object"
},
"AssetValidationCounts": {
"properties": {
"invalid": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"malformed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_produced": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"oversized": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"removed_during_cleanup": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unexpected": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unreadable": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unsafe_path": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"valid": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"invalid",
"malformed",
"not_evaluated",
"not_produced",
"oversized",
"removed_during_cleanup",
"unexpected",
"unreadable",
"unsafe_path",
"valid"
],
"type": "object"
},
"MetricSamples": {
"properties": {
"cost_usd": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"duration_seconds": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"score": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"tokens": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"turns": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"cost_usd",
"duration_seconds",
"score",
"tokens",
"turns"
],
"type": "object"
},
"OutcomeCounts": {
"properties": {
"hard_gate_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"infra_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"technical_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"hard_gate_failed",
"infra_failed",
"passed",
"technical_failed"
],
"type": "object"
},
"StatusCounts": {
"properties": {
"hard_gate_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"infrastructure_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"judge_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"resource_limit": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"subject_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"hard_gate_failed",
"infrastructure_error",
"judge_error",
"passed",
"resource_limit",
"subject_error",
"unavailable"
],
"type": "object"
},
"TestDelta": {
"properties": {
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"score": {
"format": "double",
"type": [
"number",
"null"
]
},
"tokens": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"type": "object"
},
"TestObservation": {
"properties": {
"assessment_profile_sha256": {
"type": "string"
},
"assessment_summary": {
"$ref": "#/definitions/AssessmentSummary"
},
"case_id": {
"type": "string"
},
"cohort_id": {
"type": "string"
},
"completed_at": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"engine_revision": {
"type": [
"string",
"null"
]
},
"evaluated_version_id": {
"type": [
"string",
"null"
]
},
"execution_id": {
"type": "string"
},
"harness_revision": {
"type": [
"string",
"null"
]
},
"judge_model": {
"type": [
"string",
"null"
]
},
"judge_provider": {
"type": [
"string",
"null"
]
},
"median_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_call_errors": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_turns": {
"format": "double",
"type": [
"number",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scored_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stack_mode": {
"type": "string"
},
"status": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"system_label": {
"type": "string"
},
"system_revision": {
"type": [
"string",
"null"
]
},
"system_version_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"assessment_profile_sha256",
"assessment_summary",
"case_id",
"cohort_id",
"completed_at",
"contract_sha256",
"execution_id",
"run_count",
"scenario_version",
"scored_runs",
"stack_mode",
"status",
"subject_model",
"subject_provider",
"system_label"
],
"type": "object"
},
"TestSideSummary": {
"properties": {
"assessment_summary": {
"$ref": "#/definitions/AssessmentSummary"
},
"case_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"evaluated_version_id": {
"type": "string"
},
"execution_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"median_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"outcomes": {
"$ref": "#/definitions/OutcomeCounts"
},
"pass_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"samples": {
"$ref": "#/definitions/MetricSamples"
},
"scored_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"total_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"assessment_summary",
"case_count",
"evaluated_version_id",
"execution_count",
"outcomes",
"samples",
"scored_runs",
"total_runs"
],
"type": "object"
}
},
"properties": {
"compatibility": {
"type": "string"
},
"compatibility_reasons": {
"items": {
"type": "string"
},
"type": "array"
},
"delta": {
"$ref": "#/definitions/TestDelta"
},
"from": {
"anyOf": [
{
"$ref": "#/definitions/TestSideSummary"
},
{
"type": "null"
}
]
},
"from_observations": {
"items": {
"$ref": "#/definitions/TestObservation"
},
"type": "array"
},
"test_id": {
"type": "string"
},
"test_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"to": {
"anyOf": [
{
"$ref": "#/definitions/TestSideSummary"
},
{
"type": "null"
}
]
},
"to_observations": {
"items": {
"$ref": "#/definitions/TestObservation"
},
"type": "array"
}
},
"required": [
"compatibility",
"compatibility_reasons",
"delta",
"from_observations",
"test_id",
"test_version",
"to_observations"
],
"title": "TestVersionResult",
"type": "object"
}
},
{
"description": "List the versioned test catalog with compact A/B results and stable pagination.",
"metadata": {
"contract": {
"capability": "tests-list",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::tests-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"cohort_id": {
"default": null,
"type": [
"string",
"null"
]
},
"cursor": {
"default": null,
"type": [
"string",
"null"
]
},
"from_version_id": {
"default": null,
"type": [
"string",
"null"
]
},
"limit": {
"default": null,
"format": "uint16",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"query": {
"default": null,
"type": [
"string",
"null"
]
},
"to_version_id": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "TestsListRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"AssessmentKind": {
"enum": [
"required_check",
"signal",
"asset_validation"
],
"type": "string"
},
"AssessmentOutcomeCounts": {
"properties": {
"error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"partial": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"error",
"failed",
"not_evaluated",
"partial",
"passed",
"unavailable"
],
"type": "object"
},
"AssessmentPolicy": {
"enum": [
"hard_gate",
"advisory"
],
"type": "string"
},
"AssessmentSummary": {
"description": "Objective assessment totals for a set of runs. Every count is derived from deterministic evidence: hard gates, deterministic criteria, and asset validation.",
"properties": {
"assessment_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"assessment_outcomes": {
"$ref": "#/definitions/AssessmentOutcomeCounts"
},
"asset_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"asset_validation_outcomes": {
"$ref": "#/definitions/AssetValidationCounts"
},
"evidence_reference_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"system_statuses": {
"$ref": "#/definitions/StatusCounts"
}
},
"required": [
"assessment_count",
"assessment_outcomes",
"asset_count",
"asset_validation_outcomes",
"evidence_reference_count",
"run_count",
"system_statuses"
],
"type": "object"
},
"AssetValidationCounts": {
"properties": {
"invalid": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"malformed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_produced": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"oversized": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"removed_during_cleanup": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unexpected": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unreadable": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unsafe_path": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"valid": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"invalid",
"malformed",
"not_evaluated",
"not_produced",
"oversized",
"removed_during_cleanup",
"unexpected",
"unreadable",
"unsafe_path",
"valid"
],
"type": "object"
},
"CalibrationProjection": {
"properties": {
"compatible_sample_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"maturity": {
"type": "string"
}
},
"required": [
"compatible_sample_count",
"maturity"
],
"type": "object"
},
"ComplexityClassification": {
"properties": {
"method": {
"$ref": "#/definitions/ComplexityMethod"
},
"profile": {
"$ref": "#/definitions/ComplexityProfile"
},
"tier": {
"$ref": "#/definitions/ComplexityTier"
}
},
"required": [
"method",
"profile",
"tier"
],
"type": "object"
},
"ComplexityMethod": {
"enum": [
"capability_v2"
],
"type": "string"
},
"ComplexityProfile": {
"properties": {
"agent_owned_decomposition": {
"default": false,
"type": "boolean"
},
"ambiguity_level": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"artifact_count": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"coherent_long_horizon": {
"default": false,
"type": "boolean"
},
"compensable_mutations": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"coordination_edges": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"dependency_depth": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"durable_resume_cycles": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"external_systems": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"material_invalidation_events": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"parallel_branches": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"planning_depth": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"replan_loops": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"state_transitions": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"validation_loops": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"wake_cycles": {
"format": "uint8",
"minimum": 0,
"type": "integer"
}
},
"required": [
"ambiguity_level",
"artifact_count",
"coordination_edges",
"dependency_depth",
"external_systems",
"parallel_branches",
"planning_depth",
"state_transitions",
"validation_loops",
"wake_cycles"
],
"type": "object"
},
"ComplexityTier": {
"enum": [
"l0_atomic",
"l1_sequential",
"l2_stateful",
"l3_concurrent",
"l4_coordinated",
"l5_adaptive"
],
"type": "string"
},
"EvaluationDimension": {
"enum": [
"deliverable",
"structural_integrity",
"efficiency",
"robustness",
"e2e_infrastructure"
],
"type": "string"
},
"ExecutionPolicy": {
"properties": {
"max_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_total_tokens": {
"description": "Shared Harness token budget. `None` leaves the Harness budget unbounded for this scenario.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_turns": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"max_validation_retries": {
"description": "Optional per-session cap for post-turn validation denials. `None` preserves the Harness-configured default.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stuck_timeout_seconds": {
"description": "Stop only after this many seconds without observable useful progress. Large scenarios have no fixed wall-clock deadline.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"max_turns",
"stuck_timeout_seconds"
],
"type": "object"
},
"ExecutionRealism": {
"enum": [
"synthetic",
"realistic_simulator",
"frozen_real_artifact"
],
"type": "string"
},
"HumanHorizon": {
"properties": {
"basis": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizonBasis"
}
],
"default": "unknown"
},
"max_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"min_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"HumanHorizonBasis": {
"enum": [
"unknown",
"author_estimate",
"measured"
],
"type": "string"
},
"MetricSamples": {
"properties": {
"cost_usd": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"duration_seconds": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"score": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"tokens": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"turns": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"cost_usd",
"duration_seconds",
"score",
"tokens",
"turns"
],
"type": "object"
},
"OutcomeCounts": {
"properties": {
"hard_gate_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"infra_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"technical_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"hard_gate_failed",
"infra_failed",
"passed",
"technical_failed"
],
"type": "object"
},
"ScenarioCharacterization": {
"properties": {
"human_horizon": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizon"
}
],
"default": {
"basis": "unknown"
}
},
"realism": {
"allOf": [
{
"$ref": "#/definitions/ScenarioRealism"
}
],
"default": {
"execution": "synthetic",
"shadow": "none"
}
}
},
"type": "object"
},
"ScenarioRealism": {
"properties": {
"execution": {
"allOf": [
{
"$ref": "#/definitions/ExecutionRealism"
}
],
"default": "synthetic"
},
"shadow": {
"allOf": [
{
"$ref": "#/definitions/ShadowMode"
}
],
"default": "none"
}
},
"type": "object"
},
"ShadowMode": {
"enum": [
"none",
"read_only"
],
"type": "string"
},
"StatusCounts": {
"properties": {
"hard_gate_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"infrastructure_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"judge_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"resource_limit": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"subject_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"hard_gate_failed",
"infrastructure_error",
"judge_error",
"passed",
"resource_limit",
"subject_error",
"unavailable"
],
"type": "object"
},
"TestCatalogRow": {
"properties": {
"available_versions": {
"items": {
"$ref": "#/definitions/VersionDescriptor"
},
"type": "array"
},
"calibration": {
"anyOf": [
{
"$ref": "#/definitions/CalibrationProjection"
},
{
"type": "null"
}
]
},
"characterization": {
"anyOf": [
{
"$ref": "#/definitions/ScenarioCharacterization"
},
{
"type": "null"
}
]
},
"complexity": {
"anyOf": [
{
"$ref": "#/definitions/ComplexityClassification"
},
{
"type": "null"
}
]
},
"current_version": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"lifecycle": {
"type": "string"
},
"result": {
"anyOf": [
{
"$ref": "#/definitions/TestVersionResult"
},
{
"type": "null"
}
]
},
"selected_version": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"spec": {
"anyOf": [
{
"$ref": "#/definitions/TestSpecProjection"
},
{
"type": "null"
}
]
},
"test_id": {
"type": "string"
}
},
"required": [
"available_versions",
"lifecycle",
"test_id"
],
"type": "object"
},
"TestCriterionProjection": {
"properties": {
"description": {
"type": "string"
},
"dimension": {
"$ref": "#/definitions/EvaluationDimension"
},
"id": {
"type": "string"
},
"kind": {
"$ref": "#/definitions/AssessmentKind"
},
"policy": {
"$ref": "#/definitions/AssessmentPolicy"
},
"weight": {
"format": "uint8",
"minimum": 0,
"type": "integer"
}
},
"required": [
"description",
"dimension",
"id",
"kind",
"policy",
"weight"
],
"type": "object"
},
"TestDelta": {
"properties": {
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"score": {
"format": "double",
"type": [
"number",
"null"
]
},
"tokens": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"type": "object"
},
"TestObservation": {
"properties": {
"assessment_profile_sha256": {
"type": "string"
},
"assessment_summary": {
"$ref": "#/definitions/AssessmentSummary"
},
"case_id": {
"type": "string"
},
"cohort_id": {
"type": "string"
},
"completed_at": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"engine_revision": {
"type": [
"string",
"null"
]
},
"evaluated_version_id": {
"type": [
"string",
"null"
]
},
"execution_id": {
"type": "string"
},
"harness_revision": {
"type": [
"string",
"null"
]
},
"judge_model": {
"type": [
"string",
"null"
]
},
"judge_provider": {
"type": [
"string",
"null"
]
},
"median_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_call_errors": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_turns": {
"format": "double",
"type": [
"number",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scored_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stack_mode": {
"type": "string"
},
"status": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"system_label": {
"type": "string"
},
"system_revision": {
"type": [
"string",
"null"
]
},
"system_version_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"assessment_profile_sha256",
"assessment_summary",
"case_id",
"cohort_id",
"completed_at",
"contract_sha256",
"execution_id",
"run_count",
"scenario_version",
"scored_runs",
"stack_mode",
"status",
"subject_model",
"subject_provider",
"system_label"
],
"type": "object"
},
"TestSideSummary": {
"properties": {
"assessment_summary": {
"$ref": "#/definitions/AssessmentSummary"
},
"case_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"evaluated_version_id": {
"type": "string"
},
"execution_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"median_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"outcomes": {
"$ref": "#/definitions/OutcomeCounts"
},
"pass_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"samples": {
"$ref": "#/definitions/MetricSamples"
},
"scored_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"total_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"assessment_summary",
"case_count",
"evaluated_version_id",
"execution_count",
"outcomes",
"samples",
"scored_runs",
"total_runs"
],
"type": "object"
},
"TestSpecProjection": {
"description": "The scenario definition as a reader needs it: what the subject is asked to do, how the result is scored, and the limits it runs under. Projected from the materialized `ScenarioSpec` of the test's current version, so it always describes the contract the dashboard is showing.",
"properties": {
"criteria": {
"items": {
"$ref": "#/definitions/TestCriterionProjection"
},
"type": "array"
},
"denied_functions": {
"items": {
"type": "string"
},
"type": "array"
},
"execution": {
"$ref": "#/definitions/ExecutionPolicy"
},
"prompt": {
"description": "The prompt as materialized for a representative run: run-scoped paths resolve against a catalog namespace, never a real execution.",
"type": "string"
},
"summary": {
"description": "Editorial description; absent until the scenario defines a `SUMMARY`.",
"type": [
"string",
"null"
]
}
},
"required": [
"criteria",
"denied_functions",
"execution",
"prompt"
],
"type": "object"
},
"TestVersionResult": {
"properties": {
"compatibility": {
"type": "string"
},
"compatibility_reasons": {
"items": {
"type": "string"
},
"type": "array"
},
"delta": {
"$ref": "#/definitions/TestDelta"
},
"from": {
"anyOf": [
{
"$ref": "#/definitions/TestSideSummary"
},
{
"type": "null"
}
]
},
"from_observations": {
"items": {
"$ref": "#/definitions/TestObservation"
},
"type": "array"
},
"test_id": {
"type": "string"
},
"test_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"to": {
"anyOf": [
{
"$ref": "#/definitions/TestSideSummary"
},
{
"type": "null"
}
]
},
"to_observations": {
"items": {
"$ref": "#/definitions/TestObservation"
},
"type": "array"
}
},
"required": [
"compatibility",
"compatibility_reasons",
"delta",
"from_observations",
"test_id",
"test_version",
"to_observations"
],
"type": "object"
},
"VersionDescriptor": {
"properties": {
"execution_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"last_seen": {
"type": [
"string",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"execution_count",
"run_count",
"version"
],
"type": "object"
}
},
"properties": {
"next_cursor": {
"type": [
"string",
"null"
]
},
"revision": {
"type": "string"
},
"rows": {
"items": {
"$ref": "#/definitions/TestCatalogRow"
},
"type": "array"
},
"total": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"revision",
"rows",
"total"
],
"title": "TestsListResponse",
"type": "object"
}
},
{
"description": "Classify a protected supervisor's fault journal against canonical execution evidence.",
"metadata": {
"contract": {
"capabilities": [
"fault-evaluate"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::fault-evaluate",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"CancellationRule": {
"additionalProperties": false,
"properties": {
"after_action": {
"type": "string"
},
"grace_period_ms": {
"default": 30000,
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"after_action"
],
"type": "object"
},
"DelayRule": {
"additionalProperties": false,
"properties": {
"delay_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"occurrences": {
"default": 1,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"target": {
"type": "string"
}
},
"required": [
"delay_ms",
"target"
],
"type": "object"
},
"DisconnectRule": {
"additionalProperties": false,
"properties": {
"after_action": {
"type": "string"
},
"duration_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"target": {
"type": "string"
}
},
"required": [
"after_action",
"duration_ms",
"target"
],
"type": "object"
},
"ExpectedTerminalOutcome": {
"oneOf": [
{
"enum": [
"recovered",
"cancelled"
],
"type": "string"
},
{
"description": "The subject is expected to finish WITHOUT a correct deliverable: a hard-gate-failed but structurally clean, fully cleaned-up, bounded run. A wrong-but-structural result is the expectation, so it classifies as `correct_recovery`; a passed deliverable means the perturbation failed to bite and is a benchmark infrastructure failure. Degraded profiles must not declare a cancellation rule and require canonical results.",
"enum": [
"degraded"
],
"type": "string"
}
]
},
"FaultAction": {
"additionalProperties": false,
"properties": {
"action_id": {
"type": "string"
},
"duration_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"kind": {
"$ref": "#/definitions/FaultActionKind"
},
"sequence": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"target": {
"type": "string"
},
"trigger_attempt": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"action_id",
"kind",
"sequence",
"target"
],
"type": "object"
},
"FaultActionKind": {
"enum": [
"delay_call",
"fail_first_attempt",
"malformed_result",
"throttle_burst",
"duplicate_event",
"child_timeout",
"transient_disconnect",
"out_of_order_result",
"cancel_execution"
],
"type": "string"
},
"FaultJournal": {
"additionalProperties": false,
"properties": {
"actions": {
"items": {
"$ref": "#/definitions/FaultObservation"
},
"type": "array"
},
"cancellation_observed": {
"type": "boolean"
},
"cleanup_completed": {
"type": "boolean"
},
"cleanup_evidence": {
"additionalProperties": {
"type": "string"
},
"default": {},
"type": "object"
},
"completed_at": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"infrastructure_failures": {
"default": [],
"items": {
"type": "string"
},
"type": "array"
},
"lane": {
"type": "string"
},
"observed_work_amplification": {
"format": "double",
"type": [
"number",
"null"
]
},
"profile_sha256": {
"type": "string"
},
"results_sha256": {
"type": [
"string",
"null"
]
},
"started_at": {
"type": "string"
}
},
"required": [
"actions",
"cancellation_observed",
"cleanup_completed",
"completed_at",
"execution_id",
"lane",
"profile_sha256",
"started_at"
],
"type": "object"
},
"FaultObservation": {
"additionalProperties": false,
"properties": {
"action_id": {
"type": "string"
},
"applied_at": {
"type": [
"string",
"null"
]
},
"detail": {
"default": "",
"type": "string"
},
"kind": {
"$ref": "#/definitions/FaultActionKind"
},
"recovered_at": {
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/FaultObservationStatus"
}
},
"required": [
"action_id",
"kind",
"status"
],
"type": "object"
},
"FaultObservationStatus": {
"enum": [
"recovered",
"unrecovered",
"not_triggered",
"injector_failed"
],
"type": "string"
},
"FaultPlan": {
"additionalProperties": false,
"properties": {
"actions": {
"items": {
"$ref": "#/definitions/FaultAction"
},
"type": "array"
},
"expected_outcome": {
"$ref": "#/definitions/ExpectedTerminalOutcome"
},
"profile_id": {
"type": "string"
},
"profile_sha256": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"actions",
"expected_outcome",
"profile_id",
"profile_sha256",
"seed"
],
"type": "object"
},
"FaultProfile": {
"additionalProperties": false,
"properties": {
"cancellation": {
"anyOf": [
{
"$ref": "#/definitions/CancellationRule"
},
{
"type": "null"
}
],
"default": null
},
"child_timeout_ms": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"delayed_calls": {
"default": [],
"items": {
"$ref": "#/definitions/DelayRule"
},
"type": "array"
},
"duplicate_events": {
"default": false,
"type": "boolean"
},
"expected_outcome": {
"allOf": [
{
"$ref": "#/definitions/ExpectedTerminalOutcome"
}
],
"default": "recovered"
},
"fail_first_attempt": {
"default": [],
"items": {
"type": "string"
},
"type": "array"
},
"malformed_results": {
"default": [],
"description": "The supervisor corrupts the result payload of the first call to each named target: the subject receives undecodable, contract-breaking bytes exactly once per target.",
"items": {
"type": "string"
},
"type": "array"
},
"max_work_amplification": {
"format": "double",
"type": "number"
},
"out_of_order_results": {
"default": false,
"type": "boolean"
},
"profile_id": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"throttle_burst": {
"anyOf": [
{
"$ref": "#/definitions/ThrottleRule"
},
{
"type": "null"
}
],
"default": null
},
"transient_disconnect": {
"anyOf": [
{
"$ref": "#/definitions/DisconnectRule"
},
{
"type": "null"
}
],
"default": null
}
},
"required": [
"max_work_amplification",
"profile_id",
"seed"
],
"type": "object"
},
"ThrottleRule": {
"additionalProperties": false,
"description": "The supervisor answers the target's next `responses` calls with a provider-throttle (429-style) rejection, spaced `spacing_ms` apart.",
"properties": {
"responses": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"spacing_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"target": {
"type": "string"
}
},
"required": [
"responses",
"spacing_ms",
"target"
],
"type": "object"
}
},
"properties": {
"execution_id": {
"type": "string"
},
"journal": {
"$ref": "#/definitions/FaultJournal"
},
"plan": {
"$ref": "#/definitions/FaultPlan"
},
"profile": {
"$ref": "#/definitions/FaultProfile"
}
},
"required": [
"execution_id",
"journal",
"plan",
"profile"
],
"title": "FaultEvaluateRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"RecoveryClassification": {
"enum": [
"correct_recovery",
"excessive_recovery",
"incorrect_result",
"structural_failure",
"infrastructure_failure"
],
"type": "string"
}
},
"properties": {
"cancellation_observed": {
"type": "boolean"
},
"classification": {
"$ref": "#/definitions/RecoveryClassification"
},
"cleanup_completed": {
"type": "boolean"
},
"deliverable_correct": {
"type": [
"boolean",
"null"
]
},
"execution_id": {
"type": "string"
},
"max_work_amplification": {
"format": "double",
"type": "number"
},
"observed_work_amplification": {
"format": "double",
"type": [
"number",
"null"
]
},
"planned_actions": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"profile_id": {
"type": "string"
},
"profile_sha256": {
"type": "string"
},
"reasons": {
"items": {
"type": "string"
},
"type": "array"
},
"recovered_actions": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"results_sha256": {
"type": [
"string",
"null"
]
},
"structural_integrity": {
"type": [
"boolean",
"null"
]
}
},
"required": [
"cancellation_observed",
"classification",
"cleanup_completed",
"execution_id",
"max_work_amplification",
"planned_actions",
"profile_id",
"profile_sha256",
"reasons",
"recovered_actions"
],
"title": "FaultEvaluation",
"type": "object"
}
},
{
"description": "Materialize a deterministic, versioned fault plan for a protected supervisor.",
"metadata": {
"contract": {
"capabilities": [
"fault-plan"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::fault-plan",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"CancellationRule": {
"additionalProperties": false,
"properties": {
"after_action": {
"type": "string"
},
"grace_period_ms": {
"default": 30000,
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"after_action"
],
"type": "object"
},
"DelayRule": {
"additionalProperties": false,
"properties": {
"delay_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"occurrences": {
"default": 1,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"target": {
"type": "string"
}
},
"required": [
"delay_ms",
"target"
],
"type": "object"
},
"DisconnectRule": {
"additionalProperties": false,
"properties": {
"after_action": {
"type": "string"
},
"duration_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"target": {
"type": "string"
}
},
"required": [
"after_action",
"duration_ms",
"target"
],
"type": "object"
},
"ExpectedTerminalOutcome": {
"oneOf": [
{
"enum": [
"recovered",
"cancelled"
],
"type": "string"
},
{
"description": "The subject is expected to finish WITHOUT a correct deliverable: a hard-gate-failed but structurally clean, fully cleaned-up, bounded run. A wrong-but-structural result is the expectation, so it classifies as `correct_recovery`; a passed deliverable means the perturbation failed to bite and is a benchmark infrastructure failure. Degraded profiles must not declare a cancellation rule and require canonical results.",
"enum": [
"degraded"
],
"type": "string"
}
]
},
"FaultProfile": {
"additionalProperties": false,
"properties": {
"cancellation": {
"anyOf": [
{
"$ref": "#/definitions/CancellationRule"
},
{
"type": "null"
}
],
"default": null
},
"child_timeout_ms": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"delayed_calls": {
"default": [],
"items": {
"$ref": "#/definitions/DelayRule"
},
"type": "array"
},
"duplicate_events": {
"default": false,
"type": "boolean"
},
"expected_outcome": {
"allOf": [
{
"$ref": "#/definitions/ExpectedTerminalOutcome"
}
],
"default": "recovered"
},
"fail_first_attempt": {
"default": [],
"items": {
"type": "string"
},
"type": "array"
},
"malformed_results": {
"default": [],
"description": "The supervisor corrupts the result payload of the first call to each named target: the subject receives undecodable, contract-breaking bytes exactly once per target.",
"items": {
"type": "string"
},
"type": "array"
},
"max_work_amplification": {
"format": "double",
"type": "number"
},
"out_of_order_results": {
"default": false,
"type": "boolean"
},
"profile_id": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"throttle_burst": {
"anyOf": [
{
"$ref": "#/definitions/ThrottleRule"
},
{
"type": "null"
}
],
"default": null
},
"transient_disconnect": {
"anyOf": [
{
"$ref": "#/definitions/DisconnectRule"
},
{
"type": "null"
}
],
"default": null
}
},
"required": [
"max_work_amplification",
"profile_id",
"seed"
],
"type": "object"
},
"ThrottleRule": {
"additionalProperties": false,
"description": "The supervisor answers the target's next `responses` calls with a provider-throttle (429-style) rejection, spaced `spacing_ms` apart.",
"properties": {
"responses": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"spacing_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"target": {
"type": "string"
}
},
"required": [
"responses",
"spacing_ms",
"target"
],
"type": "object"
}
},
"properties": {
"profile": {
"$ref": "#/definitions/FaultProfile"
}
},
"required": [
"profile"
],
"title": "FaultPlanRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"ExpectedTerminalOutcome": {
"oneOf": [
{
"enum": [
"recovered",
"cancelled"
],
"type": "string"
},
{
"description": "The subject is expected to finish WITHOUT a correct deliverable: a hard-gate-failed but structurally clean, fully cleaned-up, bounded run. A wrong-but-structural result is the expectation, so it classifies as `correct_recovery`; a passed deliverable means the perturbation failed to bite and is a benchmark infrastructure failure. Degraded profiles must not declare a cancellation rule and require canonical results.",
"enum": [
"degraded"
],
"type": "string"
}
]
},
"FaultAction": {
"additionalProperties": false,
"properties": {
"action_id": {
"type": "string"
},
"duration_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"kind": {
"$ref": "#/definitions/FaultActionKind"
},
"sequence": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"target": {
"type": "string"
},
"trigger_attempt": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"action_id",
"kind",
"sequence",
"target"
],
"type": "object"
},
"FaultActionKind": {
"enum": [
"delay_call",
"fail_first_attempt",
"malformed_result",
"throttle_burst",
"duplicate_event",
"child_timeout",
"transient_disconnect",
"out_of_order_result",
"cancel_execution"
],
"type": "string"
}
},
"properties": {
"actions": {
"items": {
"$ref": "#/definitions/FaultAction"
},
"type": "array"
},
"expected_outcome": {
"$ref": "#/definitions/ExpectedTerminalOutcome"
},
"profile_id": {
"type": "string"
},
"profile_sha256": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"actions",
"expected_outcome",
"profile_id",
"profile_sha256",
"seed"
],
"title": "FaultPlan",
"type": "object"
}
},
{
"description": "List hash-validated E2E history records from the iii database worker.",
"metadata": {
"contract": {
"capabilities": [
"history-list"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::history-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"lane": {
"default": null,
"type": [
"string",
"null"
]
},
"limit": {
"default": 100,
"format": "uint16",
"minimum": 0,
"type": "integer"
}
},
"title": "HistoryListRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"DurableArchiveReference": {
"additionalProperties": false,
"properties": {
"archive_id": {
"type": "string"
},
"created_at": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"expires_at": {
"type": [
"string",
"null"
]
},
"identity_sha256": {
"type": "string"
},
"manifest": {
"$ref": "#/definitions/StorageObjectReference"
},
"manifest_backup": {
"$ref": "#/definitions/StorageObjectReference"
},
"retention_class": {
"$ref": "#/definitions/RetentionClass"
}
},
"required": [
"archive_id",
"created_at",
"execution_id",
"identity_sha256",
"manifest",
"manifest_backup",
"retention_class"
],
"type": "object"
},
"HistoryRecord": {
"additionalProperties": false,
"properties": {
"archive": {
"$ref": "#/definitions/DurableArchiveReference"
},
"baseline_comparable": {
"type": "boolean"
},
"case_count": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"e2e_repository": {
"type": "string"
},
"e2e_revision": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"identity_sha256": {
"type": "string"
},
"ingestion_id": {
"type": "string"
},
"lane": {
"type": "string"
},
"occurred_at": {
"type": "string"
},
"passed": {
"type": "boolean"
},
"result_contract_sha256": {
"type": "string"
},
"scoring_profile_sha256": {
"type": "string"
},
"stack_mode": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"subject_revision": {
"type": "string"
}
},
"required": [
"archive",
"baseline_comparable",
"case_count",
"e2e_repository",
"e2e_revision",
"execution_id",
"identity_sha256",
"ingestion_id",
"lane",
"occurred_at",
"passed",
"result_contract_sha256",
"scoring_profile_sha256",
"stack_mode",
"subject_model",
"subject_provider",
"subject_revision"
],
"type": "object"
},
"RetentionClass": {
"enum": [
"temporary",
"pull_request",
"longitudinal",
"canonical"
],
"type": "string"
},
"StorageObjectReference": {
"properties": {
"bucket": {
"type": "string"
},
"key": {
"type": "string"
},
"media_type": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"uri": {
"type": "string"
}
},
"required": [
"bucket",
"key",
"media_type",
"sha256",
"size_bytes",
"uri"
],
"type": "object"
}
},
"properties": {
"records": {
"items": {
"$ref": "#/definitions/HistoryRecord"
},
"type": "array"
}
},
"required": [
"records"
],
"title": "HistoryListResponse",
"type": "object"
}
},
{
"description": "Get the immutable result and manifest of an E2E execution.",
"metadata": {
"contract": {
"capabilities": [
"results-get"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::results-get",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
}
},
"required": [
"execution_id"
],
"title": "ExecutionRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ActivationPolicy": {
"oneOf": [
{
"properties": {
"policy": {
"enum": [
"always"
],
"type": "string"
}
},
"required": [
"policy"
],
"type": "object"
},
{
"properties": {
"conditions": {
"items": {
"$ref": "#/definitions/BooleanCondition"
},
"type": "array"
},
"policy": {
"enum": [
"all"
],
"type": "string"
}
},
"required": [
"conditions",
"policy"
],
"type": "object"
},
{
"properties": {
"conditions": {
"items": {
"$ref": "#/definitions/BooleanCondition"
},
"type": "array"
},
"policy": {
"enum": [
"any"
],
"type": "string"
}
},
"required": [
"conditions",
"policy"
],
"type": "object"
}
]
},
"AdherenceAvailability": {
"enum": [
"available",
"unavailable",
"failed"
],
"type": "string"
},
"AdherenceRequirement": {
"properties": {
"confidence": {
"format": "double",
"type": "number"
},
"evidence": {
"items": {
"type": "string"
},
"type": "array"
},
"followed": {
"type": "boolean"
},
"id": {
"type": "string"
},
"instruction": {
"type": "string"
},
"reason": {
"type": "string"
}
},
"required": [
"confidence",
"followed",
"id",
"instruction",
"reason"
],
"type": "object"
},
"AnalyzerIdentity": {
"additionalProperties": false,
"properties": {
"analyzer": {
"type": "string"
},
"input_sha256": {
"type": "string"
},
"model": {
"type": [
"string",
"null"
]
},
"provider": {
"type": [
"string",
"null"
]
}
},
"required": [
"analyzer",
"input_sha256"
],
"type": "object"
},
"AnalyzerUsage": {
"properties": {
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"latency_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"ArtifactExpectation": {
"properties": {
"id": {
"type": "string"
},
"kind": {
"type": "string"
},
"max_size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"media_type": {
"type": "string"
},
"schema": true
},
"required": [
"id",
"kind",
"max_size_bytes",
"media_type",
"schema"
],
"type": "object"
},
"ArtifactReference": {
"properties": {
"id": {
"type": "string"
},
"kind": {
"type": "string"
},
"media_type": {
"type": "string"
},
"path": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"id",
"kind",
"media_type",
"path",
"sha256",
"size_bytes"
],
"type": "object"
},
"AssessmentContract": {
"additionalProperties": false,
"properties": {
"runs": {
"items": {
"$ref": "#/definitions/RunAssessmentContract"
},
"type": "array"
}
},
"required": [
"runs"
],
"type": "object"
},
"AssessmentKind": {
"enum": [
"required_check",
"signal",
"asset_validation"
],
"type": "string"
},
"AssessmentOutcome": {
"enum": [
"passed",
"failed",
"partial",
"not_evaluated",
"unavailable",
"error"
],
"type": "string"
},
"AssessmentPolicy": {
"enum": [
"hard_gate",
"advisory"
],
"type": "string"
},
"AssessmentResult": {
"properties": {
"criterion_id": {
"type": "string"
},
"dimension": {
"$ref": "#/definitions/EvaluationDimension"
},
"evidence": {
"items": {
"$ref": "#/definitions/EvidenceReference"
},
"type": "array"
},
"kind": {
"$ref": "#/definitions/AssessmentKind"
},
"outcome": {
"$ref": "#/definitions/AssessmentOutcome"
},
"policy": {
"$ref": "#/definitions/AssessmentPolicy"
},
"score": {
"anyOf": [
{
"$ref": "#/definitions/AssessmentScore"
},
{
"type": "null"
}
]
},
"summary": {
"type": "string"
},
"target": {
"$ref": "#/definitions/AssessmentTarget"
}
},
"required": [
"criterion_id",
"dimension",
"kind",
"outcome",
"policy",
"summary",
"target"
],
"type": "object"
},
"AssessmentScore": {
"properties": {
"awarded": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"possible": {
"format": "uint8",
"minimum": 0,
"type": "integer"
}
},
"required": [
"awarded",
"possible"
],
"type": "object"
},
"AssessmentTarget": {
"properties": {
"id": {
"type": "string"
},
"kind": {
"$ref": "#/definitions/AssessmentTargetKind"
}
},
"required": [
"id",
"kind"
],
"type": "object"
},
"AssessmentTargetKind": {
"enum": [
"criterion"
],
"type": "string"
},
"AssetValidationOutcome": {
"enum": [
"valid",
"invalid",
"malformed",
"oversized",
"not_produced",
"unreadable",
"unsafe_path",
"removed_during_cleanup",
"unexpected",
"not_evaluated"
],
"type": "string"
},
"AssetValidationResult": {
"properties": {
"asset_id": {
"type": "string"
},
"evidence": {
"items": {
"$ref": "#/definitions/EvidenceReference"
},
"type": "array"
},
"outcome": {
"$ref": "#/definitions/AssetValidationOutcome"
},
"summary": {
"type": "string"
}
},
"required": [
"asset_id",
"outcome",
"summary"
],
"type": "object"
},
"AuditAnalyzerStatus": {
"oneOf": [
{
"enum": [
"completed",
"failed"
],
"type": "string"
},
{
"description": "No audit analyzer model was configured; only deterministic rules ran.",
"enum": [
"not_configured"
],
"type": "string"
},
{
"description": "No transcript was captured, so there was nothing to analyze.",
"enum": [
"no_transcript"
],
"type": "string"
}
]
},
"AuditEvidence": {
"properties": {
"detail": {
"description": "Sanitized, bounded description. Never contains secret material.",
"type": "string"
},
"function_id": {
"type": [
"string",
"null"
]
},
"message_index": {
"description": "Index of the transcript message the observation anchors to.",
"format": "uint",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"detail"
],
"type": "object"
},
"AuditFlag": {
"properties": {
"confidence": {
"format": "double",
"type": [
"number",
"null"
]
},
"evidence": {
"items": {
"$ref": "#/definitions/AuditEvidence"
},
"type": "array"
},
"kind": {
"$ref": "#/definitions/AuditFlagKind"
},
"severity": {
"$ref": "#/definitions/AuditSeverity"
},
"source": {
"$ref": "#/definitions/AuditFlagSource"
},
"summary": {
"type": "string"
}
},
"required": [
"kind",
"severity",
"source",
"summary"
],
"type": "object"
},
"AuditFlagKind": {
"oneOf": [
{
"description": "The subject invoked the E2E control plane or another function the scenario explicitly denies — the verification mechanism itself.",
"enum": [
"verifier_tampering"
],
"type": "string"
},
{
"description": "The subject addressed a session outside its own session tree.",
"enum": [
"out_of_scope_session_access"
],
"type": "string"
},
{
"description": "Live secret material from the runner environment appeared in the transcript.",
"enum": [
"secret_exposure"
],
"type": "string"
},
{
"description": "The subject invoked a destructive session or state operation.",
"enum": [
"destructive_action"
],
"type": "string"
},
{
"description": "Observed work exceeded the scenario's expectation by an anomalous factor.",
"enum": [
"anomalous_work"
],
"type": "string"
}
]
},
"AuditFlagSource": {
"enum": [
"deterministic",
"analyzer"
],
"type": "string"
},
"AuditReport": {
"properties": {
"analyzer": {
"anyOf": [
{
"$ref": "#/definitions/AnalyzerIdentity"
},
{
"type": "null"
}
]
},
"analyzer_error": {
"type": [
"string",
"null"
]
},
"analyzer_status": {
"$ref": "#/definitions/AuditAnalyzerStatus"
},
"analyzer_usage": {
"anyOf": [
{
"$ref": "#/definitions/AnalyzerUsage"
},
{
"type": "null"
}
]
},
"flags": {
"items": {
"$ref": "#/definitions/AuditFlag"
},
"type": "array"
}
},
"required": [
"analyzer_status",
"flags"
],
"type": "object"
},
"AuditSeverity": {
"enum": [
"info",
"warning",
"critical"
],
"type": "string"
},
"BooleanCondition": {
"additionalProperties": false,
"properties": {
"equals": {
"type": "boolean"
},
"node_id": {
"type": "string"
},
"port": {
"type": "string"
}
},
"required": [
"equals",
"node_id",
"port"
],
"type": "object"
},
"CapturedInvariant": {
"properties": {
"id": {
"type": "string"
},
"passed": {
"type": "boolean"
},
"reason": {
"type": "string"
}
},
"required": [
"id",
"passed",
"reason"
],
"type": "object"
},
"CompletionState": {
"enum": [
"completed",
"task_incomplete",
"undetermined"
],
"type": "string"
},
"ComplexityClassification": {
"properties": {
"method": {
"$ref": "#/definitions/ComplexityMethod"
},
"profile": {
"$ref": "#/definitions/ComplexityProfile"
},
"tier": {
"$ref": "#/definitions/ComplexityTier"
}
},
"required": [
"method",
"profile",
"tier"
],
"type": "object"
},
"ComplexityMethod": {
"enum": [
"capability_v2"
],
"type": "string"
},
"ComplexityProfile": {
"properties": {
"agent_owned_decomposition": {
"default": false,
"type": "boolean"
},
"ambiguity_level": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"artifact_count": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"coherent_long_horizon": {
"default": false,
"type": "boolean"
},
"compensable_mutations": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"coordination_edges": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"dependency_depth": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"durable_resume_cycles": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"external_systems": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"material_invalidation_events": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"parallel_branches": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"planning_depth": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"replan_loops": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"state_transitions": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"validation_loops": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"wake_cycles": {
"format": "uint8",
"minimum": 0,
"type": "integer"
}
},
"required": [
"ambiguity_level",
"artifact_count",
"coordination_edges",
"dependency_depth",
"external_systems",
"parallel_branches",
"planning_depth",
"state_transitions",
"validation_loops",
"wake_cycles"
],
"type": "object"
},
"ComplexityTier": {
"enum": [
"l0_atomic",
"l1_sequential",
"l2_stateful",
"l3_concurrent",
"l4_coordinated",
"l5_adaptive"
],
"type": "string"
},
"ContaminationScope": {
"enum": [
"none",
"case",
"execution"
],
"type": "string"
},
"ControlPlaneEvidence": {
"properties": {
"functions": {
"items": {
"$ref": "#/definitions/FunctionContractEvidence"
},
"type": "array"
}
},
"required": [
"functions"
],
"type": "object"
},
"CostReport": {
"properties": {
"subject_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"total_usd": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"type": "object"
},
"CriterionReport": {
"properties": {
"awarded": {
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"description": {
"type": [
"string",
"null"
]
},
"id": {
"type": "string"
},
"possible": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"reason": {
"type": "string"
}
},
"required": [
"id",
"possible",
"reason"
],
"type": "object"
},
"DeliverableContentFormat": {
"enum": [
"json",
"text_utf8"
],
"type": "string"
},
"DeliverableContract": {
"properties": {
"artifacts": {
"items": {
"$ref": "#/definitions/ArtifactExpectation"
},
"type": "array"
},
"capture_before_cleanup": {
"type": "boolean"
},
"invariants": {
"items": {
"$ref": "#/definitions/InvariantSpec"
},
"type": "array"
},
"provenance_required": {
"type": "boolean"
}
},
"required": [
"artifacts",
"capture_before_cleanup",
"invariants",
"provenance_required"
],
"type": "object"
},
"DeliverableReport": {
"properties": {
"artifact": {
"anyOf": [
{
"$ref": "#/definitions/ArtifactReference"
},
{
"type": "null"
}
]
},
"content_format": {
"$ref": "#/definitions/DeliverableContentFormat"
},
"content_sha256": {
"type": "string"
},
"content_size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"id": {
"type": "string"
},
"invariants": {
"items": {
"$ref": "#/definitions/CapturedInvariant"
},
"type": "array"
},
"kind": {
"type": "string"
},
"media_type": {
"type": "string"
},
"preview": true,
"provenance": {
"items": {
"$ref": "#/definitions/ProvenanceEvidence"
},
"type": "array"
},
"provenance_valid": {
"type": "boolean"
},
"schema_valid": {
"type": "boolean"
}
},
"required": [
"content_format",
"content_sha256",
"content_size_bytes",
"id",
"invariants",
"kind",
"media_type",
"preview",
"provenance",
"provenance_valid",
"schema_valid"
],
"type": "object"
},
"DependencyPolicy": {
"enum": [
"succeeded",
"terminal"
],
"type": "string"
},
"DimensionReport": {
"properties": {
"dimension": {
"$ref": "#/definitions/EvaluationDimension"
},
"passed": {
"type": [
"boolean",
"null"
]
},
"signals": true
},
"required": [
"dimension",
"signals"
],
"type": "object"
},
"DurableArchiveReference": {
"additionalProperties": false,
"properties": {
"archive_id": {
"type": "string"
},
"created_at": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"expires_at": {
"type": [
"string",
"null"
]
},
"identity_sha256": {
"type": "string"
},
"manifest": {
"$ref": "#/definitions/StorageObjectReference"
},
"manifest_backup": {
"$ref": "#/definitions/StorageObjectReference"
},
"retention_class": {
"$ref": "#/definitions/RetentionClass"
}
},
"required": [
"archive_id",
"created_at",
"execution_id",
"identity_sha256",
"manifest",
"manifest_backup",
"retention_class"
],
"type": "object"
},
"E2eManifest": {
"additionalProperties": false,
"properties": {
"control_plane": {
"$ref": "#/definitions/ControlPlaneEvidence"
},
"execution": {
"$ref": "#/definitions/ExecutionIdentity"
},
"judge": {
"anyOf": [
{
"$ref": "#/definitions/ModelArtifact"
},
{
"type": "null"
}
]
},
"observation_contract": {
"anyOf": [
{
"$ref": "#/definitions/ObservationRunContract"
},
{
"type": "null"
}
]
},
"subject": {
"$ref": "#/definitions/ModelArtifact"
},
"system_under_test": {
"$ref": "#/definitions/SystemUnderTestIdentity"
},
"worker_contracts": {
"items": {
"$ref": "#/definitions/ObservedWorkerContract"
},
"type": "array"
}
},
"required": [
"control_plane",
"execution",
"subject",
"system_under_test"
],
"type": "object"
},
"E2eObservationEnvelope": {
"additionalProperties": false,
"properties": {
"evidence": {
"$ref": "#/definitions/ObservationEvidence"
},
"identity": {
"$ref": "#/definitions/ObservationIdentity"
},
"mode": {
"$ref": "#/definitions/ObservationMode"
},
"outcome": {
"$ref": "#/definitions/ObservationOutcome"
},
"provenance": {
"$ref": "#/definitions/ObservationProvenance"
},
"samples": {
"items": {
"$ref": "#/definitions/ObservationSample"
},
"type": "array"
},
"schema": {
"type": "string"
}
},
"required": [
"evidence",
"identity",
"mode",
"outcome",
"provenance",
"samples",
"schema"
],
"type": "object"
},
"E2eReport": {
"additionalProperties": false,
"properties": {
"assessment_contract": {
"$ref": "#/definitions/AssessmentContract"
},
"engine_revision": {
"type": [
"string",
"null"
]
},
"execution": {
"$ref": "#/definitions/ExecutionIdentity"
},
"judge": {
"anyOf": [
{
"$ref": "#/definitions/ModelArtifact"
},
{
"type": "null"
}
]
},
"manifest": {
"anyOf": [
{
"$ref": "#/definitions/ArtifactReference"
},
{
"type": "null"
}
]
},
"objective_outcome": {
"$ref": "#/definitions/ObjectiveOutcome"
},
"observation_contract": {
"anyOf": [
{
"$ref": "#/definitions/ObservationRunContract"
},
{
"type": "null"
}
]
},
"passed": {
"type": "boolean"
},
"persistence_errors": {
"items": {
"type": "string"
},
"type": "array"
},
"redaction": {
"allOf": [
{
"$ref": "#/definitions/RedactionReport"
}
],
"default": {
"redacted_fields": 0,
"redacted_values": 0,
"rules": []
}
},
"report_state": {
"$ref": "#/definitions/ReportState"
},
"result_contract_sha256": {
"type": "string"
},
"scenarios": {
"items": {
"$ref": "#/definitions/E2eScenarioReport"
},
"type": "array"
},
"schema_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scoring_profile_sha256": {
"type": "string"
},
"slot_start_deadline_seconds": {
"description": "Soft scheduling deadline: prevents starting slots after this interval.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"subject": {
"$ref": "#/definitions/ModelArtifact"
},
"system_under_test": {
"$ref": "#/definitions/SystemUnderTestIdentity"
}
},
"required": [
"assessment_contract",
"execution",
"objective_outcome",
"passed",
"report_state",
"result_contract_sha256",
"scenarios",
"schema_version",
"scoring_profile_sha256",
"subject",
"system_under_test"
],
"type": "object"
},
"E2eRunReport": {
"properties": {
"attempt_id": {
"type": "string"
},
"attempt_number": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"audit": {
"anyOf": [
{
"$ref": "#/definitions/AuditReport"
},
{
"type": "null"
}
],
"description": "Advisory behavioral audit over the captured transcript and metrics. Never contributes to score, status, gates, or longitudinal inputs."
},
"completion": {
"$ref": "#/definitions/CompletionState"
},
"cost": {
"$ref": "#/definitions/CostReport"
},
"criteria": {
"items": {
"$ref": "#/definitions/CriterionReport"
},
"type": "array"
},
"deliverables": {
"items": {
"$ref": "#/definitions/DeliverableReport"
},
"type": "array"
},
"dimensions": {
"items": {
"$ref": "#/definitions/DimensionReport"
},
"type": "array"
},
"efficiency": {
"anyOf": [
{
"$ref": "#/definitions/EfficiencyReport"
},
{
"type": "null"
}
]
},
"evaluators": {
"$ref": "#/definitions/EvaluatorStates"
},
"evidence": {
"items": {
"$ref": "#/definitions/ArtifactReference"
},
"type": "array"
},
"failures": {
"items": {
"$ref": "#/definitions/FailureRecord"
},
"type": "array"
},
"instruction_adherence": {
"anyOf": [
{
"$ref": "#/definitions/InstructionAdherenceReport"
},
{
"type": "null"
}
]
},
"markdown_execution": {
"anyOf": [
{
"$ref": "#/definitions/MarkdownExecutionReport"
},
{
"type": "null"
}
]
},
"metrics": {
"anyOf": [
{
"$ref": "#/definitions/SessionMetricsPayload"
},
{
"type": "null"
}
]
},
"objective_score": {
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"prompt": {
"type": "string"
},
"quality_score_completed": {
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"retry_attempts": {
"items": {
"$ref": "#/definitions/RetryAttemptReport"
},
"type": "array"
},
"run_id": {
"type": "string"
},
"scenario_flow": {
"anyOf": [
{
"$ref": "#/definitions/ScenarioFlowEvidence"
},
{
"type": "null"
}
],
"description": "Evidence-only flow identity emitted by Rust. This value is never accepted as runner input and cannot reconstruct executable configuration."
},
"scenario_measurements": {
"items": {
"$ref": "#/definitions/ScenarioMeasurement"
},
"type": "array"
},
"score": {
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"semantic_tests": {
"description": "Semantic tests executed inside a code-defined composite scenario. Product requests, polling and captures stay inside these meaningful test units.",
"items": {
"$ref": "#/definitions/WorkflowStepReport"
},
"type": "array"
},
"session_id": {
"type": "string"
},
"status": {
"$ref": "#/definitions/RunStatus"
},
"technical": {
"$ref": "#/definitions/TechnicalState"
},
"transcript": true,
"validation_score": {
"description": "Explicit Markdown validation score. Mirrors `score` for Markdown-authored scenarios while keeping the legacy aggregate field compatible.",
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wall_time_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"worker_contracts": {
"description": "Exact request/response schemas observed after this scenario's setup. Run-scoped fixture functions cannot be observed in the suite preflight, so attempts retain them here and the suite folds them into the manifest.",
"items": {
"$ref": "#/definitions/ObservedWorkerContract"
},
"type": "array"
}
},
"required": [
"attempt_id",
"attempt_number",
"completion",
"cost",
"criteria",
"evaluators",
"prompt",
"run_id",
"session_id",
"status",
"technical",
"wall_time_ms"
],
"type": "object"
},
"E2eScenarioReport": {
"properties": {
"aggregate": {
"$ref": "#/definitions/ScenarioAggregate"
},
"case": {
"anyOf": [
{
"$ref": "#/definitions/ScenarioCase"
},
{
"type": "null"
}
]
},
"case_id": {
"default": "",
"type": "string"
},
"deferral_reason": {
"type": [
"string",
"null"
]
},
"execution_policy": {
"$ref": "#/definitions/ExecutionPolicy"
},
"passed": {
"type": "boolean"
},
"runs": {
"items": {
"$ref": "#/definitions/E2eRunReport"
},
"type": "array"
},
"scenario_id": {
"type": "string"
},
"scenario_version": {
"default": 1,
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"aggregate",
"execution_policy",
"passed",
"runs",
"scenario_id"
],
"type": "object"
},
"EfficiencyReport": {
"properties": {
"child_sessions": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"child_turns": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"critical_path_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"effective_fan_out": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"minimum_expected_work": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"observed_complexity": {
"$ref": "#/definitions/ObservedComplexityReport"
},
"observed_work": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"root_turns": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_attempts": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"total_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"transient_resumes": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"unavailable": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"validation_retries": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wake_resumes": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wall_time_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"work_amplification": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"minimum_expected_work",
"observed_complexity",
"technical_attempts",
"wall_time_ms"
],
"type": "object"
},
"EvaluationDimension": {
"enum": [
"deliverable",
"structural_integrity",
"efficiency",
"robustness",
"e2e_infrastructure"
],
"type": "string"
},
"EvaluatorAvailability": {
"enum": [
"not_required",
"pending",
"available",
"unavailable"
],
"type": "string"
},
"EvaluatorStates": {
"properties": {
"completion": {
"$ref": "#/definitions/EvaluatorAvailability"
},
"quality": {
"$ref": "#/definitions/EvaluatorAvailability"
}
},
"required": [
"completion",
"quality"
],
"type": "object"
},
"EvidenceReference": {
"properties": {
"artifact_id": {
"type": "string"
},
"artifact_sha256": {
"type": "string"
},
"locator": {
"type": [
"string",
"null"
]
}
},
"required": [
"artifact_id",
"artifact_sha256"
],
"type": "object"
},
"ExecutionIdentity": {
"properties": {
"completed_at": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"lane": {
"type": "string"
},
"started_at": {
"type": "string"
}
},
"required": [
"completed_at",
"execution_id",
"lane",
"started_at"
],
"type": "object"
},
"ExecutionPhase": {
"enum": [
"requested",
"admitted",
"preflighting",
"materializing",
"setting_up",
"executing",
"collecting",
"evaluating",
"persisting",
"cleaning_up",
"finalizing",
"completed",
"failed",
"cancelled",
"needs_reconciliation",
"unsupported"
],
"type": "string"
},
"ExecutionPolicy": {
"properties": {
"max_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_total_tokens": {
"description": "Shared Harness token budget. `None` leaves the Harness budget unbounded for this scenario.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_turns": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"max_validation_retries": {
"description": "Optional per-session cap for post-turn validation denials. `None` preserves the Harness-configured default.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stuck_timeout_seconds": {
"description": "Stop only after this many seconds without observable useful progress. Large scenarios have no fixed wall-clock deadline.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"max_turns",
"stuck_timeout_seconds"
],
"type": "object"
},
"ExecutionRealism": {
"enum": [
"synthetic",
"realistic_simulator",
"frozen_real_artifact"
],
"type": "string"
},
"FailureDomain": {
"enum": [
"subject",
"judge",
"resource",
"e2e_infrastructure"
],
"type": "string"
},
"FailurePhase": {
"enum": [
"setup",
"execute",
"collect",
"evaluate",
"cleanup"
],
"type": "string"
},
"FailureRecord": {
"properties": {
"code": {
"default": "",
"type": "string"
},
"contamination": {
"allOf": [
{
"$ref": "#/definitions/ContaminationScope"
}
],
"default": "none"
},
"domain": {
"$ref": "#/definitions/FailureDomain"
},
"message": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/FailurePhase"
},
"retry_scope": {
"allOf": [
{
"$ref": "#/definitions/RetryScope"
}
],
"default": "none"
},
"status": {
"allOf": [
{
"$ref": "#/definitions/RunStatus"
}
],
"description": "The failure's original classification, independent of the primary run status."
}
},
"required": [
"domain",
"message",
"phase",
"status"
],
"type": "object"
},
"FunctionContractEvidence": {
"properties": {
"function_id": {
"type": "string"
},
"request_schema": true,
"response_schema": true,
"sha256": {
"type": "string"
}
},
"required": [
"function_id",
"request_schema",
"response_schema",
"sha256"
],
"type": "object"
},
"HumanHorizon": {
"properties": {
"basis": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizonBasis"
}
],
"default": "unknown"
},
"max_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"min_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"HumanHorizonBasis": {
"enum": [
"unknown",
"author_estimate",
"measured"
],
"type": "string"
},
"InstructionAdherenceReport": {
"properties": {
"analyzer": {
"anyOf": [
{
"$ref": "#/definitions/AnalyzerIdentity"
},
{
"type": "null"
}
]
},
"analyzer_usage": {
"anyOf": [
{
"$ref": "#/definitions/AnalyzerUsage"
},
{
"type": "null"
}
]
},
"availability": {
"$ref": "#/definitions/AdherenceAvailability"
},
"requirements": {
"items": {
"$ref": "#/definitions/AdherenceRequirement"
},
"type": "array"
},
"score": {
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"summary": {
"type": "string"
}
},
"required": [
"availability",
"summary"
],
"type": "object"
},
"InvariantSpec": {
"properties": {
"description": {
"type": "string"
},
"id": {
"type": "string"
}
},
"required": [
"description",
"id"
],
"type": "object"
},
"MarkdownExecutionReport": {
"properties": {
"behavior_sha256": {
"type": "string"
},
"compiled_sha256": {
"type": "string"
},
"materialized_plan_sha256": {
"type": [
"string",
"null"
]
},
"phases": {
"items": {
"$ref": "#/definitions/MarkdownPhaseReport"
},
"type": "array"
},
"pipeline_complete": {
"type": "boolean"
},
"prompt_sha256": {
"type": "string"
},
"source_path": {
"type": "string"
},
"source_sha256": {
"type": "string"
}
},
"required": [
"behavior_sha256",
"compiled_sha256",
"phases",
"pipeline_complete",
"prompt_sha256",
"source_path",
"source_sha256"
],
"type": "object"
},
"MarkdownPhaseReport": {
"properties": {
"input_sha256": {
"type": "string"
},
"phase": {
"type": "string"
},
"reason": {
"type": "string"
},
"session_id": {
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/MarkdownPhaseStatus"
},
"transcript_sha256": {
"type": [
"string",
"null"
]
}
},
"required": [
"input_sha256",
"phase",
"status"
],
"type": "object"
},
"MarkdownPhaseStatus": {
"enum": [
"pending",
"completed",
"failed",
"skipped"
],
"type": "string"
},
"ModelArtifact": {
"properties": {
"context_window": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"max_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"supports_tools": {
"type": [
"boolean",
"null"
]
},
"supports_vision": {
"type": [
"boolean",
"null"
]
}
},
"required": [
"context_window",
"max_output_tokens",
"model",
"provider"
],
"type": "object"
},
"ObjectiveOutcome": {
"enum": [
"passed",
"failed",
"inconclusive"
],
"type": "string"
},
"ObservationCorrelation": {
"additionalProperties": false,
"properties": {
"deployment_id": {
"type": "string"
},
"operation_id": {
"type": "string"
},
"system": {
"type": "string"
}
},
"required": [
"deployment_id",
"operation_id",
"system"
],
"type": "object"
},
"ObservationDataAvailability": {
"enum": [
"complete",
"partial",
"unavailable"
],
"type": "string"
},
"ObservationDecision": {
"enum": [
"observe_only"
],
"type": "string"
},
"ObservationEnvironment": {
"enum": [
"demonstration"
],
"type": "string"
},
"ObservationEvidence": {
"additionalProperties": false,
"properties": {
"artifacts": {
"items": {
"$ref": "#/definitions/ArtifactReference"
},
"type": "array"
},
"manifest_sha256": {
"type": [
"string",
"null"
]
},
"results_sha256": {
"type": [
"string",
"null"
]
}
},
"type": "object"
},
"ObservationExecutionIdentity": {
"additionalProperties": false,
"properties": {
"attempt": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"completed_at": {
"type": "string"
},
"id": {
"type": "string"
},
"request_sha256": {
"type": "string"
},
"run_contract_sha256": {
"type": "string"
},
"started_at": {
"type": "string"
}
},
"required": [
"attempt",
"completed_at",
"id",
"request_sha256",
"run_contract_sha256",
"started_at"
],
"type": "object"
},
"ObservationIdentity": {
"additionalProperties": false,
"properties": {
"execution": {
"$ref": "#/definitions/ObservationExecutionIdentity"
},
"plan": {
"$ref": "#/definitions/ObservationPlanIdentity"
},
"runner": {
"$ref": "#/definitions/RunnerIdentity"
},
"target": {
"$ref": "#/definitions/ObservationTargetIdentity"
}
},
"required": [
"execution",
"plan",
"runner",
"target"
],
"type": "object"
},
"ObservationMetric": {
"additionalProperties": false,
"description": "A metric is explicit about its unit and availability so a recorded zero cannot be mistaken for a missing measurement. The legacy `metrics` object remains alongside this additive projection for current consumers.",
"properties": {
"availability": {
"$ref": "#/definitions/ObservationDataAvailability"
},
"origin": {
"$ref": "#/definitions/ObservationMetricOrigin"
},
"unit": {
"type": "string"
},
"value": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"availability",
"origin",
"unit"
],
"type": "object"
},
"ObservationMetricDerivation": {
"additionalProperties": false,
"properties": {
"formula": {
"type": "string"
},
"formula_version": {
"type": "string"
},
"metric": {
"type": "string"
},
"origin": {
"$ref": "#/definitions/ObservationMetricOrigin"
}
},
"required": [
"formula",
"formula_version",
"metric",
"origin"
],
"type": "object"
},
"ObservationMetricOrigin": {
"enum": [
"observed",
"derived_from_observed"
],
"type": "string"
},
"ObservationMode": {
"additionalProperties": false,
"properties": {
"decision": {
"$ref": "#/definitions/ObservationDecision"
},
"environment": {
"$ref": "#/definitions/ObservationEnvironment"
}
},
"required": [
"decision",
"environment"
],
"type": "object"
},
"ObservationObjective": {
"enum": [
"passed",
"hard_gate_failed",
"technical_failed",
"infrastructure_failed",
"cancelled",
"unsupported_plan"
],
"type": "string"
},
"ObservationOutcome": {
"additionalProperties": false,
"properties": {
"control_phase": {
"type": "string"
},
"data_availability": {
"$ref": "#/definitions/ObservationDataAvailability"
},
"error": {
"type": "string"
},
"objective": {
"$ref": "#/definitions/ObservationObjective"
},
"passed": {
"type": [
"boolean",
"null"
]
}
},
"required": [
"control_phase",
"data_availability",
"objective"
],
"type": "object"
},
"ObservationPlanIdentity": {
"additionalProperties": false,
"properties": {
"catalog_sha256": {
"type": "string"
},
"id": {
"type": "string"
},
"revision": {
"type": "string"
},
"sha256": {
"type": "string"
}
},
"required": [
"catalog_sha256",
"id",
"revision",
"sha256"
],
"type": "object"
},
"ObservationProvenance": {
"additionalProperties": false,
"properties": {
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"system_under_test": {
"anyOf": [
{
"$ref": "#/definitions/SystemUnderTestIdentity"
},
{
"type": "null"
}
]
}
},
"required": [
"subject_model",
"subject_provider"
],
"type": "object"
},
"ObservationRunContract": {
"additionalProperties": false,
"properties": {
"attempt": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"correlation": {
"$ref": "#/definitions/ObservationCorrelation"
},
"mode": {
"$ref": "#/definitions/ObservationMode"
},
"plan": {
"$ref": "#/definitions/ObservationPlanIdentity"
},
"runner": {
"$ref": "#/definitions/RunnerIdentity"
},
"selected_cases": {
"items": {
"$ref": "#/definitions/ObservationSelectedCase"
},
"type": "array"
},
"target": {
"$ref": "#/definitions/ObservationTargetIdentity"
}
},
"required": [
"attempt",
"correlation",
"mode",
"plan",
"runner",
"selected_cases",
"target"
],
"type": "object"
},
"ObservationSample": {
"additionalProperties": false,
"properties": {
"attempt_id": {
"type": "string"
},
"case_id": {
"type": "string"
},
"data_availability": {
"$ref": "#/definitions/ObservationDataAvailability"
},
"derivations": {
"items": {
"$ref": "#/definitions/ObservationMetricDerivation"
},
"type": "array"
},
"metric_values": {
"additionalProperties": {
"$ref": "#/definitions/ObservationMetric"
},
"default": {},
"type": "object"
},
"metrics": {
"anyOf": [
{
"$ref": "#/definitions/EfficiencyReport"
},
{
"type": "null"
}
]
},
"origin": {
"$ref": "#/definitions/ObservationMetricOrigin"
},
"run_id": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"status": {
"$ref": "#/definitions/RunStatus"
}
},
"required": [
"attempt_id",
"case_id",
"data_availability",
"origin",
"run_id",
"scenario_id",
"scenario_version",
"seed",
"status"
],
"type": "object"
},
"ObservationSelectedCase": {
"additionalProperties": false,
"properties": {
"case_id": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"inputs_sha256": {
"type": "string"
},
"scenario_id": {
"$ref": "#/definitions/ScenarioKey"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"case_id",
"contract_sha256",
"inputs_sha256",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"ObservationTargetIdentity": {
"additionalProperties": false,
"properties": {
"application": {
"type": "string"
},
"stack": {
"$ref": "#/definitions/StackIdentity"
},
"version": {
"type": "string"
}
},
"required": [
"application",
"stack",
"version"
],
"type": "object"
},
"ObservedComplexityReport": {
"properties": {
"ambiguity_level": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"artifact_count": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"coordination_edges": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"dependency_depth": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"external_systems": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"parallel_branches": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"planning_depth": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"state_transitions": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"unavailable": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"validation_loops": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wake_cycles": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"ObservedWorkerContract": {
"additionalProperties": false,
"properties": {
"function_id": {
"type": "string"
},
"request_schema_sha256": {
"type": "string"
},
"response_schema_sha256": {
"type": "string"
}
},
"required": [
"function_id",
"request_schema_sha256",
"response_schema_sha256"
],
"type": "object"
},
"PortValueKind": {
"enum": [
"boolean",
"json",
"text_utf8",
"assessment"
],
"type": "string"
},
"ProvenanceEvidence": {
"properties": {
"kind": {
"type": "string"
},
"relation": {
"type": "string"
},
"source_id": {
"type": "string"
}
},
"required": [
"kind",
"relation",
"source_id"
],
"type": "object"
},
"RedactionReport": {
"properties": {
"redacted_fields": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"redacted_values": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"rules": {
"items": {
"type": "string"
},
"type": "array",
"uniqueItems": true
}
},
"required": [
"redacted_fields",
"redacted_values",
"rules"
],
"type": "object"
},
"ReportState": {
"enum": [
"complete",
"partial"
],
"type": "string"
},
"RetentionClass": {
"enum": [
"temporary",
"pull_request",
"longitudinal",
"canonical"
],
"type": "string"
},
"RetryAttemptReport": {
"properties": {
"attempt_id": {
"type": "string"
},
"attempt_number": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"completion": {
"$ref": "#/definitions/CompletionState"
},
"cost": {
"$ref": "#/definitions/CostReport"
},
"deliverables": {
"items": {
"$ref": "#/definitions/DeliverableReport"
},
"type": "array"
},
"dimensions": {
"items": {
"$ref": "#/definitions/DimensionReport"
},
"type": "array"
},
"efficiency": {
"anyOf": [
{
"$ref": "#/definitions/EfficiencyReport"
},
{
"type": "null"
}
]
},
"evaluators": {
"$ref": "#/definitions/EvaluatorStates"
},
"evidence": {
"items": {
"$ref": "#/definitions/ArtifactReference"
},
"type": "array"
},
"failures": {
"items": {
"$ref": "#/definitions/FailureRecord"
},
"type": "array"
},
"instruction_adherence": {
"anyOf": [
{
"$ref": "#/definitions/InstructionAdherenceReport"
},
{
"type": "null"
}
]
},
"markdown_execution": {
"anyOf": [
{
"$ref": "#/definitions/MarkdownExecutionReport"
},
{
"type": "null"
}
]
},
"metrics": {
"anyOf": [
{
"$ref": "#/definitions/SessionMetricsPayload"
},
{
"type": "null"
}
]
},
"objective_score": {
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"quality_score_completed": {
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"run_id": {
"type": "string"
},
"scenario_flow": {
"anyOf": [
{
"$ref": "#/definitions/ScenarioFlowEvidence"
},
{
"type": "null"
}
]
},
"semantic_tests": {
"items": {
"$ref": "#/definitions/WorkflowStepReport"
},
"type": "array"
},
"session_id": {
"type": "string"
},
"status": {
"$ref": "#/definitions/RunStatus"
},
"technical": {
"$ref": "#/definitions/TechnicalState"
},
"transcript": true,
"validation_score": {
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wall_time_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"attempt_id",
"attempt_number",
"completion",
"cost",
"evaluators",
"failures",
"run_id",
"session_id",
"status",
"technical",
"wall_time_ms"
],
"type": "object"
},
"RetryScope": {
"enum": [
"none",
"same_slot"
],
"type": "string"
},
"RobustnessReport": {
"properties": {
"cost_per_successful_deliverable": {
"format": "double",
"type": [
"number",
"null"
]
},
"deliverable_success_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"eligible": {
"type": "boolean"
},
"flaky_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_wall_time_ms": {
"format": "double",
"type": [
"number",
"null"
]
},
"minimum_sample_size": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"p95_wall_time_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"sample_size": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"structural_integrity_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"tail_minimum_sample_size": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"technical_failure_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"unavailable": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"wall_time_variance": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"eligible",
"minimum_sample_size",
"sample_size",
"tail_minimum_sample_size"
],
"type": "object"
},
"RunAssessmentContract": {
"properties": {
"assessments": {
"items": {
"$ref": "#/definitions/AssessmentResult"
},
"type": "array"
},
"assets": {
"items": {
"$ref": "#/definitions/AssetValidationResult"
},
"type": "array"
},
"attempt_id": {
"type": "string"
},
"run_id": {
"type": "string"
},
"system_status": {
"$ref": "#/definitions/SystemStatus"
}
},
"required": [
"attempt_id",
"run_id",
"system_status"
],
"type": "object"
},
"RunStatus": {
"enum": [
"passed",
"hard_gate_failed",
"subject_error",
"judge_error",
"resource_limit",
"infrastructure_error"
],
"type": "string"
},
"RunnerIdentity": {
"additionalProperties": false,
"properties": {
"name": {
"type": "string"
},
"revision": {
"type": "string"
},
"version": {
"type": "string"
}
},
"required": [
"name",
"revision",
"version"
],
"type": "object"
},
"ScenarioAggregate": {
"properties": {
"completed_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"completion_evidence_coverage": {
"format": "double",
"type": [
"number",
"null"
]
},
"completion_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"cost": {
"$ref": "#/definitions/CostReport"
},
"deferred_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"execution_reliability": {
"format": "double",
"type": [
"number",
"null"
]
},
"failed_attempt_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"median_score": {
"description": "Compatibility alias for `objective_median_score`.",
"format": "double",
"type": [
"number",
"null"
]
},
"objective_median_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"objective_score_coverage": {
"format": "double",
"type": [
"number",
"null"
]
},
"objective_scored_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"observed_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"pass_rate": {
"format": "double",
"type": "number"
},
"passed_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"planned_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"quality_coverage": {
"format": "double",
"type": [
"number",
"null"
]
},
"quality_score_completed": {
"format": "double",
"type": [
"number",
"null"
]
},
"quality_scored_completed_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"required_passes": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"robustness": {
"$ref": "#/definitions/RobustnessReport"
},
"runs": {
"description": "Compatibility alias for `observed_runs`.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scored_runs": {
"description": "Compatibility alias for `objective_scored_runs`.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"task_incomplete_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"technical_failures": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"technical_invalid_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"technical_valid_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"tokens_completed_p50": {
"format": "double",
"type": [
"number",
"null"
]
},
"tokens_per_completion": {
"format": "double",
"type": [
"number",
"null"
]
},
"total_tokens_consumed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"undetermined_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"completed_runs",
"cost",
"deferred_runs",
"objective_scored_runs",
"observed_runs",
"pass_rate",
"passed_runs",
"planned_runs",
"quality_scored_completed_runs",
"required_passes",
"robustness",
"runs",
"scored_runs",
"task_incomplete_runs",
"technical_failures",
"technical_invalid_runs",
"technical_valid_runs",
"undetermined_runs"
],
"type": "object"
},
"ScenarioCase": {
"properties": {
"case_id": {
"type": "string"
},
"characterization": {
"allOf": [
{
"$ref": "#/definitions/ScenarioCharacterization"
}
],
"default": {
"human_horizon": {
"basis": "unknown"
},
"realism": {
"execution": "synthetic",
"shadow": "none"
}
}
},
"complexity": {
"$ref": "#/definitions/ComplexityClassification"
},
"deliverable_contract": {
"$ref": "#/definitions/DeliverableContract"
},
"inputs": true,
"inputs_sha256": {
"type": "string"
},
"required_capabilities": {
"items": {
"type": "string"
},
"type": "array"
},
"scenario_id": {
"type": "string"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"work": {
"$ref": "#/definitions/WorkExpectation"
}
},
"required": [
"case_id",
"complexity",
"deliverable_contract",
"inputs",
"inputs_sha256",
"required_capabilities",
"scenario_id",
"scenario_version",
"seed",
"work"
],
"type": "object"
},
"ScenarioCharacterization": {
"properties": {
"human_horizon": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizon"
}
],
"default": {
"basis": "unknown"
}
},
"realism": {
"allOf": [
{
"$ref": "#/definitions/ScenarioRealism"
}
],
"default": {
"execution": "synthetic",
"shadow": "none"
}
}
},
"type": "object"
},
"ScenarioFlowEvidence": {
"additionalProperties": false,
"properties": {
"checkpoint": {
"$ref": "#/definitions/ArtifactReference"
},
"cleanup": {
"$ref": "#/definitions/WorkflowCleanupReport"
},
"definition_sha256": {
"type": "string"
},
"snapshot": true
},
"required": [
"checkpoint",
"cleanup",
"definition_sha256",
"snapshot"
],
"type": "object"
},
"ScenarioKey": {
"type": "string"
},
"ScenarioMeasurement": {
"additionalProperties": false,
"properties": {
"id": {
"type": "string"
},
"origin": {
"$ref": "#/definitions/ObservationMetricOrigin"
},
"unit": {
"type": "string"
},
"value": {
"format": "double",
"type": "number"
}
},
"required": [
"id",
"origin",
"unit",
"value"
],
"type": "object"
},
"ScenarioRealism": {
"properties": {
"execution": {
"allOf": [
{
"$ref": "#/definitions/ExecutionRealism"
}
],
"default": "synthetic"
},
"shadow": {
"allOf": [
{
"$ref": "#/definitions/ShadowMode"
}
],
"default": "none"
}
},
"type": "object"
},
"SessionMetricsPayload": {
"properties": {
"by_session": {
"items": {
"$ref": "#/definitions/SessionUsage"
},
"type": "array"
},
"complete": {
"type": "boolean"
},
"root_session_id": {
"type": "string"
},
"totals": {
"$ref": "#/definitions/SessionUsageTotals"
},
"traces": {
"default": null
}
},
"required": [
"by_session",
"complete",
"root_session_id",
"totals"
],
"type": "object"
},
"SessionUsage": {
"properties": {
"cache_read_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"context": {
"default": null
},
"cost_usd": {
"default": null,
"format": "double",
"type": [
"number",
"null"
]
},
"depth": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"parent_session_id": {
"default": null,
"type": [
"string",
"null"
]
},
"reasoning_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"session_id": {
"type": "string"
},
"transient_resumes": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"turns": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"validation_retries": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wake_resumes": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"depth",
"function_call_errors",
"function_calls",
"session_id",
"turns"
],
"type": "object"
},
"SessionUsageTotals": {
"properties": {
"cache_read_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"default": null,
"format": "double",
"type": [
"number",
"null"
]
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"reasoning_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"sessions": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"transient_resumes": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"turns": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"validation_retries": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wake_resumes": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"function_call_errors",
"function_calls",
"sessions",
"turns"
],
"type": "object"
},
"ShadowMode": {
"enum": [
"none",
"read_only"
],
"type": "string"
},
"StackIdentity": {
"oneOf": [
{
"properties": {
"mode": {
"enum": [
"source"
],
"type": "string"
},
"workers_repository": {
"type": "string"
},
"workers_revision": {
"type": "string"
}
},
"required": [
"mode",
"workers_repository",
"workers_revision"
],
"type": "object"
},
{
"properties": {
"mode": {
"enum": [
"registry"
],
"type": "string"
},
"stack_lock_digest": {
"type": "string"
},
"stack_versions": {
"additionalProperties": {
"type": "string"
},
"type": "object"
}
},
"required": [
"mode",
"stack_lock_digest",
"stack_versions"
],
"type": "object"
}
]
},
"StorageObjectReference": {
"properties": {
"bucket": {
"type": "string"
},
"key": {
"type": "string"
},
"media_type": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"uri": {
"type": "string"
}
},
"required": [
"bucket",
"key",
"media_type",
"sha256",
"size_bytes",
"uri"
],
"type": "object"
},
"SystemStatus": {
"enum": [
"unavailable",
"passed",
"hard_gate_failed",
"subject_error",
"judge_error",
"resource_limit",
"infrastructure_error"
],
"type": "string"
},
"SystemUnderTestIdentity": {
"properties": {
"contract_hashes": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"e2e_repository": {
"type": "string"
},
"e2e_revision": {
"type": "string"
},
"engine_revision": {
"type": [
"string",
"null"
]
},
"engine_version": {
"type": "string"
},
"harness_version": {
"type": "string"
},
"stack": {
"$ref": "#/definitions/StackIdentity"
}
},
"required": [
"contract_hashes",
"e2e_repository",
"e2e_revision",
"engine_version",
"harness_version",
"stack"
],
"type": "object"
},
"TechnicalState": {
"enum": [
"valid",
"technical_invalid"
],
"type": "string"
},
"TypedPortValue": {
"additionalProperties": false,
"properties": {
"kind": {
"$ref": "#/definitions/PortValueKind"
},
"value": true
},
"required": [
"kind",
"value"
],
"type": "object"
},
"WorkExpectation": {
"properties": {
"minimum_expected_work": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"minimum_expected_work"
],
"type": "object"
},
"WorkflowAssetReport": {
"additionalProperties": false,
"properties": {
"artifact": {
"$ref": "#/definitions/ArtifactReference"
},
"content_sha256": {
"type": "string"
},
"id": {
"type": "string"
},
"kind": {
"type": "string"
},
"media_type": {
"type": "string"
},
"namespaced_id": {
"type": "string"
},
"preview": true,
"preview_truncated": {
"type": "boolean"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"artifact",
"content_sha256",
"id",
"kind",
"media_type",
"namespaced_id",
"preview",
"preview_truncated",
"size_bytes"
],
"type": "object"
},
"WorkflowCleanupReport": {
"additionalProperties": false,
"properties": {
"duration_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"failure": {
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/WorkflowCleanupStatus"
}
},
"required": [
"duration_ms",
"status"
],
"type": "object"
},
"WorkflowCleanupStatus": {
"enum": [
"succeeded",
"failed"
],
"type": "string"
},
"WorkflowEvaluationOutcome": {
"enum": [
"passed",
"failed",
"advisory",
"not_evaluated"
],
"type": "string"
},
"WorkflowEvaluationResult": {
"additionalProperties": false,
"properties": {
"evidence_ids": {
"default": [],
"items": {
"type": "string"
},
"type": "array"
},
"id": {
"type": "string"
},
"outcome": {
"$ref": "#/definitions/WorkflowEvaluationOutcome"
},
"score": {
"format": "double",
"type": [
"number",
"null"
]
},
"summary": {
"type": "string"
}
},
"required": [
"id",
"outcome",
"summary"
],
"type": "object"
},
"WorkflowFailurePhase": {
"enum": [
"preflight",
"execute",
"capture",
"persist",
"evaluate",
"cleanup",
"cancel"
],
"type": "string"
},
"WorkflowGateResult": {
"additionalProperties": false,
"properties": {
"evidence_ids": {
"default": [],
"items": {
"type": "string"
},
"type": "array"
},
"id": {
"type": "string"
},
"passed": {
"type": "boolean"
},
"reason": {
"type": "string"
}
},
"required": [
"id",
"passed",
"reason"
],
"type": "object"
},
"WorkflowStepFailure": {
"additionalProperties": false,
"properties": {
"message": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/WorkflowFailurePhase"
},
"technical": {
"type": "boolean"
}
},
"required": [
"message",
"phase",
"technical"
],
"type": "object"
},
"WorkflowStepReport": {
"additionalProperties": false,
"properties": {
"activation": {
"$ref": "#/definitions/ActivationPolicy"
},
"assets": {
"items": {
"$ref": "#/definitions/WorkflowAssetReport"
},
"type": "array"
},
"completed_at": {
"type": [
"string",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"dependencies": {
"items": {
"type": "string"
},
"type": "array"
},
"dependency_policy": {
"$ref": "#/definitions/DependencyPolicy"
},
"duration_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"evaluations": {
"items": {
"$ref": "#/definitions/WorkflowEvaluationResult"
},
"type": "array"
},
"failures": {
"items": {
"$ref": "#/definitions/WorkflowStepFailure"
},
"type": "array"
},
"hard_gates": {
"items": {
"$ref": "#/definitions/WorkflowGateResult"
},
"type": "array"
},
"harness_session_id": {
"type": [
"string",
"null"
]
},
"metrics": true,
"node_id": {
"type": "string"
},
"outputs": {
"additionalProperties": {
"$ref": "#/definitions/TypedPortValue"
},
"type": "object"
},
"redaction": {
"allOf": [
{
"$ref": "#/definitions/RedactionReport"
}
],
"default": {
"redacted_fields": 0,
"redacted_values": 0,
"rules": []
}
},
"required": {
"type": "boolean"
},
"skip_reason": {
"type": [
"string",
"null"
]
},
"started_at": {
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/WorkflowStepStatus"
},
"step_type": {
"type": "string"
},
"step_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"transcript": true
},
"required": [
"activation",
"dependencies",
"dependency_policy",
"duration_ms",
"node_id",
"required",
"status",
"step_type",
"step_version"
],
"type": "object"
},
"WorkflowStepStatus": {
"enum": [
"pending",
"running",
"succeeded",
"hard_gate_failed",
"failed",
"skipped",
"cancelled"
],
"type": "string"
}
},
"properties": {
"archive": {
"anyOf": [
{
"$ref": "#/definitions/DurableArchiveReference"
},
{
"type": "null"
}
]
},
"execution_id": {
"type": "string"
},
"manifest": {
"anyOf": [
{
"$ref": "#/definitions/E2eManifest"
},
{
"type": "null"
}
]
},
"observation": {
"anyOf": [
{
"$ref": "#/definitions/E2eObservationEnvelope"
},
{
"type": "null"
}
]
},
"observation_artifact": {
"anyOf": [
{
"$ref": "#/definitions/ArtifactReference"
},
{
"type": "null"
}
]
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
},
"report": {
"anyOf": [
{
"$ref": "#/definitions/E2eReport"
},
{
"type": "null"
}
]
},
"request_sha256": {
"type": "string"
},
"result_path": {
"type": [
"string",
"null"
]
},
"run_contract_sha256": {
"type": [
"string",
"null"
]
}
},
"required": [
"execution_id",
"phase",
"request_sha256"
],
"title": "ResultsGetResponse",
"type": "object"
}
},
{
"description": "List durable E2E executions using comparable identity filters.",
"metadata": {
"contract": {
"capabilities": [
"results-list"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::results-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ScenarioKey": {
"type": "string"
}
},
"properties": {
"lane": {
"default": null,
"type": [
"string",
"null"
]
},
"limit": {
"default": 50,
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"scenario_id": {
"anyOf": [
{
"$ref": "#/definitions/ScenarioKey"
},
{
"type": "null"
}
],
"default": null
}
},
"title": "ResultsListRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ActiveAttempt": {
"properties": {
"attempt_id": {
"type": "string"
},
"resume_state_path": {
"type": [
"string",
"null"
]
},
"resume_state_sha256": {
"type": [
"string",
"null"
]
},
"run_id": {
"default": "",
"type": "string"
},
"scenario_id": {
"$ref": "#/definitions/ScenarioKey"
},
"session_id": {
"type": "string"
}
},
"required": [
"attempt_id",
"scenario_id",
"session_id"
],
"type": "object"
},
"ExecutionPhase": {
"enum": [
"requested",
"admitted",
"preflighting",
"materializing",
"setting_up",
"executing",
"collecting",
"evaluating",
"persisting",
"cleaning_up",
"finalizing",
"completed",
"failed",
"cancelled",
"needs_reconciliation",
"unsupported"
],
"type": "string"
},
"JournalProgress": {
"properties": {
"active_attempt_id": {
"type": [
"string",
"null"
]
},
"attempts_finished": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"attempts_started": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"committed_events": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"last_event_sha256": {
"type": [
"string",
"null"
]
},
"phase": {
"type": [
"string",
"null"
]
},
"planned_slots": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"runs_committed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"slots_deferred": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"subject_observations_committed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"terminal": {
"type": "boolean"
},
"terminal_reason": {
"type": [
"string",
"null"
]
},
"terminal_state": {
"anyOf": [
{
"$ref": "#/definitions/JournalTerminalState"
},
{
"type": "null"
}
]
}
},
"required": [
"attempts_finished",
"attempts_started",
"committed_events",
"planned_slots",
"runs_committed",
"slots_deferred",
"subject_observations_committed",
"terminal"
],
"type": "object"
},
"JournalTerminalState": {
"enum": [
"completed",
"failed",
"cancelled",
"needs_reconciliation",
"unsupported"
],
"type": "string"
},
"PhaseTransition": {
"properties": {
"at": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
},
"reason": {
"type": "string"
}
},
"required": [
"at",
"phase",
"reason"
],
"type": "object"
},
"ScenarioKey": {
"type": "string"
},
"StatusResponse": {
"properties": {
"active_attempt": {
"anyOf": [
{
"$ref": "#/definitions/ActiveAttempt"
},
{
"type": "null"
}
]
},
"cancel_requested": {
"type": "boolean"
},
"error": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"journal_progress": {
"$ref": "#/definitions/JournalProgress"
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
},
"terminal": {
"type": "boolean"
},
"transitions": {
"items": {
"$ref": "#/definitions/PhaseTransition"
},
"type": "array"
},
"updated_at": {
"type": "string"
}
},
"required": [
"cancel_requested",
"execution_id",
"journal_progress",
"phase",
"terminal",
"transitions",
"updated_at"
],
"type": "object"
}
},
"properties": {
"executions": {
"items": {
"$ref": "#/definitions/StatusResponse"
},
"type": "array"
}
},
"required": [
"executions"
],
"title": "ResultsListResponse",
"type": "object"
}
},
{
"description": "Delete expired E2E objects and tombstone their history records.",
"metadata": {
"contract": {
"capabilities": [
"retention-sweep"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::retention-sweep",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"before": {
"type": "string"
},
"dry_run": {
"default": false,
"type": "boolean"
},
"limit": {
"default": 100,
"format": "uint16",
"minimum": 0,
"type": "integer"
}
},
"required": [
"before"
],
"title": "RetentionSweepRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"deleted_archives": {
"items": {
"type": "string"
},
"type": "array"
},
"dry_run": {
"type": "boolean"
},
"inspected": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"deleted_archives",
"dry_run",
"inspected"
],
"title": "RetentionSweepResponse",
"type": "object"
}
},
{
"description": "Admit an asynchronous, idempotent E2E execution.",
"metadata": {
"contract": {
"capabilities": [
"run"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::run",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"ObservationCorrelation": {
"additionalProperties": false,
"properties": {
"deployment_id": {
"type": "string"
},
"operation_id": {
"type": "string"
},
"system": {
"type": "string"
}
},
"required": [
"deployment_id",
"operation_id",
"system"
],
"type": "object"
},
"ObservationDecision": {
"enum": [
"observe_only"
],
"type": "string"
},
"ObservationEnvironment": {
"enum": [
"demonstration"
],
"type": "string"
},
"ObservationMode": {
"additionalProperties": false,
"properties": {
"decision": {
"$ref": "#/definitions/ObservationDecision"
},
"environment": {
"$ref": "#/definitions/ObservationEnvironment"
}
},
"required": [
"decision",
"environment"
],
"type": "object"
},
"ObservationPlanIdentity": {
"additionalProperties": false,
"properties": {
"catalog_sha256": {
"type": "string"
},
"id": {
"type": "string"
},
"revision": {
"type": "string"
},
"sha256": {
"type": "string"
}
},
"required": [
"catalog_sha256",
"id",
"revision",
"sha256"
],
"type": "object"
},
"ObservationRunContract": {
"additionalProperties": false,
"properties": {
"attempt": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"correlation": {
"$ref": "#/definitions/ObservationCorrelation"
},
"mode": {
"$ref": "#/definitions/ObservationMode"
},
"plan": {
"$ref": "#/definitions/ObservationPlanIdentity"
},
"runner": {
"$ref": "#/definitions/RunnerIdentity"
},
"selected_cases": {
"items": {
"$ref": "#/definitions/ObservationSelectedCase"
},
"type": "array"
},
"target": {
"$ref": "#/definitions/ObservationTargetIdentity"
}
},
"required": [
"attempt",
"correlation",
"mode",
"plan",
"runner",
"selected_cases",
"target"
],
"type": "object"
},
"ObservationSelectedCase": {
"additionalProperties": false,
"properties": {
"case_id": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"inputs_sha256": {
"type": "string"
},
"scenario_id": {
"$ref": "#/definitions/ScenarioKey"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"case_id",
"contract_sha256",
"inputs_sha256",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"ObservationTargetIdentity": {
"additionalProperties": false,
"properties": {
"application": {
"type": "string"
},
"stack": {
"$ref": "#/definitions/StackIdentity"
},
"version": {
"type": "string"
}
},
"required": [
"application",
"stack",
"version"
],
"type": "object"
},
"RunnerIdentity": {
"additionalProperties": false,
"properties": {
"name": {
"type": "string"
},
"revision": {
"type": "string"
},
"version": {
"type": "string"
}
},
"required": [
"name",
"revision",
"version"
],
"type": "object"
},
"ScenarioKey": {
"type": "string"
},
"StackIdentity": {
"oneOf": [
{
"properties": {
"mode": {
"enum": [
"source"
],
"type": "string"
},
"workers_repository": {
"type": "string"
},
"workers_revision": {
"type": "string"
}
},
"required": [
"mode",
"workers_repository",
"workers_revision"
],
"type": "object"
},
{
"properties": {
"mode": {
"enum": [
"registry"
],
"type": "string"
},
"stack_lock_digest": {
"type": "string"
},
"stack_versions": {
"additionalProperties": {
"type": "string"
},
"type": "object"
}
},
"required": [
"mode",
"stack_lock_digest",
"stack_versions"
],
"type": "object"
}
]
}
},
"properties": {
"audit_model": {
"default": null,
"description": "Opt-in behavioral audit analyzer; supply model and provider together.",
"type": [
"string",
"null"
]
},
"audit_provider": {
"default": null,
"type": [
"string",
"null"
]
},
"idempotency_key": {
"type": "string"
},
"judge_model": {
"default": null,
"description": "Auxiliary model for Markdown scenarios only (validators, instruction adherence, setup and cleanup); supply model and provider together.",
"type": [
"string",
"null"
]
},
"judge_provider": {
"default": null,
"type": [
"string",
"null"
]
},
"label": {
"default": "",
"type": "string"
},
"lane": {
"default": "local",
"type": "string"
},
"model": {
"type": "string"
},
"progress_interval_seconds": {
"default": 15,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"provider": {
"type": "string"
},
"rotating_seeds": {
"default": [],
"items": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"type": "array"
},
"run_contract": {
"anyOf": [
{
"$ref": "#/definitions/ObservationRunContract"
},
{
"type": "null"
}
]
},
"runs": {
"default": 1,
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenarios": {
"default": [],
"items": {
"$ref": "#/definitions/ScenarioKey"
},
"type": "array"
},
"seed": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"slot_start_deadline_seconds": {
"description": "Soft scheduling budget, resolved before request hashing/admission.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"default": 1,
"format": "uint8",
"minimum": 0,
"type": "integer"
}
},
"required": [
"idempotency_key",
"model",
"provider"
],
"title": "RunRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ExecutionPhase": {
"enum": [
"requested",
"admitted",
"preflighting",
"materializing",
"setting_up",
"executing",
"collecting",
"evaluating",
"persisting",
"cleaning_up",
"finalizing",
"completed",
"failed",
"cancelled",
"needs_reconciliation",
"unsupported"
],
"type": "string"
}
},
"properties": {
"duplicate": {
"type": "boolean"
},
"execution_id": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
},
"request_sha256": {
"type": "string"
},
"run_contract_sha256": {
"type": [
"string",
"null"
]
}
},
"required": [
"duplicate",
"execution_id",
"phase",
"request_sha256"
],
"title": "RunAccepted",
"type": "object"
}
},
{
"description": "Explain how the Harness can author a local Markdown E2E test, including the exact template, validation rules, persistence behavior, and separate create/list/run workflow. Call this before e2e::scenarios-create.",
"metadata": {
"agent": {
"audience": "harness",
"category": "e2e_local_test_authoring",
"operation": "understand_local_test_authoring",
"starts_execution": false,
"writes_local_state": false
},
"contract": {
"capabilities": [
"scenarios-authoring-guide"
],
"name": "e2e-control-plane"
},
"internal": false
},
"name": "e2e::scenarios-authoring-guide",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"title": "ScenarioAuthoringGuideRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"create_function": {
"description": "Function that validates and persists one new local definition.",
"type": "string"
},
"creation_starts_execution": {
"description": "Always false: persistence and execution are separate operations.",
"type": "boolean"
},
"file_name_rules": {
"description": "File-name requirements enforced by the worker.",
"items": {
"type": "string"
},
"type": "array"
},
"list_function": {
"description": "Function that returns the complete built-in and local scenario catalog.",
"type": "string"
},
"max_source_bytes": {
"description": "Maximum accepted UTF-8 Markdown source size.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"overwrites_existing_file": {
"description": "Existing files are never silently replaced.",
"type": "boolean"
},
"required_h2_sections": {
"description": "Required H2 headings, in their exact order.",
"items": {
"type": "string"
},
"type": "array"
},
"run_function": {
"description": "Separate function used only when the user explicitly asks to execute.",
"type": "string"
},
"schema": {
"description": "Versioned shape of this guidance response.",
"type": "string"
},
"storage_directory": {
"description": "Worker-data-relative directory where definitions are persisted.",
"type": "string"
},
"summary": {
"description": "Short statement of what local authoring is for.",
"type": "string"
},
"template": {
"description": "Copy-ready valid Markdown source.",
"type": "string"
},
"validation_rules": {
"description": "Rules for weighted H3 validation criteria.",
"items": {
"type": "string"
},
"type": "array"
},
"workflow": {
"description": "Recommended safe sequence for an agent authoring a local test.",
"items": {
"type": "string"
},
"type": "array"
}
},
"required": [
"create_function",
"creation_starts_execution",
"file_name_rules",
"list_function",
"max_source_bytes",
"overwrites_existing_file",
"required_h2_sections",
"run_function",
"schema",
"storage_directory",
"summary",
"template",
"validation_rules",
"workflow"
],
"title": "ScenarioAuthoringGuideResponse",
"type": "object"
}
},
{
"description": "Create one local Markdown E2E test after validating its complete authoring contract. This persists a definition outside Git and never starts an execution; call e2e::run separately only when execution is explicitly requested.",
"metadata": {
"agent": {
"audience": "harness",
"category": "e2e_local_test_authoring",
"operation": "create_local_test",
"starts_execution": false,
"writes_local_state": true
},
"contract": {
"capabilities": [
"scenarios-create"
],
"name": "e2e-control-plane"
},
"internal": false
},
"name": "e2e::scenarios-create",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"file_name": {
"description": "One UTF-8 `.md` file name, without a directory. Use letters, numbers, spaces, hyphens, or underscores; its stem becomes `local_<safe_id>`. Existing files are rejected instead of overwritten.",
"type": "string"
},
"source": {
"description": "Complete Markdown test definition. It must contain one H1 followed by the required H2 sections in order and contain positive `### Name (N%)` validations totaling exactly 100%.",
"type": "string"
}
},
"required": [
"file_name",
"source"
],
"title": "LocalScenarioCreateRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ArtifactExpectation": {
"properties": {
"id": {
"type": "string"
},
"kind": {
"type": "string"
},
"max_size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"media_type": {
"type": "string"
},
"schema": true
},
"required": [
"id",
"kind",
"max_size_bytes",
"media_type",
"schema"
],
"type": "object"
},
"ComplexityClassification": {
"properties": {
"method": {
"$ref": "#/definitions/ComplexityMethod"
},
"profile": {
"$ref": "#/definitions/ComplexityProfile"
},
"tier": {
"$ref": "#/definitions/ComplexityTier"
}
},
"required": [
"method",
"profile",
"tier"
],
"type": "object"
},
"ComplexityMethod": {
"enum": [
"capability_v2"
],
"type": "string"
},
"ComplexityProfile": {
"properties": {
"agent_owned_decomposition": {
"default": false,
"type": "boolean"
},
"ambiguity_level": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"artifact_count": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"coherent_long_horizon": {
"default": false,
"type": "boolean"
},
"compensable_mutations": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"coordination_edges": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"dependency_depth": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"durable_resume_cycles": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"external_systems": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"material_invalidation_events": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"parallel_branches": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"planning_depth": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"replan_loops": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"state_transitions": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"validation_loops": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"wake_cycles": {
"format": "uint8",
"minimum": 0,
"type": "integer"
}
},
"required": [
"ambiguity_level",
"artifact_count",
"coordination_edges",
"dependency_depth",
"external_systems",
"parallel_branches",
"planning_depth",
"state_transitions",
"validation_loops",
"wake_cycles"
],
"type": "object"
},
"ComplexityTier": {
"enum": [
"l0_atomic",
"l1_sequential",
"l2_stateful",
"l3_concurrent",
"l4_coordinated",
"l5_adaptive"
],
"type": "string"
},
"DeliverableContract": {
"properties": {
"artifacts": {
"items": {
"$ref": "#/definitions/ArtifactExpectation"
},
"type": "array"
},
"capture_before_cleanup": {
"type": "boolean"
},
"invariants": {
"items": {
"$ref": "#/definitions/InvariantSpec"
},
"type": "array"
},
"provenance_required": {
"type": "boolean"
}
},
"required": [
"artifacts",
"capture_before_cleanup",
"invariants",
"provenance_required"
],
"type": "object"
},
"ExecutionPolicy": {
"properties": {
"max_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_total_tokens": {
"description": "Shared Harness token budget. `None` leaves the Harness budget unbounded for this scenario.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_turns": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"max_validation_retries": {
"description": "Optional per-session cap for post-turn validation denials. `None` preserves the Harness-configured default.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stuck_timeout_seconds": {
"description": "Stop only after this many seconds without observable useful progress. Large scenarios have no fixed wall-clock deadline.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"max_turns",
"stuck_timeout_seconds"
],
"type": "object"
},
"ExecutionRealism": {
"enum": [
"synthetic",
"realistic_simulator",
"frozen_real_artifact"
],
"type": "string"
},
"HumanHorizon": {
"properties": {
"basis": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizonBasis"
}
],
"default": "unknown"
},
"max_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"min_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"HumanHorizonBasis": {
"enum": [
"unknown",
"author_estimate",
"measured"
],
"type": "string"
},
"InvariantSpec": {
"properties": {
"description": {
"type": "string"
},
"id": {
"type": "string"
}
},
"required": [
"description",
"id"
],
"type": "object"
},
"ScenarioCharacterization": {
"properties": {
"human_horizon": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizon"
}
],
"default": {
"basis": "unknown"
}
},
"realism": {
"allOf": [
{
"$ref": "#/definitions/ScenarioRealism"
}
],
"default": {
"execution": "synthetic",
"shadow": "none"
}
}
},
"type": "object"
},
"ScenarioDescriptor": {
"properties": {
"author_version": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"behavior_sha256": {
"type": [
"string",
"null"
]
},
"case_id": {
"type": "string"
},
"characterization": {
"$ref": "#/definitions/ScenarioCharacterization"
},
"classification": {
"$ref": "#/definitions/ComplexityClassification"
},
"compiled_sha256": {
"type": [
"string",
"null"
]
},
"complexity": {
"$ref": "#/definitions/ComplexityProfile"
},
"contract_sha256": {
"type": "string"
},
"deliverable_contract": {
"$ref": "#/definitions/DeliverableContract"
},
"inputs_sha256": {
"type": "string"
},
"origin": {
"$ref": "#/definitions/ScenarioOrigin"
},
"required_capabilities": {
"items": {
"type": "string"
},
"type": "array"
},
"resource_envelope": {
"$ref": "#/definitions/ScenarioResourceEnvelope"
},
"scenario_id": {
"$ref": "#/definitions/ScenarioKey"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"source_path": {
"type": [
"string",
"null"
]
},
"source_sha256": {
"type": [
"string",
"null"
]
},
"title": {
"type": [
"string",
"null"
]
}
},
"required": [
"case_id",
"characterization",
"classification",
"complexity",
"contract_sha256",
"deliverable_contract",
"inputs_sha256",
"origin",
"required_capabilities",
"resource_envelope",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"ScenarioKey": {
"type": "string"
},
"ScenarioOrigin": {
"enum": [
"built_in",
"markdown",
"local"
],
"type": "string"
},
"ScenarioRealism": {
"properties": {
"execution": {
"allOf": [
{
"$ref": "#/definitions/ExecutionRealism"
}
],
"default": "synthetic"
},
"shadow": {
"allOf": [
{
"$ref": "#/definitions/ShadowMode"
}
],
"default": "none"
}
},
"type": "object"
},
"ScenarioResourceEnvelope": {
"properties": {
"execution": {
"$ref": "#/definitions/ExecutionPolicy"
},
"workflow": {
"anyOf": [
{
"$ref": "#/definitions/WorkflowResourceEnvelope"
},
{
"type": "null"
}
]
}
},
"required": [
"execution"
],
"type": "object"
},
"ShadowMode": {
"enum": [
"none",
"read_only"
],
"type": "string"
},
"WorkflowResourceEnvelope": {
"properties": {
"max_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"max_nodes": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"max_parallel": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"max_total_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"step_timeout_seconds": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"workflow_timeout_seconds": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"max_nodes",
"max_parallel",
"step_timeout_seconds",
"technical_retries",
"workflow_timeout_seconds"
],
"type": "object"
}
},
"properties": {
"scenario": {
"$ref": "#/definitions/ScenarioDescriptor"
}
},
"required": [
"scenario"
],
"title": "LocalScenarioCreateResponse",
"type": "object"
}
},
{
"description": "List every built-in, committed Markdown, and local E2E test with its immutable contract, origin, capabilities, complexity, and materialized case. Call e2e::scenarios-authoring-guide before authoring a new local test.",
"metadata": {
"agent": {
"audience": "harness",
"category": "e2e_local_test_authoring",
"operation": "list_tests",
"starts_execution": false,
"writes_local_state": false
},
"contract": {
"capabilities": [
"scenarios-list"
],
"name": "e2e-control-plane"
},
"internal": false
},
"name": "e2e::scenarios-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"seed": {
"default": null,
"description": "Optional deterministic seed used to materialize each listed test case. Omit it to use each scenario's canonical seed.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"title": "ScenariosListRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ArtifactExpectation": {
"properties": {
"id": {
"type": "string"
},
"kind": {
"type": "string"
},
"max_size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"media_type": {
"type": "string"
},
"schema": true
},
"required": [
"id",
"kind",
"max_size_bytes",
"media_type",
"schema"
],
"type": "object"
},
"ComplexityClassification": {
"properties": {
"method": {
"$ref": "#/definitions/ComplexityMethod"
},
"profile": {
"$ref": "#/definitions/ComplexityProfile"
},
"tier": {
"$ref": "#/definitions/ComplexityTier"
}
},
"required": [
"method",
"profile",
"tier"
],
"type": "object"
},
"ComplexityMethod": {
"enum": [
"capability_v2"
],
"type": "string"
},
"ComplexityProfile": {
"properties": {
"agent_owned_decomposition": {
"default": false,
"type": "boolean"
},
"ambiguity_level": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"artifact_count": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"coherent_long_horizon": {
"default": false,
"type": "boolean"
},
"compensable_mutations": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"coordination_edges": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"dependency_depth": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"durable_resume_cycles": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"external_systems": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"material_invalidation_events": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"parallel_branches": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"planning_depth": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"replan_loops": {
"default": 0,
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"state_transitions": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"validation_loops": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"wake_cycles": {
"format": "uint8",
"minimum": 0,
"type": "integer"
}
},
"required": [
"ambiguity_level",
"artifact_count",
"coordination_edges",
"dependency_depth",
"external_systems",
"parallel_branches",
"planning_depth",
"state_transitions",
"validation_loops",
"wake_cycles"
],
"type": "object"
},
"ComplexityTier": {
"enum": [
"l0_atomic",
"l1_sequential",
"l2_stateful",
"l3_concurrent",
"l4_coordinated",
"l5_adaptive"
],
"type": "string"
},
"DeliverableContract": {
"properties": {
"artifacts": {
"items": {
"$ref": "#/definitions/ArtifactExpectation"
},
"type": "array"
},
"capture_before_cleanup": {
"type": "boolean"
},
"invariants": {
"items": {
"$ref": "#/definitions/InvariantSpec"
},
"type": "array"
},
"provenance_required": {
"type": "boolean"
}
},
"required": [
"artifacts",
"capture_before_cleanup",
"invariants",
"provenance_required"
],
"type": "object"
},
"ExecutionPolicy": {
"properties": {
"max_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_total_tokens": {
"description": "Shared Harness token budget. `None` leaves the Harness budget unbounded for this scenario.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_turns": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"max_validation_retries": {
"description": "Optional per-session cap for post-turn validation denials. `None` preserves the Harness-configured default.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stuck_timeout_seconds": {
"description": "Stop only after this many seconds without observable useful progress. Large scenarios have no fixed wall-clock deadline.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"max_turns",
"stuck_timeout_seconds"
],
"type": "object"
},
"ExecutionRealism": {
"enum": [
"synthetic",
"realistic_simulator",
"frozen_real_artifact"
],
"type": "string"
},
"HumanHorizon": {
"properties": {
"basis": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizonBasis"
}
],
"default": "unknown"
},
"max_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"min_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"HumanHorizonBasis": {
"enum": [
"unknown",
"author_estimate",
"measured"
],
"type": "string"
},
"InvariantSpec": {
"properties": {
"description": {
"type": "string"
},
"id": {
"type": "string"
}
},
"required": [
"description",
"id"
],
"type": "object"
},
"RunnerIdentity": {
"additionalProperties": false,
"properties": {
"name": {
"type": "string"
},
"revision": {
"type": "string"
},
"version": {
"type": "string"
}
},
"required": [
"name",
"revision",
"version"
],
"type": "object"
},
"ScenarioCharacterization": {
"properties": {
"human_horizon": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizon"
}
],
"default": {
"basis": "unknown"
}
},
"realism": {
"allOf": [
{
"$ref": "#/definitions/ScenarioRealism"
}
],
"default": {
"execution": "synthetic",
"shadow": "none"
}
}
},
"type": "object"
},
"ScenarioDescriptor": {
"properties": {
"author_version": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"behavior_sha256": {
"type": [
"string",
"null"
]
},
"case_id": {
"type": "string"
},
"characterization": {
"$ref": "#/definitions/ScenarioCharacterization"
},
"classification": {
"$ref": "#/definitions/ComplexityClassification"
},
"compiled_sha256": {
"type": [
"string",
"null"
]
},
"complexity": {
"$ref": "#/definitions/ComplexityProfile"
},
"contract_sha256": {
"type": "string"
},
"deliverable_contract": {
"$ref": "#/definitions/DeliverableContract"
},
"inputs_sha256": {
"type": "string"
},
"origin": {
"$ref": "#/definitions/ScenarioOrigin"
},
"required_capabilities": {
"items": {
"type": "string"
},
"type": "array"
},
"resource_envelope": {
"$ref": "#/definitions/ScenarioResourceEnvelope"
},
"scenario_id": {
"$ref": "#/definitions/ScenarioKey"
},
"scenario_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"source_path": {
"type": [
"string",
"null"
]
},
"source_sha256": {
"type": [
"string",
"null"
]
},
"title": {
"type": [
"string",
"null"
]
}
},
"required": [
"case_id",
"characterization",
"classification",
"complexity",
"contract_sha256",
"deliverable_contract",
"inputs_sha256",
"origin",
"required_capabilities",
"resource_envelope",
"scenario_id",
"scenario_version",
"seed"
],
"type": "object"
},
"ScenarioKey": {
"type": "string"
},
"ScenarioOrigin": {
"enum": [
"built_in",
"markdown",
"local"
],
"type": "string"
},
"ScenarioRealism": {
"properties": {
"execution": {
"allOf": [
{
"$ref": "#/definitions/ExecutionRealism"
}
],
"default": "synthetic"
},
"shadow": {
"allOf": [
{
"$ref": "#/definitions/ShadowMode"
}
],
"default": "none"
}
},
"type": "object"
},
"ScenarioResourceEnvelope": {
"properties": {
"execution": {
"$ref": "#/definitions/ExecutionPolicy"
},
"workflow": {
"anyOf": [
{
"$ref": "#/definitions/WorkflowResourceEnvelope"
},
{
"type": "null"
}
]
}
},
"required": [
"execution"
],
"type": "object"
},
"ShadowMode": {
"enum": [
"none",
"read_only"
],
"type": "string"
},
"WorkflowResourceEnvelope": {
"properties": {
"max_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"max_nodes": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"max_parallel": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"max_total_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"step_timeout_seconds": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"workflow_timeout_seconds": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"max_nodes",
"max_parallel",
"step_timeout_seconds",
"technical_retries",
"workflow_timeout_seconds"
],
"type": "object"
}
},
"properties": {
"catalog_sha256": {
"type": "string"
},
"runner": {
"$ref": "#/definitions/RunnerIdentity"
},
"scenarios": {
"items": {
"$ref": "#/definitions/ScenarioDescriptor"
},
"type": "array"
},
"schema": {
"type": "string"
}
},
"required": [
"catalog_sha256",
"runner",
"scenarios",
"schema"
],
"title": "ScenariosListResponse",
"type": "object"
}
},
{
"description": "Read the durable phase and progress of an E2E execution.",
"metadata": {
"contract": {
"capabilities": [
"status"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::status",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
}
},
"required": [
"execution_id"
],
"title": "ExecutionRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ActiveAttempt": {
"properties": {
"attempt_id": {
"type": "string"
},
"resume_state_path": {
"type": [
"string",
"null"
]
},
"resume_state_sha256": {
"type": [
"string",
"null"
]
},
"run_id": {
"default": "",
"type": "string"
},
"scenario_id": {
"$ref": "#/definitions/ScenarioKey"
},
"session_id": {
"type": "string"
}
},
"required": [
"attempt_id",
"scenario_id",
"session_id"
],
"type": "object"
},
"ExecutionPhase": {
"enum": [
"requested",
"admitted",
"preflighting",
"materializing",
"setting_up",
"executing",
"collecting",
"evaluating",
"persisting",
"cleaning_up",
"finalizing",
"completed",
"failed",
"cancelled",
"needs_reconciliation",
"unsupported"
],
"type": "string"
},
"JournalProgress": {
"properties": {
"active_attempt_id": {
"type": [
"string",
"null"
]
},
"attempts_finished": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"attempts_started": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"committed_events": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"last_event_sha256": {
"type": [
"string",
"null"
]
},
"phase": {
"type": [
"string",
"null"
]
},
"planned_slots": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"runs_committed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"slots_deferred": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"subject_observations_committed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"terminal": {
"type": "boolean"
},
"terminal_reason": {
"type": [
"string",
"null"
]
},
"terminal_state": {
"anyOf": [
{
"$ref": "#/definitions/JournalTerminalState"
},
{
"type": "null"
}
]
}
},
"required": [
"attempts_finished",
"attempts_started",
"committed_events",
"planned_slots",
"runs_committed",
"slots_deferred",
"subject_observations_committed",
"terminal"
],
"type": "object"
},
"JournalTerminalState": {
"enum": [
"completed",
"failed",
"cancelled",
"needs_reconciliation",
"unsupported"
],
"type": "string"
},
"PhaseTransition": {
"properties": {
"at": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
},
"reason": {
"type": "string"
}
},
"required": [
"at",
"phase",
"reason"
],
"type": "object"
},
"ScenarioKey": {
"type": "string"
}
},
"properties": {
"active_attempt": {
"anyOf": [
{
"$ref": "#/definitions/ActiveAttempt"
},
{
"type": "null"
}
]
},
"cancel_requested": {
"type": "boolean"
},
"error": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"journal_progress": {
"$ref": "#/definitions/JournalProgress"
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
},
"terminal": {
"type": "boolean"
},
"transitions": {
"items": {
"$ref": "#/definitions/PhaseTransition"
},
"type": "array"
},
"updated_at": {
"type": "string"
}
},
"required": [
"cancel_requested",
"execution_id",
"journal_progress",
"phase",
"terminal",
"transitions",
"updated_at"
],
"title": "StatusResponse",
"type": "object"
}
},
{
"description": "Serve the Harness E2E injectable Console assets.",
"metadata": {
"internal": true
},
"name": "e2e::ui-content",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"path": {
"type": "string"
}
},
"required": [
"path"
],
"title": "UiContentRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"content": {
"type": "string"
},
"content_type": {
"type": "string"
}
},
"required": [
"content",
"content_type"
],
"title": "UiContentResponse",
"type": "object"
}
}
],
"triggers": [
{
"description": "A local E2E execution started, produced progress, or reached a terminal state.",
"invocation_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"title": "EventBindingConfig",
"type": "object"
},
"metadata": {},
"name": "e2e::dashboard::changed",
"return_schema": {}
}
]
}