eval::assert::exact
functionBuilt-in deterministic evaluator: deep-compare output with arguments.expected.
evaluation_idstring*required
metricsobject*required
by_sessionobject[]*required
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
contextany of
The session's latest per-generation context snapshot (categories, budget, usage) — absent for sessions that have not generated since snapshots landed.
… expand 2 nestedcollapse
any of (2)
variant 1
categoriesobject*required
Where the request's tokens sit. Categories are assembly-time estimates; `hook_guidance` is the measured growth after assembly (pre-generate hook appends and orphan-repair patches), 0 when the request left assembly unchanged.
… expand 5 nestedcollapse
hook_guidanceinteger· uint64min 0
messagesobject*required
Estimated tokens of the assembled window's messages, by role.
… expand 4 nestedcollapse
assistantinteger· uint64*requiredmin 0
custominteger· uint64*requiredmin 0
function_resultinteger· uint64*requiredmin 0
userinteger· uint64*requiredmin 0
overheadinteger· uint64*requiredmin 0
Provider framing plus response_format / provider_options fields.
system_promptinteger· uint64*requiredmin 0
Final assembled system prompt: mode paragraph, identity, per-step aids, and any compaction summary section.
toolsinteger· uint64*requiredmin 0
Function schemas exposed to the model.
compactedboolean*required
effective_max_output_tokensinteger· uint64*requiredmin 0
Output allocation `usable` was derived against.
estimatorstring
Which estimator produced the numbers (`heuristic` until the context-manager resolves a real tokenizer). Absent when the context-manager predates the breakdown response.
freeinteger· uint64*requiredmin 0
session_cost_usdnumber· double
Running cost of the whole session in USD, accumulated across every generation step. `usage.cost_usd` is one step's bill — on providers with steep cache discounts the per-step number swings two orders of magnitude, so a chip showing it alone reads as a bouncing total.
session_idstring*required
stepinteger· uint64*requiredmin 0
summarized_head_tokensinteger· uint64min 0
timestampinteger· int64*required
totalinteger· uint64*requiredmin 0
Final request estimate: categories plus post-assembly growth.
usableinteger· uint64*requiredmin 0
The input budget the window was fit into.
usageany of
Actual provider usage for this generation, stamped after the terminal frame; absent when the provider returned none (or the generation never completed).
… expand 2 nestedcollapse
any of (2)
variant 1
cache_readinteger· uint64min 0
cache_writeinteger· uint64min 0
inputinteger· uint64min 0
outputinteger· uint64min 0
reasoninginteger· uint64min 0
depthinteger· uint32*requiredmin 0
function_call_errorsinteger· uint64*requiredmin 0
function_callsinteger· uint64*requiredmin 0
input_tokensinteger· uint64min 0
output_tokensinteger· uint64min 0
reasoning_tokensinteger· uint64min 0
session_idstring*required
turnsinteger· uint64*requiredmin 0
root_session_idstring*required
totalsobject*required
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
function_call_errorsinteger· uint64*requiredmin 0
function_callsinteger· uint64*requiredmin 0
input_tokensinteger· uint64min 0
output_tokensinteger· uint64min 0
reasoning_tokensinteger· uint64min 0
sessionsinteger· uint64*requiredmin 0
turnsinteger· uint64*requiredmin 0
tracesany of
Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not.
any of (2)
variant 1
by_sessionobject[]*required
… expand 7 nestedcollapse
depthinteger· uint32*requiredmin 0
duration_msinteger· uint64*requiredmin 0
Elapsed window from the session's first observed span to its last.
error_span_countinteger· uint64*requiredmin 0
session_idstring*required
span_countinteger· uint64*requiredmin 0
trace_countinteger· uint64*requiredmin 0
duration_msinteger· uint64*requiredmin 0
Elapsed window from the first observed span to the last observed span.
error_span_countinteger· uint64*requiredmin 0
span_countinteger· uint64*requiredmin 0
trace_countinteger· uint64*requiredmin 0
Distinct traces across the root session and all descendants.
rolestring*requiredenum: control, treatment
session_idstring*required
Built-in deterministic evaluator: compare text after normalizing case, whitespace, and surrounding punctuation.
evaluation_idstring*required
metricsobject*required
by_sessionobject[]*required
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
contextany of
The session's latest per-generation context snapshot (categories, budget, usage) — absent for sessions that have not generated since snapshots landed.
… expand 2 nestedcollapse
any of (2)
variant 1
categoriesobject*required
Where the request's tokens sit. Categories are assembly-time estimates; `hook_guidance` is the measured growth after assembly (pre-generate hook appends and orphan-repair patches), 0 when the request left assembly unchanged.
… expand 5 nestedcollapse
hook_guidanceinteger· uint64min 0
messagesobject*required
Estimated tokens of the assembled window's messages, by role.
… expand 4 nestedcollapse
assistantinteger· uint64*requiredmin 0
custominteger· uint64*requiredmin 0
function_resultinteger· uint64*requiredmin 0
userinteger· uint64*requiredmin 0
overheadinteger· uint64*requiredmin 0
Provider framing plus response_format / provider_options fields.
system_promptinteger· uint64*requiredmin 0
Final assembled system prompt: mode paragraph, identity, per-step aids, and any compaction summary section.
toolsinteger· uint64*requiredmin 0
Function schemas exposed to the model.
compactedboolean*required
effective_max_output_tokensinteger· uint64*requiredmin 0
Output allocation `usable` was derived against.
estimatorstring
Which estimator produced the numbers (`heuristic` until the context-manager resolves a real tokenizer). Absent when the context-manager predates the breakdown response.
freeinteger· uint64*requiredmin 0
session_cost_usdnumber· double
Running cost of the whole session in USD, accumulated across every generation step. `usage.cost_usd` is one step's bill — on providers with steep cache discounts the per-step number swings two orders of magnitude, so a chip showing it alone reads as a bouncing total.
session_idstring*required
stepinteger· uint64*requiredmin 0
summarized_head_tokensinteger· uint64min 0
timestampinteger· int64*required
totalinteger· uint64*requiredmin 0
Final request estimate: categories plus post-assembly growth.
usableinteger· uint64*requiredmin 0
The input budget the window was fit into.
usageany of
Actual provider usage for this generation, stamped after the terminal frame; absent when the provider returned none (or the generation never completed).
… expand 2 nestedcollapse
any of (2)
variant 1
cache_readinteger· uint64min 0
cache_writeinteger· uint64min 0
inputinteger· uint64min 0
outputinteger· uint64min 0
reasoninginteger· uint64min 0
depthinteger· uint32*requiredmin 0
function_call_errorsinteger· uint64*requiredmin 0
function_callsinteger· uint64*requiredmin 0
input_tokensinteger· uint64min 0
output_tokensinteger· uint64min 0
reasoning_tokensinteger· uint64min 0
session_idstring*required
turnsinteger· uint64*requiredmin 0
root_session_idstring*required
totalsobject*required
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
function_call_errorsinteger· uint64*requiredmin 0
function_callsinteger· uint64*requiredmin 0
input_tokensinteger· uint64min 0
output_tokensinteger· uint64min 0
reasoning_tokensinteger· uint64min 0
sessionsinteger· uint64*requiredmin 0
turnsinteger· uint64*requiredmin 0
tracesany of
Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not.
any of (2)
variant 1
by_sessionobject[]*required
… expand 7 nestedcollapse
depthinteger· uint32*requiredmin 0
duration_msinteger· uint64*requiredmin 0
Elapsed window from the session's first observed span to its last.
error_span_countinteger· uint64*requiredmin 0
session_idstring*required
span_countinteger· uint64*requiredmin 0
trace_countinteger· uint64*requiredmin 0
duration_msinteger· uint64*requiredmin 0
Elapsed window from the first observed span to the last observed span.
error_span_countinteger· uint64*requiredmin 0
span_countinteger· uint64*requiredmin 0
trace_countinteger· uint64*requiredmin 0
Distinct traces across the root session and all descendants.
rolestring*requiredenum: control, treatment
session_idstring*required
Cancel an evaluation and its active harness turn.
evaluation_idstring*required
cancelledboolean*required
statusstring*requiredenum: queued, running, completed, failed, cancelled
eval::compare-sessions
functionCompare 2–5 existing root sessions from live session, lifecycle, and Harness metrics. The response contains objective values and arithmetic deltas against the chosen baseline; it does not score, rank, validate equivalence, or persist a snapshot.
baseline_session_idstring*required
session_idsstring[]*required2…5 items
baseline_session_idstring*required
captured_atinteger· int64*required
schema_versionstring*required
sessionsobject[]*required
deltasobject
One entry per objective numeric metric. A missing value is represented by null in either delta field, never by zero.
lifecycleobject*required
completeboolean
Null means the metrics read was unavailable.
expects_wakeboolean
Null means the status read was unavailable.
session_statusstring*requiredenum: idle, working, done, error
terminalboolean
Null means `harness::status` did not provide a turn record.
turn_statusany of
… expand 2 nestedcollapse
any of (2)
variant 1
valuestringenum: running, awaiting_functions, completed, cancelled, failed
metricsany of
any of (2)
variant 1
by_sessionobject[]*required
… expand 13 nestedcollapse
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
contextany of
The session's latest per-generation context snapshot (categories, budget, usage) — absent for sessions that have not generated since snapshots landed.
… expand 2 nestedcollapse
any of (2)
variant 1
categoriesobject*required
Where the request's tokens sit. Categories are assembly-time estimates; `hook_guidance` is the measured growth after assembly (pre-generate hook appends and orphan-repair patches), 0 when the request left assembly unchanged.
… expand 5 nestedcollapse
hook_guidanceinteger· uint64min 0
messagesobject*required
Estimated tokens of the assembled window's messages, by role.
… expand 4 nestedcollapse
assistantinteger· uint64*requiredmin 0
custominteger· uint64*requiredmin 0
function_resultinteger· uint64*requiredmin 0
userinteger· uint64*requiredmin 0
overheadinteger· uint64*requiredmin 0
Provider framing plus response_format / provider_options fields.
system_promptinteger· uint64*requiredmin 0
Final assembled system prompt: mode paragraph, identity, per-step aids, and any compaction summary section.
toolsinteger· uint64*requiredmin 0
Function schemas exposed to the model.
compactedboolean*required
effective_max_output_tokensinteger· uint64*requiredmin 0
Output allocation `usable` was derived against.
estimatorstring
Which estimator produced the numbers (`heuristic` until the context-manager resolves a real tokenizer). Absent when the context-manager predates the breakdown response.
freeinteger· uint64*requiredmin 0
session_cost_usdnumber· double
Running cost of the whole session in USD, accumulated across every generation step. `usage.cost_usd` is one step's bill — on providers with steep cache discounts the per-step number swings two orders of magnitude, so a chip showing it alone reads as a bouncing total.
session_idstring*required
stepinteger· uint64*requiredmin 0
summarized_head_tokensinteger· uint64min 0
timestampinteger· int64*required
totalinteger· uint64*requiredmin 0
Final request estimate: categories plus post-assembly growth.
usableinteger· uint64*requiredmin 0
The input budget the window was fit into.
usageany of
Actual provider usage for this generation, stamped after the terminal frame; absent when the provider returned none (or the generation never completed).
… expand 2 nestedcollapse
any of (2)
variant 1
cache_readinteger· uint64min 0
cache_writeinteger· uint64min 0
inputinteger· uint64min 0
outputinteger· uint64min 0
reasoninginteger· uint64min 0
depthinteger· uint32*requiredmin 0
function_call_errorsinteger· uint64*requiredmin 0
function_callsinteger· uint64*requiredmin 0
input_tokensinteger· uint64min 0
output_tokensinteger· uint64min 0
reasoning_tokensinteger· uint64min 0
session_idstring*required
turnsinteger· uint64*requiredmin 0
root_session_idstring*required
totalsobject*required
… expand 10 nestedcollapse
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
function_call_errorsinteger· uint64*requiredmin 0
function_callsinteger· uint64*requiredmin 0
input_tokensinteger· uint64min 0
output_tokensinteger· uint64min 0
reasoning_tokensinteger· uint64min 0
sessionsinteger· uint64*requiredmin 0
turnsinteger· uint64*requiredmin 0
tracesany of
Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not.
… expand 2 nestedcollapse
any of (2)
variant 1
by_sessionobject[]*required
… expand 7 nestedcollapse
depthinteger· uint32*requiredmin 0
duration_msinteger· uint64*requiredmin 0
Elapsed window from the session's first observed span to its last.
error_span_countinteger· uint64*requiredmin 0
session_idstring*required
span_countinteger· uint64*requiredmin 0
trace_countinteger· uint64*requiredmin 0
duration_msinteger· uint64*requiredmin 0
Elapsed window from the first observed span to the last observed span.
error_span_countinteger· uint64*requiredmin 0
span_countinteger· uint64*requiredmin 0
trace_countinteger· uint64*requiredmin 0
Distinct traces across the root session and all descendants.
sessionobject*required
message_countinteger· uint64min 0
session_idstring*required
statusstring*requiredenum: idle, working, done, error
summaryany of
any of (2)
variant 1
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
compacted_sessionsinteger· uint64min 0
context_free_tokensinteger· uint64min 0
context_occupancynumber· double
context_total_tokensinteger· uint64min 0
context_usable_tokensinteger· uint64min 0
cost_per_generation_usdnumber· double
descendantsinteger· uint64min 0
error_span_countinteger· uint64min 0
function_call_errorsinteger· uint64min 0
function_callsinteger· uint64min 0
function_error_ratenumber· double
generationsinteger· uint64min 0
input_tokensinteger· uint64min 0
max_depthinteger· uint32min 0
output_tokensinteger· uint64min 0
reasoning_tokensinteger· uint64min 0
sessionsinteger· uint64min 0
span_countinteger· uint64min 0
subject_cost_usdnumber· double
tokens_per_generationnumber· double
total_tokensinteger· uint64min 0
trace_countinteger· uint64min 0
trace_duration_msinteger· uint64min 0
Delete a terminal evaluation report and its session indexes.
evaluation_idstring*required
List recent evaluations as lightweight summaries, newest first.
limitinteger· uint32min 0
evaluationsobject[]*required
completed_atinteger· int64
created_atinteger· int64*required
dimensionstring*requiredenum: prompt, system_prompt
evaluation_idstring*required
failed_runsinteger· uint32*requiredmin 0
passed_runsinteger· uint32*requiredmin 0
statusstring*requiredenum: queued, running, completed, failed, cancelled
terminal_runsinteger· uint32*requiredmin 0
total_runsinteger· uint32*requiredmin 0
updated_atinteger· int64*required
eval::on-turn-completed
functionInternal harness turn-completed wake-up.
Repeat a terminal evaluation from its persisted request. Set reverse_order to invert the balanced A/B order. The rerun receives a new evaluation ID and fresh sessions.
evaluation_idstring*required
evaluation_idstring*required
statusstring*requiredenum: queued, running, completed, failed, cancelled
Read an evaluation result. The report is present only after terminal completion.
evaluation_idstring*required
any of (2)
variant 1
progressobject*required
control_aggregateobject*required
… expand 18 nestedcollapse
benchmarked_runsinteger· uint32*requiredmin 0
evaluated_runsinteger· uint32*requiredmin 0
median_cost_usdnumber· double
median_error_span_countinteger· uint64min 0
median_function_call_errorsinteger· uint64min 0
median_function_callsinteger· uint64min 0
median_input_tokensinteger· uint64min 0
median_output_tokensinteger· uint64min 0
median_reasoning_tokensinteger· uint64min 0
median_scorenumber· double
median_span_countinteger· uint64min 0
median_total_tokensinteger· uint64min 0
median_trace_countinteger· uint64min 0
median_trace_duration_msinteger· uint64min 0
median_wall_time_msinteger· uint64min 0
pass_ratenumber· double*required
passedinteger· uint32*requiredmin 0
runsinteger· uint32*requiredmin 0
deltaobject*required
… expand 12 nestedcollapse
median_cost_usdnumber· double
median_error_span_countnumber· double
median_function_call_errorsnumber· double
median_function_callsnumber· double
median_reasoning_tokensnumber· double
median_scorenumber· double
median_span_countnumber· double
median_total_tokensnumber· double
median_trace_countnumber· double
median_trace_duration_msnumber· double
median_wall_time_msnumber· double
pass_ratenumber· double*required
effective_execution_orderstring[]*required
order_sensitivityobject*required
… expand 3 nestedcollapse
controlobject*required
… expand 5 nestedcollapse
first_passedinteger· uint32*requiredmin 0
first_runsinteger· uint32*requiredmin 0
second_passedinteger· uint32*requiredmin 0
second_runsinteger· uint32*requiredmin 0
treatmentobject*required
… expand 5 nestedcollapse
first_passedinteger· uint32*requiredmin 0
first_runsinteger· uint32*requiredmin 0
second_passedinteger· uint32*requiredmin 0
second_runsinteger· uint32*requiredmin 0
runsobject[]*required
… expand 16 nestedcollapse
benchmarkany of
… expand 2 nestedcollapse
any of (2)
variant 1
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
error_span_countinteger· uint64min 0
function_call_errorsinteger· uint64*requiredmin 0
function_callsinteger· uint64*requiredmin 0
input_tokensinteger· uint64min 0
output_tokensinteger· uint64min 0
reasoning_tokensinteger· uint64min 0
sessionsinteger· uint64*requiredmin 0
span_countinteger· uint64min 0
total_tokensinteger· uint64min 0
trace_countinteger· uint64min 0
trace_duration_msinteger· uint64min 0
turnsinteger· uint64*requiredmin 0
wall_time_msinteger· uint64*requiredmin 0
completed_atinteger· int64
evaluationany of
… expand 2 nestedcollapse
execution_positioninteger· uint32min 0
failuresobject[]
… expand 4 nestedcollapse
phasestring*requiredenum: setup, send, await, collect, evaluate, limit, …
iterationinteger· uint32*requiredmin 0
metricsany of
… expand 2 nestedcollapse
any of (2)
variant 1
by_sessionobject[]*required
… expand 13 nestedcollapse
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
contextany of
The session's latest per-generation context snapshot (categories, budget, usage) — absent for sessions that have not generated since snapshots landed.
… expand 2 nestedcollapse
any of (2)
variant 1
categoriesobject*required
Where the request's tokens sit. Categories are assembly-time estimates; `hook_guidance` is the measured growth after assembly (pre-generate hook appends and orphan-repair patches), 0 when the request left assembly unchanged.
… expand 5 nestedcollapse
hook_guidanceinteger· uint64min 0
messagesobject*required
Estimated tokens of the assembled window's messages, by role.
… expand 4 nestedcollapse
assistantinteger· uint64*requiredmin 0
custominteger· uint64*requiredmin 0
function_resultinteger· uint64*requiredmin 0
userinteger· uint64*requiredmin 0
overheadinteger· uint64*requiredmin 0
Provider framing plus response_format / provider_options fields.
system_promptinteger· uint64*requiredmin 0
Final assembled system prompt: mode paragraph, identity, per-step aids, and any compaction summary section.
toolsinteger· uint64*requiredmin 0
Function schemas exposed to the model.
compactedboolean*required
effective_max_output_tokensinteger· uint64*requiredmin 0
Output allocation `usable` was derived against.
estimatorstring
Which estimator produced the numbers (`heuristic` until the context-manager resolves a real tokenizer). Absent when the context-manager predates the breakdown response.
freeinteger· uint64*requiredmin 0
session_cost_usdnumber· double
Running cost of the whole session in USD, accumulated across every generation step. `usage.cost_usd` is one step's bill — on providers with steep cache discounts the per-step number swings two orders of magnitude, so a chip showing it alone reads as a bouncing total.
session_idstring*required
stepinteger· uint64*requiredmin 0
summarized_head_tokensinteger· uint64min 0
timestampinteger· int64*required
totalinteger· uint64*requiredmin 0
Final request estimate: categories plus post-assembly growth.
usableinteger· uint64*requiredmin 0
The input budget the window was fit into.
usageany of
Actual provider usage for this generation, stamped after the terminal frame; absent when the provider returned none (or the generation never completed).
… expand 2 nestedcollapse
any of (2)
variant 1
cache_readinteger· uint64min 0
cache_writeinteger· uint64min 0
inputinteger· uint64min 0
outputinteger· uint64min 0
reasoninginteger· uint64min 0
depthinteger· uint32*requiredmin 0
function_call_errorsinteger· uint64*requiredmin 0
function_callsinteger· uint64*requiredmin 0
input_tokensinteger· uint64min 0
output_tokensinteger· uint64min 0
reasoning_tokensinteger· uint64min 0
session_idstring*required
turnsinteger· uint64*requiredmin 0
root_session_idstring*required
totalsobject*required
… expand 10 nestedcollapse
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
function_call_errorsinteger· uint64*requiredmin 0
function_callsinteger· uint64*requiredmin 0
input_tokensinteger· uint64min 0
output_tokensinteger· uint64min 0
reasoning_tokensinteger· uint64min 0
sessionsinteger· uint64*requiredmin 0
turnsinteger· uint64*requiredmin 0
tracesany of
Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not.
… expand 2 nestedcollapse
any of (2)
variant 1
by_sessionobject[]*required
… expand 7 nestedcollapse
depthinteger· uint32*requiredmin 0
duration_msinteger· uint64*requiredmin 0
Elapsed window from the session's first observed span to its last.
error_span_countinteger· uint64*requiredmin 0
session_idstring*required
span_countinteger· uint64*requiredmin 0
trace_countinteger· uint64*requiredmin 0
duration_msinteger· uint64*requiredmin 0
Elapsed window from the first observed span to the last observed span.
error_span_countinteger· uint64*requiredmin 0
span_countinteger· uint64*requiredmin 0
trace_countinteger· uint64*requiredmin 0
Distinct traces across the root session and all descendants.
pair_positioninteger· uint8min 0
rolestring*requiredenum: control, treatment
session_idstring*required
started_atinteger· int64*required
statusstring*requiredenum: pending, running, completed, failed, cancelled
treatment_aggregateobject*required
… expand 18 nestedcollapse
benchmarked_runsinteger· uint32*requiredmin 0
evaluated_runsinteger· uint32*requiredmin 0
median_cost_usdnumber· double
median_error_span_countinteger· uint64min 0
median_function_call_errorsinteger· uint64min 0
median_function_callsinteger· uint64min 0
median_input_tokensinteger· uint64min 0
median_output_tokensinteger· uint64min 0
median_reasoning_tokensinteger· uint64min 0
median_scorenumber· double
median_span_countinteger· uint64min 0
median_total_tokensinteger· uint64min 0
median_trace_countinteger· uint64min 0
median_trace_duration_msinteger· uint64min 0
median_wall_time_msinteger· uint64min 0
pass_ratenumber· double*required
passedinteger· uint32*requiredmin 0
runsinteger· uint32*requiredmin 0
reportany of
any of (2)
variant 1
completed_atinteger· int64*required
controlobject*required
… expand 3 nestedcollapse
prompt_sha256string*required
system_prompt_sha256string
control_aggregateobject*required
… expand 18 nestedcollapse
benchmarked_runsinteger· uint32*requiredmin 0
evaluated_runsinteger· uint32*requiredmin 0
median_cost_usdnumber· double
median_error_span_countinteger· uint64min 0
median_function_call_errorsinteger· uint64min 0
median_function_callsinteger· uint64min 0
median_input_tokensinteger· uint64min 0
median_output_tokensinteger· uint64min 0
median_reasoning_tokensinteger· uint64min 0
median_scorenumber· double
median_span_countinteger· uint64min 0
median_total_tokensinteger· uint64min 0
median_trace_countinteger· uint64min 0
median_trace_duration_msinteger· uint64min 0
median_wall_time_msinteger· uint64min 0
pass_ratenumber· double*required
passedinteger· uint32*requiredmin 0
runsinteger· uint32*requiredmin 0
created_atinteger· int64*required
deltaall of*required
Treatment minus control. Negative efficiency deltas are improvements.
… expand 1 nestedcollapse
all of (1)
variant 1
median_cost_usdnumber· double
median_error_span_countnumber· double
median_function_call_errorsnumber· double
median_function_callsnumber· double
median_reasoning_tokensnumber· double
median_scorenumber· double
median_span_countnumber· double
median_total_tokensnumber· double
median_trace_countnumber· double
median_trace_duration_msnumber· double
median_wall_time_msnumber· double
pass_ratenumber· double*required
dimensionstring*requiredenum: prompt, system_prompt
effective_execution_orderstring[]*required
evaluation_idstring*required
evaluatorany of
… expand 2 nestedcollapse
any of (2)
variant 1
arguments_sha256string*required
function_idstring*required
execution_order_policystring*requiredenum: balanced_control_first, balanced_treatment_first
limitsobject*required
… expand 2 nestedcollapse
evaluationall of
… expand 1 nestedcollapse
all of (1)
variant 1
max_error_spansinteger· uint64min 0
max_function_call_errorsinteger· uint64min 0
executionall of
… expand 1 nestedcollapse
all of (1)
variant 1
invocation_timeout_secondsinteger· uint64min 0
max_cost_usdnumber· double
max_output_tokens_per_callinteger· uint64min 0
max_total_tokensinteger· uint64min 0
max_turnsinteger· uint32min 0
scenario_timeout_secondsinteger· uint64min 0
modelobject*required
… expand 6 nestedcollapse
modeany of
… expand 2 nestedcollapse
any of (2)
variant 1
valuestringenum: ask, agent
system_prompt_strategyall of
… expand 1 nestedcollapse
all of (1)
variant 1
one of (3)
variant 1
valuestringenum: override
variant 2
valuestringenum: enrich
variant 3
valuestringenum: disabled
thinking_levelany of
… expand 2 nestedcollapse
any of (2)
variant 1
valuestringenum: minimal, low, medium, high, xhigh
order_sensitivityobject*required
… expand 3 nestedcollapse
controlobject*required
… expand 5 nestedcollapse
first_passedinteger· uint32*requiredmin 0
first_runsinteger· uint32*requiredmin 0
second_passedinteger· uint32*requiredmin 0
second_runsinteger· uint32*requiredmin 0
treatmentobject*required
… expand 5 nestedcollapse
first_passedinteger· uint32*requiredmin 0
first_runsinteger· uint32*requiredmin 0
second_passedinteger· uint32*requiredmin 0
second_runsinteger· uint32*requiredmin 0
runsobject[]*required
… expand 16 nestedcollapse
benchmarkany of
… expand 2 nestedcollapse
any of (2)
variant 1
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
error_span_countinteger· uint64min 0
function_call_errorsinteger· uint64*requiredmin 0
function_callsinteger· uint64*requiredmin 0
input_tokensinteger· uint64min 0
output_tokensinteger· uint64min 0
reasoning_tokensinteger· uint64min 0
sessionsinteger· uint64*requiredmin 0
span_countinteger· uint64min 0
total_tokensinteger· uint64min 0
trace_countinteger· uint64min 0
trace_duration_msinteger· uint64min 0
turnsinteger· uint64*requiredmin 0
wall_time_msinteger· uint64*requiredmin 0
completed_atinteger· int64
evaluationany of
… expand 2 nestedcollapse
execution_positioninteger· uint32min 0
failuresobject[]
… expand 4 nestedcollapse
phasestring*requiredenum: setup, send, await, collect, evaluate, limit, …
iterationinteger· uint32*requiredmin 0
metricsany of
… expand 2 nestedcollapse
any of (2)
variant 1
by_sessionobject[]*required
… expand 13 nestedcollapse
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
contextany of
The session's latest per-generation context snapshot (categories, budget, usage) — absent for sessions that have not generated since snapshots landed.
… expand 2 nestedcollapse
any of (2)
variant 1
categoriesobject*required
Where the request's tokens sit. Categories are assembly-time estimates; `hook_guidance` is the measured growth after assembly (pre-generate hook appends and orphan-repair patches), 0 when the request left assembly unchanged.
… expand 5 nestedcollapse
hook_guidanceinteger· uint64min 0
messagesobject*required
Estimated tokens of the assembled window's messages, by role.
… expand 4 nestedcollapse
assistantinteger· uint64*requiredmin 0
custominteger· uint64*requiredmin 0
function_resultinteger· uint64*requiredmin 0
userinteger· uint64*requiredmin 0
overheadinteger· uint64*requiredmin 0
Provider framing plus response_format / provider_options fields.
system_promptinteger· uint64*requiredmin 0
Final assembled system prompt: mode paragraph, identity, per-step aids, and any compaction summary section.
toolsinteger· uint64*requiredmin 0
Function schemas exposed to the model.
compactedboolean*required
effective_max_output_tokensinteger· uint64*requiredmin 0
Output allocation `usable` was derived against.
estimatorstring
Which estimator produced the numbers (`heuristic` until the context-manager resolves a real tokenizer). Absent when the context-manager predates the breakdown response.
freeinteger· uint64*requiredmin 0
session_cost_usdnumber· double
Running cost of the whole session in USD, accumulated across every generation step. `usage.cost_usd` is one step's bill — on providers with steep cache discounts the per-step number swings two orders of magnitude, so a chip showing it alone reads as a bouncing total.
session_idstring*required
stepinteger· uint64*requiredmin 0
summarized_head_tokensinteger· uint64min 0
timestampinteger· int64*required
totalinteger· uint64*requiredmin 0
Final request estimate: categories plus post-assembly growth.
usableinteger· uint64*requiredmin 0
The input budget the window was fit into.
usageany of
Actual provider usage for this generation, stamped after the terminal frame; absent when the provider returned none (or the generation never completed).
… expand 2 nestedcollapse
any of (2)
variant 1
cache_readinteger· uint64min 0
cache_writeinteger· uint64min 0
inputinteger· uint64min 0
outputinteger· uint64min 0
reasoninginteger· uint64min 0
depthinteger· uint32*requiredmin 0
function_call_errorsinteger· uint64*requiredmin 0
function_callsinteger· uint64*requiredmin 0
input_tokensinteger· uint64min 0
output_tokensinteger· uint64min 0
reasoning_tokensinteger· uint64min 0
session_idstring*required
turnsinteger· uint64*requiredmin 0
root_session_idstring*required
totalsobject*required
… expand 10 nestedcollapse
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
function_call_errorsinteger· uint64*requiredmin 0
function_callsinteger· uint64*requiredmin 0
input_tokensinteger· uint64min 0
output_tokensinteger· uint64min 0
reasoning_tokensinteger· uint64min 0
sessionsinteger· uint64*requiredmin 0
turnsinteger· uint64*requiredmin 0
tracesany of
Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not.
… expand 2 nestedcollapse
any of (2)
variant 1
by_sessionobject[]*required
… expand 7 nestedcollapse
depthinteger· uint32*requiredmin 0
duration_msinteger· uint64*requiredmin 0
Elapsed window from the session's first observed span to its last.
error_span_countinteger· uint64*requiredmin 0
session_idstring*required
span_countinteger· uint64*requiredmin 0
trace_countinteger· uint64*requiredmin 0
duration_msinteger· uint64*requiredmin 0
Elapsed window from the first observed span to the last observed span.
error_span_countinteger· uint64*requiredmin 0
span_countinteger· uint64*requiredmin 0
trace_countinteger· uint64*requiredmin 0
Distinct traces across the root session and all descendants.
pair_positioninteger· uint8min 0
rolestring*requiredenum: control, treatment
session_idstring*required
started_atinteger· int64*required
statusstring*requiredenum: pending, running, completed, failed, cancelled
schema_versionstring*required
shared_artifactsobject*required
… expand 4 nestedcollapse
function_policy_sha256string*required
limits_sha256string*required
model_sha256string*required
output_sha256string*required
source_evaluation_idstring
treatmentobject*required
… expand 3 nestedcollapse
prompt_sha256string*required
system_prompt_sha256string
treatment_aggregateobject*required
… expand 18 nestedcollapse
benchmarked_runsinteger· uint32*requiredmin 0
evaluated_runsinteger· uint32*requiredmin 0
median_cost_usdnumber· double
median_error_span_countinteger· uint64min 0
median_function_call_errorsinteger· uint64min 0
median_function_callsinteger· uint64min 0
median_input_tokensinteger· uint64min 0
median_output_tokensinteger· uint64min 0
median_reasoning_tokensinteger· uint64min 0
median_scorenumber· double
median_span_countinteger· uint64min 0
median_total_tokensinteger· uint64min 0
median_trace_countinteger· uint64min 0
median_trace_duration_msinteger· uint64min 0
median_wall_time_msinteger· uint64min 0
pass_ratenumber· double*required
passedinteger· uint32*requiredmin 0
runsinteger· uint32*requiredmin 0
requestobject*required
controlobject*required
… expand 3 nestedcollapse
system_promptstring
`null` disables the system prompt for this variant. An empty string keeps the harness behavior of resolving the provider or built-in prompt.
dimensionstring*requiredenum: prompt, system_prompt
evaluatorany of
… expand 2 nestedcollapse
any of (2)
variant 1
function_idstring*required
execution_orderall of
… expand 1 nestedcollapse
all of (1)
variant 1
valuestringenum: balanced_control_first, balanced_treatment_first
functionsobject*required
The fail-closed dispatch policy (harness.md § Functions). Absent on the send => every call denied (a plain chat loop).
… expand 3 nestedcollapse
exposeall of
… expand 1 nestedcollapse
all of (1)
variant 1
valuestringenum: agent_trigger, native
limitsobject*required
… expand 2 nestedcollapse
evaluationall of
… expand 1 nestedcollapse
all of (1)
variant 1
max_error_spansinteger· uint64min 0
max_function_call_errorsinteger· uint64min 0
executionall of
… expand 1 nestedcollapse
all of (1)
variant 1
invocation_timeout_secondsinteger· uint64min 0
max_cost_usdnumber· double
max_output_tokens_per_callinteger· uint64min 0
max_total_tokensinteger· uint64min 0
max_turnsinteger· uint32min 0
scenario_timeout_secondsinteger· uint64min 0
modelobject*required
… expand 6 nestedcollapse
modeany of
… expand 2 nestedcollapse
any of (2)
variant 1
valuestringenum: ask, agent
system_prompt_strategyall of
… expand 1 nestedcollapse
all of (1)
variant 1
one of (3)
variant 1
valuestringenum: override
variant 2
valuestringenum: enrich
variant 3
valuestringenum: disabled
thinking_levelany of
… expand 2 nestedcollapse
any of (2)
variant 1
valuestringenum: minimal, low, medium, high, xhigh
outputone of*required
Free text by default; `json` constrains the final answer to a JSON value, validated against `schema` when supplied.
… expand 2 nestedcollapse
one of (2)
variant 1
typestring*requiredenum: text
variant 2
typestring*requiredenum: json
runsinteger· uint32*requiredmin 0
source_evaluation_idstring
treatmentobject*required
… expand 3 nestedcollapse
system_promptstring
`null` disables the system prompt for this variant. An empty string keeps the harness behavior of resolving the provider or built-in prompt.
statusstring*requiredenum: queued, running, completed, failed, cancelled
Start a durable same-model A/B evaluation. Exactly one dimension may change: `prompt` or `system_prompt`. Returns evaluation_id immediately; use eval::status, eval::result, or the eval::completed trigger instead of polling in a tight loop.
controlobject*required
system_promptstring
`null` disables the system prompt for this variant. An empty string keeps the harness behavior of resolving the provider or built-in prompt.
dimensionstring*requiredenum: prompt, system_prompt
evaluatorany of
any of (2)
variant 1
function_idstring*required
execution_orderany of
any of (2)
variant 1
valuestringenum: balanced_control_first, balanced_treatment_first
functionsany of
Shared function policy for both variants. Omitted means deny all.
any of (2)
variant 1
exposeall of
… expand 1 nestedcollapse
all of (1)
variant 1
valuestringenum: agent_trigger, native
limitsall of
all of (1)
variant 1
evaluationall of
… expand 1 nestedcollapse
all of (1)
variant 1
max_error_spansinteger· uint64min 0
max_function_call_errorsinteger· uint64min 0
executionall of
… expand 1 nestedcollapse
all of (1)
variant 1
invocation_timeout_secondsinteger· uint64min 0
max_cost_usdnumber· double
max_output_tokens_per_callinteger· uint64min 0
max_total_tokensinteger· uint64min 0
max_turnsinteger· uint32min 0
scenario_timeout_secondsinteger· uint64min 0
metadataunknown
Shared harness metadata, including an optional filesystem scope.
modelobject*required
modeany of
any of (2)
variant 1
valuestringenum: ask, agent
system_prompt_strategyall of
all of (1)
variant 1
one of (3)
variant 1
valuestringenum: override
variant 2
valuestringenum: enrich
variant 3
valuestringenum: disabled
thinking_levelany of
any of (2)
variant 1
valuestringenum: minimal, low, medium, high, xhigh
outputany of
Shared output contract for both variants. Omitted means text.
any of (2)
variant 1
one of (2)
variant 1
typestring*requiredenum: text
variant 2
typestring*requiredenum: json
treatmentobject*required
system_promptstring
`null` disables the system prompt for this variant. An empty string keeps the harness behavior of resolving the provider or built-in prompt.
evaluation_idstring*required
statusstring*requiredenum: queued, running, completed, failed, cancelled
Read evaluation progress without loading the complete report.
evaluation_idstring*required
any of (2)
variant 1
activeany of
any of (2)
variant 1
iterationinteger· uint32*requiredmin 0
rolestring*requiredenum: control, treatment
session_idstring*required
started_atinteger· int64*required
completed_atinteger· int64
created_atinteger· int64*required
evaluation_idstring*required
failed_runsinteger· uint32*requiredmin 0
passed_runsinteger· uint32*requiredmin 0
statusstring*requiredenum: queued, running, completed, failed, cancelled
terminal_runsinteger· uint32*requiredmin 0
total_runsinteger· uint32*requiredmin 0
updated_atinteger· int64*required
Internal durable evaluation step.
evaluation_idstring*required
stepinteger· uint64*requiredmin 0
statusstring*requiredenum: queued, running, completed, failed, cancelled
Internal recovery and timeout sweep.
scheduled_atinteger· int64
sweptinteger· uint64*requiredmin 0
Serve the eval worker's injected console UI assets (content function for its console:script / console:style triggers).
pathstring*required
The asset path from the trigger config (e.g. `state/page.js`).
contentstring*required
The asset source, verbatim.
content_typestring*required
MIME type the console should serve the asset with.
