eval

Name: eval
Author: iii

v0.2.14

Compare live objective metrics for 2-5 root sessions; retain durable prompt experiments as an advanced surface.

license: Apache-2.0Published 22d agobenchmark evaluation metrics prompt sessions system-prompt

Install

bash

iii trigger compose::add worker=eval@0.2.14

Run in your project with the iii engine running. Exact versions are immutable and digest-pinned.

Functions14

eval::assert::exact

Function

Deep-compare output with arguments.expected; deterministic built-in evaluator.

Request

argumentsunknown
evaluation_idstringrequired
metricsobjectrequired
- by_sessionobject[]required
  cache_read_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  cache_write_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  contextany of
  The session's latest per-generation context snapshot (categories, budget, usage) — absent for sessions that have not generated since snapshots landed.
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  categoriesobjectrequired
  Where the request's tokens sit. Categories are assembly-time estimates; `hook_guidance` is the measured growth after assembly (pre-generate hook appends and orphan-repair patches), 0 when the request left assembly unchanged.
  Show 6 nestedHide nested
  hook_guidanceinteger· uint64min 0
  messagesobjectrequired
  Estimated tokens of the assembled window's messages, by role.
  Show 4 nestedHide nested
  assistantinteger· uint64requiredmin 0
  custominteger· uint64requiredmin 0
  function_resultinteger· uint64requiredmin 0
  userinteger· uint64requiredmin 0
  overheadinteger· uint64requiredmin 0
  Provider framing plus response_format / provider_options fields.
  skillsinteger· uint64min 0
  Selected skill bodies contained in the system prompt.
  system_promptinteger· uint64requiredmin 0
  Final assembled system prompt excluding tokens attributed to `skills`: identity, per-step aids, and any compaction summary section.
  toolsinteger· uint64requiredmin 0
  Function schemas exposed to the model.
  compactedbooleanrequired
  effective_max_output_tokensinteger· uint64requiredmin 0
  Output allocation `usable` was derived against.
  estimatorstring
  Which estimator produced the numbers (`heuristic` until the context-manager resolves a real tokenizer). Absent when the context-manager predates the breakdown response.
  freeinteger· uint64requiredmin 0
  modelstringrequired
  prompt_sections_fallbackstring
  Why no shared prefix went out: `disabled` (config), `no_stable_prefix` (empty identity prompt) or `prefix_rewritten` (a pre_generate hook replaced the prompt head; its prompt still wins, it just shares nothing).
  prompt_surface_digeststring
  Identity of the shared prompt prefix this generation went out with (`sha256:` of the stable section) — present only when the request carried `system_sections`. A local identity, never a provider hit; `usage.cache_read` is the reuse signal.
  providerstring
  session_cost_usdnumber· double
  Running cost of the whole session in USD, accumulated across every generation step. `usage.cost_usd` is one step's bill — on providers with steep cache discounts the per-step number swings two orders of magnitude, so a chip showing it alone reads as a bouncing total.
  session_idstringrequired
  stepinteger· uint64requiredmin 0
  summarized_head_tokensinteger· uint64min 0
  timestampinteger· int64required
  totalinteger· uint64requiredmin 0
  Final request estimate: categories plus post-assembly growth.
  turn_idstringrequired
  usableinteger· uint64requiredmin 0
  The input budget the window was fit into.
  usageany of
  Actual provider usage for this generation, stamped after the terminal frame; absent when the provider returned none (or the generation never completed).
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  cache_readinteger· uint64min 0
  cache_writeinteger· uint64min 0
  cost_usdnumber· double
  inputinteger· uint64min 0
  outputinteger· uint64min 0
  reasoninginteger· uint64min 0
  Variant 2
  valuenull
  Variant 2
  valuenull
  cost_usdnumber· doublerequired
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  depthinteger· uint32requiredmin 0
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  input_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  output_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  parent_session_idstring
  reasoning_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  session_idstringrequired
  turnsinteger· uint64requiredmin 0
- completebooleanrequired
- root_session_idstringrequired
- totalsobjectrequired
  cache_read_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  cache_write_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  cost_usdnumber· doublerequired
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  input_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  output_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  reasoning_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  sessionsinteger· uint64requiredmin 0
  turnsinteger· uint64requiredmin 0
- tracesany of
  Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not.
  any of (2)
  Variant 1
  by_sessionobject[]required
  Show 7 nestedHide nested
  depthinteger· uint32requiredmin 0
  duration_msinteger· uint64requiredmin 0
  Elapsed window from the session's first observed span to its last.
  error_span_countinteger· uint64requiredmin 0
  parent_session_idstring
  session_idstringrequired
  span_countinteger· uint64requiredmin 0
  trace_countinteger· uint64requiredmin 0
  duration_msinteger· uint64requiredmin 0
  Elapsed window from the first observed span to the last observed span.
  error_span_countinteger· uint64requiredmin 0
  span_countinteger· uint64requiredmin 0
  trace_countinteger· uint64requiredmin 0
  Distinct traces across the root session and all descendants.
  Variant 2
  valuenull
outputunknownrequired
rolestringrequiredenum control, treatment
run_idstringrequired
session_idstringrequired

Response

detailsunknown
passedbooleanrequired
reasonstring
scorenumber· double

eval::assert::normalized_text

Function

Compare text after normalizing case, whitespace, and surrounding punctuation; deterministic built-in evaluator.

Request

argumentsunknown
evaluation_idstringrequired
metricsobjectrequired
- by_sessionobject[]required
  cache_read_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  cache_write_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  contextany of
  The session's latest per-generation context snapshot (categories, budget, usage) — absent for sessions that have not generated since snapshots landed.
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  categoriesobjectrequired
  Where the request's tokens sit. Categories are assembly-time estimates; `hook_guidance` is the measured growth after assembly (pre-generate hook appends and orphan-repair patches), 0 when the request left assembly unchanged.
  Show 6 nestedHide nested
  hook_guidanceinteger· uint64min 0
  messagesobjectrequired
  Estimated tokens of the assembled window's messages, by role.
  Show 4 nestedHide nested
  assistantinteger· uint64requiredmin 0
  custominteger· uint64requiredmin 0
  function_resultinteger· uint64requiredmin 0
  userinteger· uint64requiredmin 0
  overheadinteger· uint64requiredmin 0
  Provider framing plus response_format / provider_options fields.
  skillsinteger· uint64min 0
  Selected skill bodies contained in the system prompt.
  system_promptinteger· uint64requiredmin 0
  Final assembled system prompt excluding tokens attributed to `skills`: identity, per-step aids, and any compaction summary section.
  toolsinteger· uint64requiredmin 0
  Function schemas exposed to the model.
  compactedbooleanrequired
  effective_max_output_tokensinteger· uint64requiredmin 0
  Output allocation `usable` was derived against.
  estimatorstring
  Which estimator produced the numbers (`heuristic` until the context-manager resolves a real tokenizer). Absent when the context-manager predates the breakdown response.
  freeinteger· uint64requiredmin 0
  modelstringrequired
  prompt_sections_fallbackstring
  Why no shared prefix went out: `disabled` (config), `no_stable_prefix` (empty identity prompt) or `prefix_rewritten` (a pre_generate hook replaced the prompt head; its prompt still wins, it just shares nothing).
  prompt_surface_digeststring
  Identity of the shared prompt prefix this generation went out with (`sha256:` of the stable section) — present only when the request carried `system_sections`. A local identity, never a provider hit; `usage.cache_read` is the reuse signal.
  providerstring
  session_cost_usdnumber· double
  Running cost of the whole session in USD, accumulated across every generation step. `usage.cost_usd` is one step's bill — on providers with steep cache discounts the per-step number swings two orders of magnitude, so a chip showing it alone reads as a bouncing total.
  session_idstringrequired
  stepinteger· uint64requiredmin 0
  summarized_head_tokensinteger· uint64min 0
  timestampinteger· int64required
  totalinteger· uint64requiredmin 0
  Final request estimate: categories plus post-assembly growth.
  turn_idstringrequired
  usableinteger· uint64requiredmin 0
  The input budget the window was fit into.
  usageany of
  Actual provider usage for this generation, stamped after the terminal frame; absent when the provider returned none (or the generation never completed).
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  cache_readinteger· uint64min 0
  cache_writeinteger· uint64min 0
  cost_usdnumber· double
  inputinteger· uint64min 0
  outputinteger· uint64min 0
  reasoninginteger· uint64min 0
  Variant 2
  valuenull
  Variant 2
  valuenull
  cost_usdnumber· doublerequired
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  depthinteger· uint32requiredmin 0
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  input_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  output_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  parent_session_idstring
  reasoning_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  session_idstringrequired
  turnsinteger· uint64requiredmin 0
- completebooleanrequired
- root_session_idstringrequired
- totalsobjectrequired
  cache_read_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  cache_write_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  cost_usdnumber· doublerequired
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  input_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  output_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  reasoning_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  sessionsinteger· uint64requiredmin 0
  turnsinteger· uint64requiredmin 0
- tracesany of
  Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not.
  any of (2)
  Variant 1
  by_sessionobject[]required
  Show 7 nestedHide nested
  depthinteger· uint32requiredmin 0
  duration_msinteger· uint64requiredmin 0
  Elapsed window from the session's first observed span to its last.
  error_span_countinteger· uint64requiredmin 0
  parent_session_idstring
  session_idstringrequired
  span_countinteger· uint64requiredmin 0
  trace_countinteger· uint64requiredmin 0
  duration_msinteger· uint64requiredmin 0
  Elapsed window from the first observed span to the last observed span.
  error_span_countinteger· uint64requiredmin 0
  span_countinteger· uint64requiredmin 0
  trace_countinteger· uint64requiredmin 0
  Distinct traces across the root session and all descendants.
  Variant 2
  valuenull
outputunknownrequired
rolestringrequiredenum control, treatment
run_idstringrequired
session_idstringrequired

Response

detailsunknown
passedbooleanrequired
reasonstring
scorenumber· double

eval::cancel

Function

Cancel an evaluation and its active harness turn.

Request

evaluation_idstringrequired

Response

cancelledbooleanrequired
statusstringrequiredenum queued, running, completed, failed, cancelled

eval::compare-sessions

Function

Compare 2–5 existing root sessions from live session, lifecycle, and Harness metrics. The response contains objective values and arithmetic deltas against the chosen baseline; it does not score, rank, validate equivalence, or persist a snapshot.

Request

baseline_session_idstringrequired
session_idsstring[]required2…5 items

Response

baseline_session_idstringrequired
captured_atinteger· int64required
schema_versionstringrequired
sessionsobject[]required
- deltasobject
  One entry per objective numeric metric. A missing value is represented by null in either delta field, never by zero.
- errorsstring[]
- lifecycleobjectrequired
  completeboolean
  Null means the metrics read was unavailable.
  expects_wakeboolean
  Null means the status read was unavailable.
  partialbooleanrequired
  result_errorstring
  session_statusstringrequiredenum idle, working, done, error
  terminalboolean
  Null means `harness::status` did not provide a turn record.
  turn_idstring
  turn_statusany of
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  valuestringenum running, awaiting_functions, completed, cancelled, failed
  Variant 2
  valuenull
- metricsany of
  any of (2)
  Variant 1
  by_sessionobject[]required
  Show 13 nestedHide nested
  cache_read_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  cache_write_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  contextany of
  The session's latest per-generation context snapshot (categories, budget, usage) — absent for sessions that have not generated since snapshots landed.
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  categoriesobjectrequired
  Where the request's tokens sit. Categories are assembly-time estimates; `hook_guidance` is the measured growth after assembly (pre-generate hook appends and orphan-repair patches), 0 when the request left assembly unchanged.
  Show 6 nestedHide nested
  hook_guidanceinteger· uint64min 0
  messagesobjectrequired
  Estimated tokens of the assembled window's messages, by role.
  Show 4 nestedHide nested
  assistantinteger· uint64requiredmin 0
  custominteger· uint64requiredmin 0
  function_resultinteger· uint64requiredmin 0
  userinteger· uint64requiredmin 0
  overheadinteger· uint64requiredmin 0
  Provider framing plus response_format / provider_options fields.
  skillsinteger· uint64min 0
  Selected skill bodies contained in the system prompt.
  system_promptinteger· uint64requiredmin 0
  Final assembled system prompt excluding tokens attributed to `skills`: identity, per-step aids, and any compaction summary section.
  toolsinteger· uint64requiredmin 0
  Function schemas exposed to the model.
  compactedbooleanrequired
  effective_max_output_tokensinteger· uint64requiredmin 0
  Output allocation `usable` was derived against.
  estimatorstring
  Which estimator produced the numbers (`heuristic` until the context-manager resolves a real tokenizer). Absent when the context-manager predates the breakdown response.
  freeinteger· uint64requiredmin 0
  modelstringrequired
  prompt_sections_fallbackstring
  Why no shared prefix went out: `disabled` (config), `no_stable_prefix` (empty identity prompt) or `prefix_rewritten` (a pre_generate hook replaced the prompt head; its prompt still wins, it just shares nothing).
  prompt_surface_digeststring
  Identity of the shared prompt prefix this generation went out with (`sha256:` of the stable section) — present only when the request carried `system_sections`. A local identity, never a provider hit; `usage.cache_read` is the reuse signal.
  providerstring
  session_cost_usdnumber· double
  Running cost of the whole session in USD, accumulated across every generation step. `usage.cost_usd` is one step's bill — on providers with steep cache discounts the per-step number swings two orders of magnitude, so a chip showing it alone reads as a bouncing total.
  session_idstringrequired
  stepinteger· uint64requiredmin 0
  summarized_head_tokensinteger· uint64min 0
  timestampinteger· int64required
  totalinteger· uint64requiredmin 0
  Final request estimate: categories plus post-assembly growth.
  turn_idstringrequired
  usableinteger· uint64requiredmin 0
  The input budget the window was fit into.
  usageany of
  Actual provider usage for this generation, stamped after the terminal frame; absent when the provider returned none (or the generation never completed).
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  cache_readinteger· uint64min 0
  cache_writeinteger· uint64min 0
  cost_usdnumber· double
  inputinteger· uint64min 0
  outputinteger· uint64min 0
  reasoninginteger· uint64min 0
  Variant 2
  valuenull
  Variant 2
  valuenull
  cost_usdnumber· doublerequired
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  depthinteger· uint32requiredmin 0
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  input_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  output_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  parent_session_idstring
  reasoning_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  session_idstringrequired
  turnsinteger· uint64requiredmin 0
  completebooleanrequired
  root_session_idstringrequired
  totalsobjectrequired
  Show 10 nestedHide nested
  cache_read_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  cache_write_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  cost_usdnumber· doublerequired
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  input_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  output_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  reasoning_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  sessionsinteger· uint64requiredmin 0
  turnsinteger· uint64requiredmin 0
  tracesany of
  Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not.
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  by_sessionobject[]required
  Show 7 nestedHide nested
  depthinteger· uint32requiredmin 0
  duration_msinteger· uint64requiredmin 0
  Elapsed window from the session's first observed span to its last.
  error_span_countinteger· uint64requiredmin 0
  parent_session_idstring
  session_idstringrequired
  span_countinteger· uint64requiredmin 0
  trace_countinteger· uint64requiredmin 0
  duration_msinteger· uint64requiredmin 0
  Elapsed window from the first observed span to the last observed span.
  error_span_countinteger· uint64requiredmin 0
  span_countinteger· uint64requiredmin 0
  trace_countinteger· uint64requiredmin 0
  Distinct traces across the root session and all descendants.
  Variant 2
  valuenull
  Variant 2
  valuenull
- sessionobjectrequired
  created_atinteger· int64
  descriptionstring
  draftstring
  forked_fromstring
  message_countinteger· uint64min 0
  metadataunknown
  session_idstringrequired
  statusstringrequiredenum idle, working, done, error
  status_reasonstring
  titlestring
  updated_atinteger· int64
- summaryany of
  any of (2)
  Variant 1
  cache_read_tokensinteger· uint64min 0
  cache_write_tokensinteger· uint64min 0
  compacted_sessionsinteger· uint64min 0
  context_free_tokensinteger· uint64min 0
  context_occupancynumber· double
  context_total_tokensinteger· uint64min 0
  context_usable_tokensinteger· uint64min 0
  cost_per_generation_usdnumber· double
  descendantsinteger· uint64min 0
  error_span_countinteger· uint64min 0
  function_call_errorsinteger· uint64min 0
  function_callsinteger· uint64min 0
  function_error_ratenumber· double
  generationsinteger· uint64min 0
  input_tokensinteger· uint64min 0
  max_depthinteger· uint32min 0
  output_tokensinteger· uint64min 0
  reasoning_tokensinteger· uint64min 0
  sessionsinteger· uint64min 0
  span_countinteger· uint64min 0
  subject_cost_usdnumber· double
  tokens_per_generationnumber· double
  total_tokensinteger· uint64min 0
  trace_countinteger· uint64min 0
  trace_duration_msinteger· uint64min 0
  Variant 2
  valuenull

eval::delete

Function

Delete a terminal evaluation report and its session indexes.

Request

evaluation_idstringrequired

Response

deletedbooleanrequired

eval::list

Function

List recent evaluations as lightweight summaries, newest first.

Request

limitinteger· uint32min 0

Response

evaluationsobject[]required
- completed_atinteger· int64
- control_labelstring
- created_atinteger· int64required
- dimensionstringrequiredenum prompt, system_prompt
- eligibleboolean
- errorstring
- evaluation_idstringrequired
- failed_runsinteger· uint32requiredmin 0
- modelstringrequired
- passed_runsinteger· uint32requiredmin 0
- providerstring
- statusstringrequiredenum queued, running, completed, failed, cancelled
- terminal_runsinteger· uint32requiredmin 0
- total_runsinteger· uint32requiredmin 0
- treatment_labelstring
- updated_atinteger· int64required

eval::on-turn-completed

Function

Internal harness turn-completed wake-up.

Request

session_idstring
terminalboolean

Response

wokebooleanrequired

eval::rerun

Function

Repeat a terminal evaluation from its persisted request. Set reverse_order to invert the balanced A/B order. The rerun receives a new evaluation ID and fresh sessions.

Request

evaluation_idstringrequired
reverse_orderboolean

Response

evaluation_idstringrequired
statusstringrequiredenum queued, running, completed, failed, cancelled

eval::result

Function

Read an evaluation result. The report is present only after terminal completion.

Request

evaluation_idstringrequired

Response

any of (2)

Variant 1

progressobjectrequired
- control_aggregateobjectrequired
  Show 18 nestedHide nested
  benchmarked_runsinteger· uint32requiredmin 0
  evaluated_runsinteger· uint32requiredmin 0
  median_cost_usdnumber· double
  median_error_span_countinteger· uint64min 0
  median_function_call_errorsinteger· uint64min 0
  median_function_callsinteger· uint64min 0
  median_input_tokensinteger· uint64min 0
  median_output_tokensinteger· uint64min 0
  median_reasoning_tokensinteger· uint64min 0
  median_scorenumber· double
  median_span_countinteger· uint64min 0
  median_total_tokensinteger· uint64min 0
  median_trace_countinteger· uint64min 0
  median_trace_duration_msinteger· uint64min 0
  median_wall_time_msinteger· uint64min 0
  pass_ratenumber· doublerequired
  passedinteger· uint32requiredmin 0
  runsinteger· uint32requiredmin 0
- deltaobjectrequired
  Show 12 nestedHide nested
  median_cost_usdnumber· double
  median_error_span_countnumber· double
  median_function_call_errorsnumber· double
  median_function_callsnumber· double
  median_reasoning_tokensnumber· double
  median_scorenumber· double
  median_span_countnumber· double
  median_total_tokensnumber· double
  median_trace_countnumber· double
  median_trace_duration_msnumber· double
  median_wall_time_msnumber· double
  pass_ratenumber· doublerequired
- effective_execution_orderstring[]required
- order_sensitivityobjectrequired
  Show 3 nestedHide nested
  controlobjectrequired
  Show 5 nestedHide nested
  differsbooleanrequired
  first_passedinteger· uint32requiredmin 0
  first_runsinteger· uint32requiredmin 0
  second_passedinteger· uint32requiredmin 0
  second_runsinteger· uint32requiredmin 0
  detectedbooleanrequired
  treatmentobjectrequired
  Show 5 nestedHide nested
  differsbooleanrequired
  first_passedinteger· uint32requiredmin 0
  first_runsinteger· uint32requiredmin 0
  second_passedinteger· uint32requiredmin 0
  second_runsinteger· uint32requiredmin 0
- runsobject[]required
  Show 16 nestedHide nested
  benchmarkany of
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  cache_read_tokensinteger· uint64min 0
  cache_write_tokensinteger· uint64min 0
  cost_usdnumber· double
  error_span_countinteger· uint64min 0
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  input_tokensinteger· uint64min 0
  output_tokensinteger· uint64min 0
  reasoning_tokensinteger· uint64min 0
  sessionsinteger· uint64requiredmin 0
  span_countinteger· uint64min 0
  total_tokensinteger· uint64min 0
  trace_countinteger· uint64min 0
  trace_duration_msinteger· uint64min 0
  turnsinteger· uint64requiredmin 0
  wall_time_msinteger· uint64requiredmin 0
  Variant 2
  valuenull
  completed_atinteger· int64
  evaluationany of
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  detailsunknown
  passedbooleanrequired
  reasonstring
  scorenumber· double
  Variant 2
  valuenull
  execution_positioninteger· uint32min 0
  failuresobject[]
  Show 4 nestedHide nested
  codestring
  function_idstring
  messagestringrequired
  phasestringrequiredenum setup, send, await, collect, evaluate, limit, …
  iterationinteger· uint32requiredmin 0
  metricsany of
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  by_sessionobject[]required
  Show 13 nestedHide nested
  cache_read_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  cache_write_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  contextany of
  The session's latest per-generation context snapshot (categories, budget, usage) — absent for sessions that have not generated since snapshots landed.
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  categoriesobjectrequired
  Where the request's tokens sit. Categories are assembly-time estimates; `hook_guidance` is the measured growth after assembly (pre-generate hook appends and orphan-repair patches), 0 when the request left assembly unchanged.
  Show 6 nestedHide nested
  hook_guidanceinteger· uint64min 0
  messagesobjectrequired
  Estimated tokens of the assembled window's messages, by role.
  Show 4 nestedHide nested
  assistantinteger· uint64requiredmin 0
  custominteger· uint64requiredmin 0
  function_resultinteger· uint64requiredmin 0
  userinteger· uint64requiredmin 0
  overheadinteger· uint64requiredmin 0
  Provider framing plus response_format / provider_options fields.
  skillsinteger· uint64min 0
  Selected skill bodies contained in the system prompt.
  system_promptinteger· uint64requiredmin 0
  Final assembled system prompt excluding tokens attributed to `skills`: identity, per-step aids, and any compaction summary section.
  toolsinteger· uint64requiredmin 0
  Function schemas exposed to the model.
  compactedbooleanrequired
  effective_max_output_tokensinteger· uint64requiredmin 0
  Output allocation `usable` was derived against.
  estimatorstring
  Which estimator produced the numbers (`heuristic` until the context-manager resolves a real tokenizer). Absent when the context-manager predates the breakdown response.
  freeinteger· uint64requiredmin 0
  modelstringrequired
  prompt_sections_fallbackstring
  Why no shared prefix went out: `disabled` (config), `no_stable_prefix` (empty identity prompt) or `prefix_rewritten` (a pre_generate hook replaced the prompt head; its prompt still wins, it just shares nothing).
  prompt_surface_digeststring
  Identity of the shared prompt prefix this generation went out with (`sha256:` of the stable section) — present only when the request carried `system_sections`. A local identity, never a provider hit; `usage.cache_read` is the reuse signal.
  providerstring
  session_cost_usdnumber· double
  Running cost of the whole session in USD, accumulated across every generation step. `usage.cost_usd` is one step's bill — on providers with steep cache discounts the per-step number swings two orders of magnitude, so a chip showing it alone reads as a bouncing total.
  session_idstringrequired
  stepinteger· uint64requiredmin 0
  summarized_head_tokensinteger· uint64min 0
  timestampinteger· int64required
  totalinteger· uint64requiredmin 0
  Final request estimate: categories plus post-assembly growth.
  turn_idstringrequired
  usableinteger· uint64requiredmin 0
  The input budget the window was fit into.
  usageany of
  Actual provider usage for this generation, stamped after the terminal frame; absent when the provider returned none (or the generation never completed).
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  cache_readinteger· uint64min 0
  cache_writeinteger· uint64min 0
  cost_usdnumber· double
  inputinteger· uint64min 0
  outputinteger· uint64min 0
  reasoninginteger· uint64min 0
  Variant 2
  valuenull
  Variant 2
  valuenull
  cost_usdnumber· doublerequired
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  depthinteger· uint32requiredmin 0
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  input_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  output_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  parent_session_idstring
  reasoning_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  session_idstringrequired
  turnsinteger· uint64requiredmin 0
  completebooleanrequired
  root_session_idstringrequired
  totalsobjectrequired
  Show 10 nestedHide nested
  cache_read_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  cache_write_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  cost_usdnumber· doublerequired
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  input_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  output_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  reasoning_tokensinteger· uint64requiredmin 0
  Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
  sessionsinteger· uint64requiredmin 0
  turnsinteger· uint64requiredmin 0
  tracesany of
  Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not.
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  by_sessionobject[]required
  Show 7 nestedHide nested
  depthinteger· uint32requiredmin 0
  duration_msinteger· uint64requiredmin 0
  Elapsed window from the session's first observed span to its last.
  error_span_countinteger· uint64requiredmin 0
  parent_session_idstring
  session_idstringrequired
  span_countinteger· uint64requiredmin 0
  trace_countinteger· uint64requiredmin 0
  duration_msinteger· uint64requiredmin 0
  Elapsed window from the first observed span to the last observed span.
  error_span_countinteger· uint64requiredmin 0
  span_countinteger· uint64requiredmin 0
  trace_countinteger· uint64requiredmin 0
  Distinct traces across the root session and all descendants.
  Variant 2
  valuenull
  Variant 2
  valuenull
  outputunknown
  pair_positioninteger· uint8min 0
  passedboolean
  rolestringrequiredenum control, treatment
  run_idstringrequired
  session_idstringrequired
  started_atinteger· int64required
  statusstringrequiredenum pending, running, completed, failed, cancelled
  turn_idstring
- treatment_aggregateobjectrequired
  Show 18 nestedHide nested
  benchmarked_runsinteger· uint32requiredmin 0
  evaluated_runsinteger· uint32requiredmin 0
  median_cost_usdnumber· double
  median_error_span_countinteger· uint64min 0
  median_function_call_errorsinteger· uint64min 0
  median_function_callsinteger· uint64min 0
  median_input_tokensinteger· uint64min 0
  median_output_tokensinteger· uint64min 0
  median_reasoning_tokensinteger· uint64min 0
  median_scorenumber· double
  median_span_countinteger· uint64min 0
  median_total_tokensinteger· uint64min 0
  median_trace_countinteger· uint64min 0
  median_trace_duration_msinteger· uint64min 0
  median_wall_time_msinteger· uint64min 0
  pass_ratenumber· doublerequired
  passedinteger· uint32requiredmin 0
  runsinteger· uint32requiredmin 0
reportany of
any of (2)
Variant 1
completed_atinteger· int64required
controlobjectrequired
Show 3 nestedHide nested
labelstring
prompt_sha256stringrequired
system_prompt_sha256string
control_aggregateobjectrequired
Show 18 nestedHide nested
benchmarked_runsinteger· uint32requiredmin 0
evaluated_runsinteger· uint32requiredmin 0
median_cost_usdnumber· double
median_error_span_countinteger· uint64min 0
median_function_call_errorsinteger· uint64min 0
median_function_callsinteger· uint64min 0
median_input_tokensinteger· uint64min 0
median_output_tokensinteger· uint64min 0
median_reasoning_tokensinteger· uint64min 0
median_scorenumber· double
median_span_countinteger· uint64min 0
median_total_tokensinteger· uint64min 0
median_trace_countinteger· uint64min 0
median_trace_duration_msinteger· uint64min 0
median_wall_time_msinteger· uint64min 0
pass_ratenumber· doublerequired
passedinteger· uint32requiredmin 0
runsinteger· uint32requiredmin 0
created_atinteger· int64required
deltaall ofrequired
Treatment minus control. Negative efficiency deltas are improvements.
Show 1 nestedHide nested
all of (1)
Variant 1
median_cost_usdnumber· double
median_error_span_countnumber· double
median_function_call_errorsnumber· double
median_function_callsnumber· double
median_reasoning_tokensnumber· double
median_scorenumber· double
median_span_countnumber· double
median_total_tokensnumber· double
median_trace_countnumber· double
median_trace_duration_msnumber· double
median_wall_time_msnumber· double
pass_ratenumber· doublerequired
dimensionstringrequiredenum prompt, system_prompt
effective_execution_orderstring[]required
eligibleboolean
evaluation_idstringrequired
evaluatorany of
Show 2 nestedHide nested
any of (2)
Variant 1
arguments_sha256stringrequired
function_idstringrequired
Variant 2
valuenull
execution_order_policystringrequiredenum balanced_control_first, balanced_treatment_first
limitsobjectrequired
Show 2 nestedHide nested
evaluationall of
Show 1 nestedHide nested
all of (1)
Variant 1
max_error_spansinteger· uint64min 0
max_function_call_errorsinteger· uint64min 0
executionall of
Show 1 nestedHide nested
all of (1)
Variant 1
invocation_timeout_secondsinteger· uint64min 0
max_cost_usdnumber· double
max_output_tokens_per_callinteger· uint64min 0
max_total_tokensinteger· uint64min 0
max_turnsinteger· uint32min 0
scenario_timeout_secondsinteger· uint64min 0
modelobjectrequired
Show 5 nestedHide nested
modelstringrequired
providerstring
provider_optionsobject
system_prompt_strategyall of
Show 1 nestedHide nested
all of (1)
Variant 1
one of (3)
Variant 1
valuestringenum override
Variant 2
valuestringenum enrich
Variant 3
valuestringenum disabled
thinking_levelany of
Show 2 nestedHide nested
any of (2)
Variant 1
valuestringenum minimal, low, medium, high, xhigh
Variant 2
valuenull
order_sensitivityobjectrequired
Show 3 nestedHide nested
controlobjectrequired
Show 5 nestedHide nested
differsbooleanrequired
first_passedinteger· uint32requiredmin 0
first_runsinteger· uint32requiredmin 0
second_passedinteger· uint32requiredmin 0
second_runsinteger· uint32requiredmin 0
detectedbooleanrequired
treatmentobjectrequired
Show 5 nestedHide nested
differsbooleanrequired
first_passedinteger· uint32requiredmin 0
first_runsinteger· uint32requiredmin 0
second_passedinteger· uint32requiredmin 0
second_runsinteger· uint32requiredmin 0
runsobject[]required
Show 16 nestedHide nested
benchmarkany of
Show 2 nestedHide nested
any of (2)
Variant 1
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
cost_usdnumber· double
error_span_countinteger· uint64min 0
function_call_errorsinteger· uint64requiredmin 0
function_callsinteger· uint64requiredmin 0
input_tokensinteger· uint64min 0
output_tokensinteger· uint64min 0
reasoning_tokensinteger· uint64min 0
sessionsinteger· uint64requiredmin 0
span_countinteger· uint64min 0
total_tokensinteger· uint64min 0
trace_countinteger· uint64min 0
trace_duration_msinteger· uint64min 0
turnsinteger· uint64requiredmin 0
wall_time_msinteger· uint64requiredmin 0
Variant 2
valuenull
completed_atinteger· int64
evaluationany of
Show 2 nestedHide nested
any of (2)
Variant 1
detailsunknown
passedbooleanrequired
reasonstring
scorenumber· double
Variant 2
valuenull
execution_positioninteger· uint32min 0
failuresobject[]
Show 4 nestedHide nested
codestring
function_idstring
messagestringrequired
phasestringrequiredenum setup, send, await, collect, evaluate, limit, …
iterationinteger· uint32requiredmin 0
metricsany of
Show 2 nestedHide nested
any of (2)
Variant 1
by_sessionobject[]required
Show 13 nestedHide nested
cache_read_tokensinteger· uint64requiredmin 0
Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
cache_write_tokensinteger· uint64requiredmin 0
Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
contextany of
The session's latest per-generation context snapshot (categories, budget, usage) — absent for sessions that have not generated since snapshots landed.
Show 2 nestedHide nested
any of (2)
Variant 1
categoriesobjectrequired
Where the request's tokens sit. Categories are assembly-time estimates; `hook_guidance` is the measured growth after assembly (pre-generate hook appends and orphan-repair patches), 0 when the request left assembly unchanged.
Show 6 nestedHide nested
hook_guidanceinteger· uint64min 0
messagesobjectrequired
Estimated tokens of the assembled window's messages, by role.
Show 4 nestedHide nested
assistantinteger· uint64requiredmin 0
custominteger· uint64requiredmin 0
function_resultinteger· uint64requiredmin 0
userinteger· uint64requiredmin 0
overheadinteger· uint64requiredmin 0
Provider framing plus response_format / provider_options fields.
skillsinteger· uint64min 0
Selected skill bodies contained in the system prompt.
system_promptinteger· uint64requiredmin 0
Final assembled system prompt excluding tokens attributed to `skills`: identity, per-step aids, and any compaction summary section.
toolsinteger· uint64requiredmin 0
Function schemas exposed to the model.
compactedbooleanrequired
effective_max_output_tokensinteger· uint64requiredmin 0
Output allocation `usable` was derived against.
estimatorstring
Which estimator produced the numbers (`heuristic` until the context-manager resolves a real tokenizer). Absent when the context-manager predates the breakdown response.
freeinteger· uint64requiredmin 0
modelstringrequired
prompt_sections_fallbackstring
Why no shared prefix went out: `disabled` (config), `no_stable_prefix` (empty identity prompt) or `prefix_rewritten` (a pre_generate hook replaced the prompt head; its prompt still wins, it just shares nothing).
prompt_surface_digeststring
Identity of the shared prompt prefix this generation went out with (`sha256:` of the stable section) — present only when the request carried `system_sections`. A local identity, never a provider hit; `usage.cache_read` is the reuse signal.
providerstring
session_cost_usdnumber· double
Running cost of the whole session in USD, accumulated across every generation step. `usage.cost_usd` is one step's bill — on providers with steep cache discounts the per-step number swings two orders of magnitude, so a chip showing it alone reads as a bouncing total.
session_idstringrequired
stepinteger· uint64requiredmin 0
summarized_head_tokensinteger· uint64min 0
timestampinteger· int64required
totalinteger· uint64requiredmin 0
Final request estimate: categories plus post-assembly growth.
turn_idstringrequired
usableinteger· uint64requiredmin 0
The input budget the window was fit into.
usageany of
Actual provider usage for this generation, stamped after the terminal frame; absent when the provider returned none (or the generation never completed).
Show 2 nestedHide nested
any of (2)
Variant 1
cache_readinteger· uint64min 0
cache_writeinteger· uint64min 0
cost_usdnumber· double
inputinteger· uint64min 0
outputinteger· uint64min 0
reasoninginteger· uint64min 0
Variant 2
valuenull
Variant 2
valuenull
cost_usdnumber· doublerequired
Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
depthinteger· uint32requiredmin 0
function_call_errorsinteger· uint64requiredmin 0
function_callsinteger· uint64requiredmin 0
input_tokensinteger· uint64requiredmin 0
Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
output_tokensinteger· uint64requiredmin 0
Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
parent_session_idstring
reasoning_tokensinteger· uint64requiredmin 0
Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
session_idstringrequired
turnsinteger· uint64requiredmin 0
completebooleanrequired
root_session_idstringrequired
totalsobjectrequired
Show 10 nestedHide nested
cache_read_tokensinteger· uint64requiredmin 0
Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
cache_write_tokensinteger· uint64requiredmin 0
Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
cost_usdnumber· doublerequired
Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
function_call_errorsinteger· uint64requiredmin 0
function_callsinteger· uint64requiredmin 0
input_tokensinteger· uint64requiredmin 0
Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
output_tokensinteger· uint64requiredmin 0
Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
reasoning_tokensinteger· uint64requiredmin 0
Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.
sessionsinteger· uint64requiredmin 0
turnsinteger· uint64requiredmin 0
tracesany of
Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not.
Show 2 nestedHide nested
any of (2)
Variant 1
by_sessionobject[]required
Show 7 nestedHide nested
depthinteger· uint32requiredmin 0
duration_msinteger· uint64requiredmin 0
Elapsed window from the session's first observed span to its last.
error_span_countinteger· uint64requiredmin 0
parent_session_idstring
session_idstringrequired
span_countinteger· uint64requiredmin 0
trace_countinteger· uint64requiredmin 0
duration_msinteger· uint64requiredmin 0
Elapsed window from the first observed span to the last observed span.
error_span_countinteger· uint64requiredmin 0
span_countinteger· uint64requiredmin 0
trace_countinteger· uint64requiredmin 0
Distinct traces across the root session and all descendants.
Variant 2
valuenull
Variant 2
valuenull
outputunknown
pair_positioninteger· uint8min 0
passedboolean
rolestringrequiredenum control, treatment
run_idstringrequired
session_idstringrequired
started_atinteger· int64required
statusstringrequiredenum pending, running, completed, failed, cancelled
turn_idstring
schema_versionstringrequired
shared_artifactsobjectrequired
Show 4 nestedHide nested
function_policy_sha256stringrequired
limits_sha256stringrequired
model_sha256stringrequired
output_sha256stringrequired
source_evaluation_idstring
treatmentobjectrequired
Show 3 nestedHide nested
labelstring
prompt_sha256stringrequired
system_prompt_sha256string
treatment_aggregateobjectrequired
Show 18 nestedHide nested
benchmarked_runsinteger· uint32requiredmin 0
evaluated_runsinteger· uint32requiredmin 0
median_cost_usdnumber· double
median_error_span_countinteger· uint64min 0
median_function_call_errorsinteger· uint64min 0
median_function_callsinteger· uint64min 0
median_input_tokensinteger· uint64min 0
median_output_tokensinteger· uint64min 0
median_reasoning_tokensinteger· uint64min 0
median_scorenumber· double
median_span_countinteger· uint64min 0
median_total_tokensinteger· uint64min 0
median_trace_countinteger· uint64min 0
median_trace_duration_msinteger· uint64min 0
median_wall_time_msinteger· uint64min 0
pass_ratenumber· doublerequired
passedinteger· uint32requiredmin 0
runsinteger· uint32requiredmin 0
Variant 2
valuenull
requestobjectrequired
- controlobjectrequired
  Show 3 nestedHide nested
  labelstring
  promptstringrequired
  system_promptstring
  `null` disables the system prompt for this variant. An empty string keeps the harness behavior of resolving the provider or built-in prompt.
- dimensionstringrequiredenum prompt, system_prompt
- evaluatorany of
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  argumentsunknown
  function_idstringrequired
  Variant 2
  valuenull
- execution_orderall of
  Show 1 nestedHide nested
  all of (1)
  Variant 1
  valuestringenum balanced_control_first, balanced_treatment_first
- functionsobjectrequired
  The fail-closed dispatch policy (harness.md § Functions). Absent on the send => every call denied (a plain chat loop).
  Show 3 nestedHide nested
  allowstring[]
  denystring[]
  exposeall of
  Show 1 nestedHide nested
  all of (1)
  Variant 1
  valuestringenum agent_trigger, native
- limitsobjectrequired
  Show 2 nestedHide nested
  evaluationall of
  Show 1 nestedHide nested
  all of (1)
  Variant 1
  max_error_spansinteger· uint64min 0
  max_function_call_errorsinteger· uint64min 0
  executionall of
  Show 1 nestedHide nested
  all of (1)
  Variant 1
  invocation_timeout_secondsinteger· uint64min 0
  max_cost_usdnumber· double
  max_output_tokens_per_callinteger· uint64min 0
  max_total_tokensinteger· uint64min 0
  max_turnsinteger· uint32min 0
  scenario_timeout_secondsinteger· uint64min 0
- metadataunknown
- modelobjectrequired
  Show 5 nestedHide nested
  modelstringrequired
  providerstring
  provider_optionsobject
  system_prompt_strategyall of
  Show 1 nestedHide nested
  all of (1)
  Variant 1
  one of (3)
  Variant 1
  valuestringenum override
  Variant 2
  valuestringenum enrich
  Variant 3
  valuestringenum disabled
  thinking_levelany of
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  valuestringenum minimal, low, medium, high, xhigh
  Variant 2
  valuenull
- outputone ofrequired
  Free text by default; `json` constrains the final answer to a JSON value, validated against `schema` when supplied.
  Show 2 nestedHide nested
  one of (2)
  Variant 1
  typestringrequiredenum text
  Variant 2
  schemaunknown
  typestringrequiredenum json
- runsinteger· uint32requiredmin 0
- source_evaluation_idstring
- treatmentobjectrequired
  Show 3 nestedHide nested
  labelstring
  promptstringrequired
  system_promptstring
  `null` disables the system prompt for this variant. An empty string keeps the harness behavior of resolving the provider or built-in prompt.
statusstringrequiredenum queued, running, completed, failed, cancelled

Variant 2

valuenull

eval::start

Function

Start a durable same-model A/B evaluation. Exactly one dimension may change: `prompt` or `system_prompt`. Returns evaluation_id immediately; use eval::status, eval::result, or the eval::completed trigger instead of polling in a tight loop.

Request

controlobjectrequired
- labelstring
- promptstringrequired
- system_promptstring
  `null` disables the system prompt for this variant. An empty string keeps the harness behavior of resolving the provider or built-in prompt.
dimensionstringrequiredenum prompt, system_prompt
evaluatorany of
any of (2)
Variant 1
argumentsunknown
function_idstringrequired
Variant 2
valuenull
execution_orderany of
any of (2)
Variant 1
valuestringenum balanced_control_first, balanced_treatment_first
Variant 2
valuenull
functionsany of
Shared function policy for both variants. Omitted means deny all.
any of (2)
Variant 1
allowstring[]
denystring[]
exposeall of
Show 1 nestedHide nested
all of (1)
Variant 1
valuestringenum agent_trigger, native
Variant 2
valuenull
limitsall of
all of (1)
Variant 1
evaluationall of
Show 1 nestedHide nested
all of (1)
Variant 1
max_error_spansinteger· uint64min 0
max_function_call_errorsinteger· uint64min 0
executionall of
Show 1 nestedHide nested
all of (1)
Variant 1
invocation_timeout_secondsinteger· uint64min 0
max_cost_usdnumber· double
max_output_tokens_per_callinteger· uint64min 0
max_total_tokensinteger· uint64min 0
max_turnsinteger· uint32min 0
scenario_timeout_secondsinteger· uint64min 0
metadataunknown
Shared harness metadata, including an optional filesystem scope.
modelobjectrequired
- modelstringrequired
- providerstring
- provider_optionsobject
- system_prompt_strategyall of
  all of (1)
  Variant 1
  one of (3)
  Variant 1
  valuestringenum override
  Variant 2
  valuestringenum enrich
  Variant 3
  valuestringenum disabled
- thinking_levelany of
  any of (2)
  Variant 1
  valuestringenum minimal, low, medium, high, xhigh
  Variant 2
  valuenull
outputany of
Shared output contract for both variants. Omitted means text.
any of (2)
Variant 1
one of (2)
Variant 1
typestringrequiredenum text
Variant 2
schemaunknown
typestringrequiredenum json
Variant 2
valuenull
runsinteger· uint32min 0
treatmentobjectrequired
- labelstring
- promptstringrequired
- system_promptstring
  `null` disables the system prompt for this variant. An empty string keeps the harness behavior of resolving the provider or built-in prompt.

Response

evaluation_idstringrequired
statusstringrequiredenum queued, running, completed, failed, cancelled

eval::status

Function

Read evaluation progress without loading the complete report.

Request

evaluation_idstringrequired

Response

any of (2)

Variant 1

activeany of
any of (2)
Variant 1
iterationinteger· uint32requiredmin 0
rolestringrequiredenum control, treatment
run_idstringrequired
session_idstringrequired
started_atinteger· int64required
turn_idstring
Variant 2
valuenull
completed_atinteger· int64
created_atinteger· int64required
errorstring
evaluation_idstringrequired
failed_runsinteger· uint32requiredmin 0
passed_runsinteger· uint32requiredmin 0
statusstringrequiredenum queued, running, completed, failed, cancelled
terminal_runsinteger· uint32requiredmin 0
total_runsinteger· uint32requiredmin 0
updated_atinteger· int64required

Variant 2

valuenull

eval::step

Function

Internal durable evaluation step.

Request

evaluation_idstringrequired
stepinteger· uint64requiredmin 0

Response

skippedbooleanrequired
statusstringrequiredenum queued, running, completed, failed, cancelled

eval::sweep

Function

Internal recovery and timeout sweep.

Request

scheduled_atinteger· int64
scheduled_timestring

Response

sweptinteger· uint64requiredmin 0

eval::ui-content

Function

Serve the eval worker's injected console UI assets (content function for its console:script / console:style triggers).

Request

pathstringrequired
The asset path from the trigger config (e.g. `state/page.js`).

Response

contentstringrequired
The asset source, verbatim.
content_typestringrequired
MIME type the console should serve the asset with.

Triggers1

eval::completed

Trigger

An evaluation reached a terminal status.

Invocation

evaluation_idstring

Return

valueunknown

Install

Functions14

Triggers1

Search the registry