$worker

eval

Name: eval
Author: iii

v0.1.0

Compare live objective metrics for 2-5 root sessions; retain durable prompt experiments as an advanced surface.

iiiverified

43 installs0 in 7d0 today

install

$iii worker add eval@0.1.0

binaryevaluation benchmark sessions metrics prompt system-prompt

macOS: arm64 · x64
Linux: arm64 · armv7 · x64
Windows: arm64 · x64 · x86

13functions 1triggers 0skills 5deps v0.1.0version

repoiii-hq/workers

exact versions are immutable; binary and bundle artifacts are digest-pinned.

functions

eval::assert::exact

function

Built-in deterministic evaluator: deep-compare output with arguments.expected.

request

argumentsunknown
evaluation_idstringrequired
metricsobjectrequired
- by_sessionobject[]required
  cache_read_tokensinteger· uint64min 0
  cache_write_tokensinteger· uint64min 0
  cost_usdnumber· double
  depthinteger· uint32requiredmin 0
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  input_tokensinteger· uint64min 0
  output_tokensinteger· uint64min 0
  parent_session_idstring
  reasoning_tokensinteger· uint64min 0
  session_idstringrequired
  turnsinteger· uint64requiredmin 0
- completebooleanrequired
- root_session_idstringrequired
- totalsobjectrequired
  cache_read_tokensinteger· uint64min 0
  cache_write_tokensinteger· uint64min 0
  cost_usdnumber· double
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  input_tokensinteger· uint64min 0
  output_tokensinteger· uint64min 0
  reasoning_tokensinteger· uint64min 0
  sessionsinteger· uint64requiredmin 0
  turnsinteger· uint64requiredmin 0
- tracesany of
  Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not.
  any of (2)
  variant 1
  by_sessionobject[]required
  … expand 7 nestedcollapse
  depthinteger· uint32requiredmin 0
  duration_msinteger· uint64requiredmin 0
  Elapsed window from the session's first observed span to its last.
  error_span_countinteger· uint64requiredmin 0
  parent_session_idstring
  session_idstringrequired
  span_countinteger· uint64requiredmin 0
  trace_countinteger· uint64requiredmin 0
  duration_msinteger· uint64requiredmin 0
  Elapsed window from the first observed span to the last observed span.
  error_span_countinteger· uint64requiredmin 0
  span_countinteger· uint64requiredmin 0
  trace_countinteger· uint64requiredmin 0
  Distinct traces across the root session and all descendants.
  variant 2
  valuenull
outputunknownrequired
rolestringrequiredenum: control, treatment
run_idstringrequired
session_idstringrequired

response

detailsunknown
passedbooleanrequired
reasonstring
scorenumber· double

eval::assert::normalized_text

function

Built-in deterministic evaluator: compare text after normalizing case, whitespace, and surrounding punctuation.

request

argumentsunknown
evaluation_idstringrequired
metricsobjectrequired
- by_sessionobject[]required
  cache_read_tokensinteger· uint64min 0
  cache_write_tokensinteger· uint64min 0
  cost_usdnumber· double
  depthinteger· uint32requiredmin 0
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  input_tokensinteger· uint64min 0
  output_tokensinteger· uint64min 0
  parent_session_idstring
  reasoning_tokensinteger· uint64min 0
  session_idstringrequired
  turnsinteger· uint64requiredmin 0
- completebooleanrequired
- root_session_idstringrequired
- totalsobjectrequired
  cache_read_tokensinteger· uint64min 0
  cache_write_tokensinteger· uint64min 0
  cost_usdnumber· double
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  input_tokensinteger· uint64min 0
  output_tokensinteger· uint64min 0
  reasoning_tokensinteger· uint64min 0
  sessionsinteger· uint64requiredmin 0
  turnsinteger· uint64requiredmin 0
- tracesany of
  Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not.
  any of (2)
  variant 1
  by_sessionobject[]required
  … expand 7 nestedcollapse
  depthinteger· uint32requiredmin 0
  duration_msinteger· uint64requiredmin 0
  Elapsed window from the session's first observed span to its last.
  error_span_countinteger· uint64requiredmin 0
  parent_session_idstring
  session_idstringrequired
  span_countinteger· uint64requiredmin 0
  trace_countinteger· uint64requiredmin 0
  duration_msinteger· uint64requiredmin 0
  Elapsed window from the first observed span to the last observed span.
  error_span_countinteger· uint64requiredmin 0
  span_countinteger· uint64requiredmin 0
  trace_countinteger· uint64requiredmin 0
  Distinct traces across the root session and all descendants.
  variant 2
  valuenull
outputunknownrequired
rolestringrequiredenum: control, treatment
run_idstringrequired
session_idstringrequired

response

detailsunknown
passedbooleanrequired
reasonstring
scorenumber· double

eval::cancel

function

Cancel an evaluation and its active harness turn.

request

evaluation_idstringrequired

response

cancelledbooleanrequired
statusstringrequiredenum: queued, running, completed, failed, cancelled

eval::delete

function

Delete a terminal evaluation report and its session indexes.

request

evaluation_idstringrequired

response

deletedbooleanrequired

eval::list

function

List recent evaluations as lightweight summaries, newest first.

request

limitinteger· uint32min 0

response

evaluationsobject[]required
- completed_atinteger· int64
- control_labelstring
- created_atinteger· int64required
- dimensionstringrequiredenum: prompt, system_prompt
- eligibleboolean
- errorstring
- evaluation_idstringrequired
- failed_runsinteger· uint32requiredmin 0
- modelstringrequired
- passed_runsinteger· uint32requiredmin 0
- providerstring
- statusstringrequiredenum: queued, running, completed, failed, cancelled
- terminal_runsinteger· uint32requiredmin 0
- total_runsinteger· uint32requiredmin 0
- treatment_labelstring
- updated_atinteger· int64required

eval::on-turn-completed

function

Internal harness turn-completed wake-up.

request

session_idstring
terminalboolean

response

wokebooleanrequired

eval::rerun

function

Repeat a terminal evaluation from its persisted request. Set reverse_order to invert the balanced A/B order. The rerun receives a new evaluation ID and fresh sessions.

request

evaluation_idstringrequired
reverse_orderboolean

response

evaluation_idstringrequired
statusstringrequiredenum: queued, running, completed, failed, cancelled

eval::result

function

Read an evaluation result. The report is present only after terminal completion.

request

evaluation_idstringrequired

response

any of (2)

variant 1

progressobjectrequired
- control_aggregateobjectrequired
  … expand 18 nestedcollapse
  benchmarked_runsinteger· uint32requiredmin 0
  evaluated_runsinteger· uint32requiredmin 0
  median_cost_usdnumber· double
  median_error_span_countinteger· uint64min 0
  median_function_call_errorsinteger· uint64min 0
  median_function_callsinteger· uint64min 0
  median_input_tokensinteger· uint64min 0
  median_output_tokensinteger· uint64min 0
  median_reasoning_tokensinteger· uint64min 0
  median_scorenumber· double
  median_span_countinteger· uint64min 0
  median_total_tokensinteger· uint64min 0
  median_trace_countinteger· uint64min 0
  median_trace_duration_msinteger· uint64min 0
  median_wall_time_msinteger· uint64min 0
  pass_ratenumber· doublerequired
  passedinteger· uint32requiredmin 0
  runsinteger· uint32requiredmin 0
- deltaobjectrequired
  … expand 12 nestedcollapse
  median_cost_usdnumber· double
  median_error_span_countnumber· double
  median_function_call_errorsnumber· double
  median_function_callsnumber· double
  median_reasoning_tokensnumber· double
  median_scorenumber· double
  median_span_countnumber· double
  median_total_tokensnumber· double
  median_trace_countnumber· double
  median_trace_duration_msnumber· double
  median_wall_time_msnumber· double
  pass_ratenumber· doublerequired
- effective_execution_orderstring[]required
- order_sensitivityobjectrequired
  … expand 3 nestedcollapse
  controlobjectrequired
  … expand 5 nestedcollapse
  differsbooleanrequired
  first_passedinteger· uint32requiredmin 0
  first_runsinteger· uint32requiredmin 0
  second_passedinteger· uint32requiredmin 0
  second_runsinteger· uint32requiredmin 0
  detectedbooleanrequired
  treatmentobjectrequired
  … expand 5 nestedcollapse
  differsbooleanrequired
  first_passedinteger· uint32requiredmin 0
  first_runsinteger· uint32requiredmin 0
  second_passedinteger· uint32requiredmin 0
  second_runsinteger· uint32requiredmin 0
- runsobject[]required
  … expand 16 nestedcollapse
  benchmarkany of
  … expand 2 nestedcollapse
  any of (2)
  variant 1
  cache_read_tokensinteger· uint64min 0
  cache_write_tokensinteger· uint64min 0
  cost_usdnumber· double
  error_span_countinteger· uint64min 0
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  input_tokensinteger· uint64min 0
  output_tokensinteger· uint64min 0
  reasoning_tokensinteger· uint64min 0
  sessionsinteger· uint64requiredmin 0
  span_countinteger· uint64min 0
  total_tokensinteger· uint64min 0
  trace_countinteger· uint64min 0
  trace_duration_msinteger· uint64min 0
  turnsinteger· uint64requiredmin 0
  wall_time_msinteger· uint64requiredmin 0
  variant 2
  valuenull
  completed_atinteger· int64
  evaluationany of
  … expand 2 nestedcollapse
  any of (2)
  variant 1
  detailsunknown
  passedbooleanrequired
  reasonstring
  scorenumber· double
  variant 2
  valuenull
  execution_positioninteger· uint32min 0
  failuresobject[]
  … expand 4 nestedcollapse
  codestring
  function_idstring
  messagestringrequired
  phasestringrequiredenum: setup, send, await, collect, evaluate, limit, …
  iterationinteger· uint32requiredmin 0
  metricsany of
  … expand 2 nestedcollapse
  any of (2)
  variant 1
  by_sessionobject[]required
  … expand 12 nestedcollapse
  cache_read_tokensinteger· uint64min 0
  cache_write_tokensinteger· uint64min 0
  cost_usdnumber· double
  depthinteger· uint32requiredmin 0
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  input_tokensinteger· uint64min 0
  output_tokensinteger· uint64min 0
  parent_session_idstring
  reasoning_tokensinteger· uint64min 0
  session_idstringrequired
  turnsinteger· uint64requiredmin 0
  completebooleanrequired
  root_session_idstringrequired
  totalsobjectrequired
  … expand 10 nestedcollapse
  cache_read_tokensinteger· uint64min 0
  cache_write_tokensinteger· uint64min 0
  cost_usdnumber· double
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  input_tokensinteger· uint64min 0
  output_tokensinteger· uint64min 0
  reasoning_tokensinteger· uint64min 0
  sessionsinteger· uint64requiredmin 0
  turnsinteger· uint64requiredmin 0
  tracesany of
  Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not.
  … expand 2 nestedcollapse
  any of (2)
  variant 1
  by_sessionobject[]required
  … expand 7 nestedcollapse
  depthinteger· uint32requiredmin 0
  duration_msinteger· uint64requiredmin 0
  Elapsed window from the session's first observed span to its last.
  error_span_countinteger· uint64requiredmin 0
  parent_session_idstring
  session_idstringrequired
  span_countinteger· uint64requiredmin 0
  trace_countinteger· uint64requiredmin 0
  duration_msinteger· uint64requiredmin 0
  Elapsed window from the first observed span to the last observed span.
  error_span_countinteger· uint64requiredmin 0
  span_countinteger· uint64requiredmin 0
  trace_countinteger· uint64requiredmin 0
  Distinct traces across the root session and all descendants.
  variant 2
  valuenull
  variant 2
  valuenull
  outputunknown
  pair_positioninteger· uint8min 0
  passedboolean
  rolestringrequiredenum: control, treatment
  run_idstringrequired
  session_idstringrequired
  started_atinteger· int64required
  statusstringrequiredenum: pending, running, completed, failed, cancelled
  turn_idstring
- treatment_aggregateobjectrequired
  … expand 18 nestedcollapse
  benchmarked_runsinteger· uint32requiredmin 0
  evaluated_runsinteger· uint32requiredmin 0
  median_cost_usdnumber· double
  median_error_span_countinteger· uint64min 0
  median_function_call_errorsinteger· uint64min 0
  median_function_callsinteger· uint64min 0
  median_input_tokensinteger· uint64min 0
  median_output_tokensinteger· uint64min 0
  median_reasoning_tokensinteger· uint64min 0
  median_scorenumber· double
  median_span_countinteger· uint64min 0
  median_total_tokensinteger· uint64min 0
  median_trace_countinteger· uint64min 0
  median_trace_duration_msinteger· uint64min 0
  median_wall_time_msinteger· uint64min 0
  pass_ratenumber· doublerequired
  passedinteger· uint32requiredmin 0
  runsinteger· uint32requiredmin 0
reportany of
any of (2)
variant 1
completed_atinteger· int64required
controlobjectrequired
… expand 3 nestedcollapse
labelstring
prompt_sha256stringrequired
system_prompt_sha256string
control_aggregateobjectrequired
… expand 18 nestedcollapse
benchmarked_runsinteger· uint32requiredmin 0
evaluated_runsinteger· uint32requiredmin 0
median_cost_usdnumber· double
median_error_span_countinteger· uint64min 0
median_function_call_errorsinteger· uint64min 0
median_function_callsinteger· uint64min 0
median_input_tokensinteger· uint64min 0
median_output_tokensinteger· uint64min 0
median_reasoning_tokensinteger· uint64min 0
median_scorenumber· double
median_span_countinteger· uint64min 0
median_total_tokensinteger· uint64min 0
median_trace_countinteger· uint64min 0
median_trace_duration_msinteger· uint64min 0
median_wall_time_msinteger· uint64min 0
pass_ratenumber· doublerequired
passedinteger· uint32requiredmin 0
runsinteger· uint32requiredmin 0
created_atinteger· int64required
deltaall ofrequired
Treatment minus control. Negative efficiency deltas are improvements.
… expand 1 nestedcollapse
all of (1)
variant 1
median_cost_usdnumber· double
median_error_span_countnumber· double
median_function_call_errorsnumber· double
median_function_callsnumber· double
median_reasoning_tokensnumber· double
median_scorenumber· double
median_span_countnumber· double
median_total_tokensnumber· double
median_trace_countnumber· double
median_trace_duration_msnumber· double
median_wall_time_msnumber· double
pass_ratenumber· doublerequired
dimensionstringrequiredenum: prompt, system_prompt
effective_execution_orderstring[]required
eligibleboolean
evaluation_idstringrequired
evaluatorany of
… expand 2 nestedcollapse
any of (2)
variant 1
arguments_sha256stringrequired
function_idstringrequired
variant 2
valuenull
execution_order_policystringrequiredenum: balanced_control_first, balanced_treatment_first
limitsobjectrequired
… expand 2 nestedcollapse
evaluationall of
… expand 1 nestedcollapse
all of (1)
variant 1
max_error_spansinteger· uint64min 0
max_function_call_errorsinteger· uint64min 0
executionall of
… expand 1 nestedcollapse
all of (1)
variant 1
invocation_timeout_secondsinteger· uint64min 0
max_cost_usdnumber· double
max_output_tokens_per_callinteger· uint64min 0
max_total_tokensinteger· uint64min 0
max_turnsinteger· uint32min 0
scenario_timeout_secondsinteger· uint64min 0
modelobjectrequired
… expand 6 nestedcollapse
modeany of
… expand 2 nestedcollapse
any of (2)
variant 1
valuestringenum: ask, agent
variant 2
valuenull
modelstringrequired
providerstring
provider_optionsobject
system_prompt_strategyall of
… expand 1 nestedcollapse
all of (1)
variant 1
one of (3)
variant 1
valuestringenum: override
variant 2
valuestringenum: enrich
variant 3
valuestringenum: disabled
thinking_levelany of
… expand 2 nestedcollapse
any of (2)
variant 1
valuestringenum: minimal, low, medium, high, xhigh
variant 2
valuenull
order_sensitivityobjectrequired
… expand 3 nestedcollapse
controlobjectrequired
… expand 5 nestedcollapse
differsbooleanrequired
first_passedinteger· uint32requiredmin 0
first_runsinteger· uint32requiredmin 0
second_passedinteger· uint32requiredmin 0
second_runsinteger· uint32requiredmin 0
detectedbooleanrequired
treatmentobjectrequired
… expand 5 nestedcollapse
differsbooleanrequired
first_passedinteger· uint32requiredmin 0
first_runsinteger· uint32requiredmin 0
second_passedinteger· uint32requiredmin 0
second_runsinteger· uint32requiredmin 0
runsobject[]required
… expand 16 nestedcollapse
benchmarkany of
… expand 2 nestedcollapse
any of (2)
variant 1
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
cost_usdnumber· double
error_span_countinteger· uint64min 0
function_call_errorsinteger· uint64requiredmin 0
function_callsinteger· uint64requiredmin 0
input_tokensinteger· uint64min 0
output_tokensinteger· uint64min 0
reasoning_tokensinteger· uint64min 0
sessionsinteger· uint64requiredmin 0
span_countinteger· uint64min 0
total_tokensinteger· uint64min 0
trace_countinteger· uint64min 0
trace_duration_msinteger· uint64min 0
turnsinteger· uint64requiredmin 0
wall_time_msinteger· uint64requiredmin 0
variant 2
valuenull
completed_atinteger· int64
evaluationany of
… expand 2 nestedcollapse
any of (2)
variant 1
detailsunknown
passedbooleanrequired
reasonstring
scorenumber· double
variant 2
valuenull
execution_positioninteger· uint32min 0
failuresobject[]
… expand 4 nestedcollapse
codestring
function_idstring
messagestringrequired
phasestringrequiredenum: setup, send, await, collect, evaluate, limit, …
iterationinteger· uint32requiredmin 0
metricsany of
… expand 2 nestedcollapse
any of (2)
variant 1
by_sessionobject[]required
… expand 12 nestedcollapse
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
cost_usdnumber· double
depthinteger· uint32requiredmin 0
function_call_errorsinteger· uint64requiredmin 0
function_callsinteger· uint64requiredmin 0
input_tokensinteger· uint64min 0
output_tokensinteger· uint64min 0
parent_session_idstring
reasoning_tokensinteger· uint64min 0
session_idstringrequired
turnsinteger· uint64requiredmin 0
completebooleanrequired
root_session_idstringrequired
totalsobjectrequired
… expand 10 nestedcollapse
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
cost_usdnumber· double
function_call_errorsinteger· uint64requiredmin 0
function_callsinteger· uint64requiredmin 0
input_tokensinteger· uint64min 0
output_tokensinteger· uint64min 0
reasoning_tokensinteger· uint64min 0
sessionsinteger· uint64requiredmin 0
turnsinteger· uint64requiredmin 0
tracesany of
Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not.
… expand 2 nestedcollapse
any of (2)
variant 1
by_sessionobject[]required
… expand 7 nestedcollapse
depthinteger· uint32requiredmin 0
duration_msinteger· uint64requiredmin 0
Elapsed window from the session's first observed span to its last.
error_span_countinteger· uint64requiredmin 0
parent_session_idstring
session_idstringrequired
span_countinteger· uint64requiredmin 0
trace_countinteger· uint64requiredmin 0
duration_msinteger· uint64requiredmin 0
Elapsed window from the first observed span to the last observed span.
error_span_countinteger· uint64requiredmin 0
span_countinteger· uint64requiredmin 0
trace_countinteger· uint64requiredmin 0
Distinct traces across the root session and all descendants.
variant 2
valuenull
variant 2
valuenull
outputunknown
pair_positioninteger· uint8min 0
passedboolean
rolestringrequiredenum: control, treatment
run_idstringrequired
session_idstringrequired
started_atinteger· int64required
statusstringrequiredenum: pending, running, completed, failed, cancelled
turn_idstring
schema_versionstringrequired
shared_artifactsobjectrequired
… expand 4 nestedcollapse
function_policy_sha256stringrequired
limits_sha256stringrequired
model_sha256stringrequired
output_sha256stringrequired
source_evaluation_idstring
treatmentobjectrequired
… expand 3 nestedcollapse
labelstring
prompt_sha256stringrequired
system_prompt_sha256string
treatment_aggregateobjectrequired
… expand 18 nestedcollapse
benchmarked_runsinteger· uint32requiredmin 0
evaluated_runsinteger· uint32requiredmin 0
median_cost_usdnumber· double
median_error_span_countinteger· uint64min 0
median_function_call_errorsinteger· uint64min 0
median_function_callsinteger· uint64min 0
median_input_tokensinteger· uint64min 0
median_output_tokensinteger· uint64min 0
median_reasoning_tokensinteger· uint64min 0
median_scorenumber· double
median_span_countinteger· uint64min 0
median_total_tokensinteger· uint64min 0
median_trace_countinteger· uint64min 0
median_trace_duration_msinteger· uint64min 0
median_wall_time_msinteger· uint64min 0
pass_ratenumber· doublerequired
passedinteger· uint32requiredmin 0
runsinteger· uint32requiredmin 0
variant 2
valuenull
requestobjectrequired
- controlobjectrequired
  … expand 3 nestedcollapse
  labelstring
  promptstringrequired
  system_promptstring
  `null` disables the system prompt for this variant. An empty string keeps the harness behavior of resolving the provider or built-in prompt.
- dimensionstringrequiredenum: prompt, system_prompt
- evaluatorany of
  … expand 2 nestedcollapse
  any of (2)
  variant 1
  argumentsunknown
  function_idstringrequired
  variant 2
  valuenull
- execution_orderall of
  … expand 1 nestedcollapse
  all of (1)
  variant 1
  valuestringenum: balanced_control_first, balanced_treatment_first
- functionsobjectrequired
  The fail-closed dispatch policy (harness.md § Functions). Absent on the send => every call denied (a plain chat loop).
  … expand 3 nestedcollapse
  allowstring[]
  denystring[]
  exposeall of
  … expand 1 nestedcollapse
  all of (1)
  variant 1
  valuestringenum: agent_trigger, native
- limitsobjectrequired
  … expand 2 nestedcollapse
  evaluationall of
  … expand 1 nestedcollapse
  all of (1)
  variant 1
  max_error_spansinteger· uint64min 0
  max_function_call_errorsinteger· uint64min 0
  executionall of
  … expand 1 nestedcollapse
  all of (1)
  variant 1
  invocation_timeout_secondsinteger· uint64min 0
  max_cost_usdnumber· double
  max_output_tokens_per_callinteger· uint64min 0
  max_total_tokensinteger· uint64min 0
  max_turnsinteger· uint32min 0
  scenario_timeout_secondsinteger· uint64min 0
- metadataunknown
- modelobjectrequired
  … expand 6 nestedcollapse
  modeany of
  … expand 2 nestedcollapse
  any of (2)
  variant 1
  valuestringenum: ask, agent
  variant 2
  valuenull
  modelstringrequired
  providerstring
  provider_optionsobject
  system_prompt_strategyall of
  … expand 1 nestedcollapse
  all of (1)
  variant 1
  one of (3)
  variant 1
  valuestringenum: override
  variant 2
  valuestringenum: enrich
  variant 3
  valuestringenum: disabled
  thinking_levelany of
  … expand 2 nestedcollapse
  any of (2)
  variant 1
  valuestringenum: minimal, low, medium, high, xhigh
  variant 2
  valuenull
- outputone ofrequired
  Free text by default; `json` constrains the final answer to a JSON value, validated against `schema` when supplied.
  … expand 2 nestedcollapse
  one of (2)
  variant 1
  typestringrequiredenum: text
  variant 2
  schemaunknown
  typestringrequiredenum: json
- runsinteger· uint32requiredmin 0
- source_evaluation_idstring
- treatmentobjectrequired
  … expand 3 nestedcollapse
  labelstring
  promptstringrequired
  system_promptstring
  `null` disables the system prompt for this variant. An empty string keeps the harness behavior of resolving the provider or built-in prompt.
statusstringrequiredenum: queued, running, completed, failed, cancelled

variant 2

valuenull

eval::start

function

Start a durable same-model A/B evaluation. Exactly one dimension may change: `prompt` or `system_prompt`. Returns evaluation_id immediately; use eval::status, eval::result, or the eval::completed trigger instead of polling in a tight loop.

request

controlobjectrequired
- labelstring
- promptstringrequired
- system_promptstring
  `null` disables the system prompt for this variant. An empty string keeps the harness behavior of resolving the provider or built-in prompt.
dimensionstringrequiredenum: prompt, system_prompt
evaluatorany of
any of (2)
variant 1
argumentsunknown
function_idstringrequired
variant 2
valuenull
execution_orderany of
any of (2)
variant 1
valuestringenum: balanced_control_first, balanced_treatment_first
variant 2
valuenull
functionsany of
Shared function policy for both variants. Omitted means deny all.
any of (2)
variant 1
allowstring[]
denystring[]
exposeall of
… expand 1 nestedcollapse
all of (1)
variant 1
valuestringenum: agent_trigger, native
variant 2
valuenull
limitsall of
all of (1)
variant 1
evaluationall of
… expand 1 nestedcollapse
all of (1)
variant 1
max_error_spansinteger· uint64min 0
max_function_call_errorsinteger· uint64min 0
executionall of
… expand 1 nestedcollapse
all of (1)
variant 1
invocation_timeout_secondsinteger· uint64min 0
max_cost_usdnumber· double
max_output_tokens_per_callinteger· uint64min 0
max_total_tokensinteger· uint64min 0
max_turnsinteger· uint32min 0
scenario_timeout_secondsinteger· uint64min 0
metadataunknown
Shared harness metadata, including an optional filesystem scope.
modelobjectrequired
- modeany of
  any of (2)
  variant 1
  valuestringenum: ask, agent
  variant 2
  valuenull
- modelstringrequired
- providerstring
- provider_optionsobject
- system_prompt_strategyall of
  all of (1)
  variant 1
  one of (3)
  variant 1
  valuestringenum: override
  variant 2
  valuestringenum: enrich
  variant 3
  valuestringenum: disabled
- thinking_levelany of
  any of (2)
  variant 1
  valuestringenum: minimal, low, medium, high, xhigh
  variant 2
  valuenull
outputany of
Shared output contract for both variants. Omitted means text.
any of (2)
variant 1
one of (2)
variant 1
typestringrequiredenum: text
variant 2
schemaunknown
typestringrequiredenum: json
variant 2
valuenull
runsinteger· uint32min 0
treatmentobjectrequired
- labelstring
- promptstringrequired
- system_promptstring
  `null` disables the system prompt for this variant. An empty string keeps the harness behavior of resolving the provider or built-in prompt.

response

evaluation_idstringrequired
statusstringrequiredenum: queued, running, completed, failed, cancelled

eval::status

function

Read evaluation progress without loading the complete report.

request

evaluation_idstringrequired

response

any of (2)

variant 1

activeany of
any of (2)
variant 1
iterationinteger· uint32requiredmin 0
rolestringrequiredenum: control, treatment
run_idstringrequired
session_idstringrequired
started_atinteger· int64required
turn_idstring
variant 2
valuenull
completed_atinteger· int64
created_atinteger· int64required
errorstring
evaluation_idstringrequired
failed_runsinteger· uint32requiredmin 0
passed_runsinteger· uint32requiredmin 0
statusstringrequiredenum: queued, running, completed, failed, cancelled
terminal_runsinteger· uint32requiredmin 0
total_runsinteger· uint32requiredmin 0
updated_atinteger· int64required

variant 2

valuenull

eval::step

function

Internal durable evaluation step.

request

evaluation_idstringrequired
stepinteger· uint64requiredmin 0

response

skippedbooleanrequired
statusstringrequiredenum: queued, running, completed, failed, cancelled

eval::sweep

function

Internal recovery and timeout sweep.

request

scheduled_atinteger· int64
scheduled_timestring

response

sweptinteger· uint64requiredmin 0

eval::ui-content

function

Serve the eval worker's injected console UI assets (content function for its console:script / console:style triggers).

request

pathstringrequired
The asset path from the trigger config (e.g. `state/page.js`).

response

contentstringrequired
The asset source, verbatim.
content_typestringrequired
MIME type the console should serve the asset with.

triggers

eval::completed

trigger

An evaluation reached a terminal status.

invocation

evaluation_idstring

return

valueunknown