Skip to content
Registry

eval

v0.1.1

Compare live objective metrics for 2-5 root sessions; retain durable prompt experiments as an advanced surface.

benchmarkevaluationmetricspromptsessionssystem-prompt

Install

bash
iii trigger compose::add worker=eval@0.1.1

Run in your project with the iii engine running. Exact versions are immutable and digest-pinned.

Functions13

eval::assert::exact

Function

Built-in deterministic evaluator: deep-compare output with arguments.expected.

Request
  • argumentsunknown
  • evaluation_idstringrequired
  • metricsobjectrequired
    • by_sessionobject[]required
      • cache_read_tokensinteger· uint64min 0
      • cache_write_tokensinteger· uint64min 0
      • cost_usdnumber· double
      • depthinteger· uint32requiredmin 0
      • function_call_errorsinteger· uint64requiredmin 0
      • function_callsinteger· uint64requiredmin 0
      • input_tokensinteger· uint64min 0
      • output_tokensinteger· uint64min 0
      • parent_session_idstring
      • reasoning_tokensinteger· uint64min 0
      • session_idstringrequired
      • turnsinteger· uint64requiredmin 0
    • completebooleanrequired
    • root_session_idstringrequired
    • totalsobjectrequired
      • cache_read_tokensinteger· uint64min 0
      • cache_write_tokensinteger· uint64min 0
      • cost_usdnumber· double
      • function_call_errorsinteger· uint64requiredmin 0
      • function_callsinteger· uint64requiredmin 0
      • input_tokensinteger· uint64min 0
      • output_tokensinteger· uint64min 0
      • reasoning_tokensinteger· uint64min 0
      • sessionsinteger· uint64requiredmin 0
      • turnsinteger· uint64requiredmin 0
    • tracesany of

      Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not.

      any of (2)
      Variant 1
      • by_sessionobject[]required
        Show 7 nested
        • depthinteger· uint32requiredmin 0
        • duration_msinteger· uint64requiredmin 0

          Elapsed window from the session's first observed span to its last.

        • error_span_countinteger· uint64requiredmin 0
        • parent_session_idstring
        • session_idstringrequired
        • span_countinteger· uint64requiredmin 0
        • trace_countinteger· uint64requiredmin 0
      • duration_msinteger· uint64requiredmin 0

        Elapsed window from the first observed span to the last observed span.

      • error_span_countinteger· uint64requiredmin 0
      • span_countinteger· uint64requiredmin 0
      • trace_countinteger· uint64requiredmin 0

        Distinct traces across the root session and all descendants.

      Variant 2
      valuenull
  • outputunknownrequired
  • rolestringrequiredenum control, treatment
  • run_idstringrequired
  • session_idstringrequired
Response
  • detailsunknown
  • passedbooleanrequired
  • reasonstring
  • scorenumber· double

eval::assert::normalized_text

Function

Built-in deterministic evaluator: compare text after normalizing case, whitespace, and surrounding punctuation.

Request
  • argumentsunknown
  • evaluation_idstringrequired
  • metricsobjectrequired
    • by_sessionobject[]required
      • cache_read_tokensinteger· uint64min 0
      • cache_write_tokensinteger· uint64min 0
      • cost_usdnumber· double
      • depthinteger· uint32requiredmin 0
      • function_call_errorsinteger· uint64requiredmin 0
      • function_callsinteger· uint64requiredmin 0
      • input_tokensinteger· uint64min 0
      • output_tokensinteger· uint64min 0
      • parent_session_idstring
      • reasoning_tokensinteger· uint64min 0
      • session_idstringrequired
      • turnsinteger· uint64requiredmin 0
    • completebooleanrequired
    • root_session_idstringrequired
    • totalsobjectrequired
      • cache_read_tokensinteger· uint64min 0
      • cache_write_tokensinteger· uint64min 0
      • cost_usdnumber· double
      • function_call_errorsinteger· uint64requiredmin 0
      • function_callsinteger· uint64requiredmin 0
      • input_tokensinteger· uint64min 0
      • output_tokensinteger· uint64min 0
      • reasoning_tokensinteger· uint64min 0
      • sessionsinteger· uint64requiredmin 0
      • turnsinteger· uint64requiredmin 0
    • tracesany of

      Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not.

      any of (2)
      Variant 1
      • by_sessionobject[]required
        Show 7 nested
        • depthinteger· uint32requiredmin 0
        • duration_msinteger· uint64requiredmin 0

          Elapsed window from the session's first observed span to its last.

        • error_span_countinteger· uint64requiredmin 0
        • parent_session_idstring
        • session_idstringrequired
        • span_countinteger· uint64requiredmin 0
        • trace_countinteger· uint64requiredmin 0
      • duration_msinteger· uint64requiredmin 0

        Elapsed window from the first observed span to the last observed span.

      • error_span_countinteger· uint64requiredmin 0
      • span_countinteger· uint64requiredmin 0
      • trace_countinteger· uint64requiredmin 0

        Distinct traces across the root session and all descendants.

      Variant 2
      valuenull
  • outputunknownrequired
  • rolestringrequiredenum control, treatment
  • run_idstringrequired
  • session_idstringrequired
Response
  • detailsunknown
  • passedbooleanrequired
  • reasonstring
  • scorenumber· double

eval::cancel

Function

Cancel an evaluation and its active harness turn.

Request
  • evaluation_idstringrequired
Response
  • cancelledbooleanrequired
  • statusstringrequiredenum queued, running, completed, failed, cancelled

eval::delete

Function

Delete a terminal evaluation report and its session indexes.

Request
  • evaluation_idstringrequired
Response
  • deletedbooleanrequired

eval::list

Function

List recent evaluations as lightweight summaries, newest first.

Request
  • limitinteger· uint32min 0
Response
  • evaluationsobject[]required
    • completed_atinteger· int64
    • control_labelstring
    • created_atinteger· int64required
    • dimensionstringrequiredenum prompt, system_prompt
    • eligibleboolean
    • errorstring
    • evaluation_idstringrequired
    • failed_runsinteger· uint32requiredmin 0
    • modelstringrequired
    • passed_runsinteger· uint32requiredmin 0
    • providerstring
    • statusstringrequiredenum queued, running, completed, failed, cancelled
    • terminal_runsinteger· uint32requiredmin 0
    • total_runsinteger· uint32requiredmin 0
    • treatment_labelstring
    • updated_atinteger· int64required

eval::on-turn-completed

Function

Internal harness turn-completed wake-up.

Request
  • session_idstring
  • terminalboolean
Response
  • wokebooleanrequired

eval::rerun

Function

Repeat a terminal evaluation from its persisted request. Set reverse_order to invert the balanced A/B order. The rerun receives a new evaluation ID and fresh sessions.

Request
  • evaluation_idstringrequired
  • reverse_orderboolean
Response
  • evaluation_idstringrequired
  • statusstringrequiredenum queued, running, completed, failed, cancelled

eval::result

Function

Read an evaluation result. The report is present only after terminal completion.

Request
  • evaluation_idstringrequired
Response
any of (2)
Variant 1
  • progressobjectrequired
    • control_aggregateobjectrequired
      Show 18 nested
      • benchmarked_runsinteger· uint32requiredmin 0
      • evaluated_runsinteger· uint32requiredmin 0
      • median_cost_usdnumber· double
      • median_error_span_countinteger· uint64min 0
      • median_function_call_errorsinteger· uint64min 0
      • median_function_callsinteger· uint64min 0
      • median_input_tokensinteger· uint64min 0
      • median_output_tokensinteger· uint64min 0
      • median_reasoning_tokensinteger· uint64min 0
      • median_scorenumber· double
      • median_span_countinteger· uint64min 0
      • median_total_tokensinteger· uint64min 0
      • median_trace_countinteger· uint64min 0
      • median_trace_duration_msinteger· uint64min 0
      • median_wall_time_msinteger· uint64min 0
      • pass_ratenumber· doublerequired
      • passedinteger· uint32requiredmin 0
      • runsinteger· uint32requiredmin 0
    • deltaobjectrequired
      Show 12 nested
      • median_cost_usdnumber· double
      • median_error_span_countnumber· double
      • median_function_call_errorsnumber· double
      • median_function_callsnumber· double
      • median_reasoning_tokensnumber· double
      • median_scorenumber· double
      • median_span_countnumber· double
      • median_total_tokensnumber· double
      • median_trace_countnumber· double
      • median_trace_duration_msnumber· double
      • median_wall_time_msnumber· double
      • pass_ratenumber· doublerequired
    • effective_execution_orderstring[]required
    • order_sensitivityobjectrequired
      Show 3 nested
      • controlobjectrequired
        Show 5 nested
        • differsbooleanrequired
        • first_passedinteger· uint32requiredmin 0
        • first_runsinteger· uint32requiredmin 0
        • second_passedinteger· uint32requiredmin 0
        • second_runsinteger· uint32requiredmin 0
      • detectedbooleanrequired
      • treatmentobjectrequired
        Show 5 nested
        • differsbooleanrequired
        • first_passedinteger· uint32requiredmin 0
        • first_runsinteger· uint32requiredmin 0
        • second_passedinteger· uint32requiredmin 0
        • second_runsinteger· uint32requiredmin 0
    • runsobject[]required
      Show 16 nested
      • benchmarkany of
        Show 2 nested
        any of (2)
        Variant 1
        • cache_read_tokensinteger· uint64min 0
        • cache_write_tokensinteger· uint64min 0
        • cost_usdnumber· double
        • error_span_countinteger· uint64min 0
        • function_call_errorsinteger· uint64requiredmin 0
        • function_callsinteger· uint64requiredmin 0
        • input_tokensinteger· uint64min 0
        • output_tokensinteger· uint64min 0
        • reasoning_tokensinteger· uint64min 0
        • sessionsinteger· uint64requiredmin 0
        • span_countinteger· uint64min 0
        • total_tokensinteger· uint64min 0
        • trace_countinteger· uint64min 0
        • trace_duration_msinteger· uint64min 0
        • turnsinteger· uint64requiredmin 0
        • wall_time_msinteger· uint64requiredmin 0
        Variant 2
        valuenull
      • completed_atinteger· int64
      • evaluationany of
        Show 2 nested
        any of (2)
        Variant 1
        • detailsunknown
        • passedbooleanrequired
        • reasonstring
        • scorenumber· double
        Variant 2
        valuenull
      • execution_positioninteger· uint32min 0
      • failuresobject[]
        Show 4 nested
        • codestring
        • function_idstring
        • messagestringrequired
        • phasestringrequiredenum setup, send, await, collect, evaluate, limit, …
      • iterationinteger· uint32requiredmin 0
      • metricsany of
        Show 2 nested
        any of (2)
        Variant 1
        • by_sessionobject[]required
          Show 12 nested
          • cache_read_tokensinteger· uint64min 0
          • cache_write_tokensinteger· uint64min 0
          • cost_usdnumber· double
          • depthinteger· uint32requiredmin 0
          • function_call_errorsinteger· uint64requiredmin 0
          • function_callsinteger· uint64requiredmin 0
          • input_tokensinteger· uint64min 0
          • output_tokensinteger· uint64min 0
          • parent_session_idstring
          • reasoning_tokensinteger· uint64min 0
          • session_idstringrequired
          • turnsinteger· uint64requiredmin 0
        • completebooleanrequired
        • root_session_idstringrequired
        • totalsobjectrequired
          Show 10 nested
          • cache_read_tokensinteger· uint64min 0
          • cache_write_tokensinteger· uint64min 0
          • cost_usdnumber· double
          • function_call_errorsinteger· uint64requiredmin 0
          • function_callsinteger· uint64requiredmin 0
          • input_tokensinteger· uint64min 0
          • output_tokensinteger· uint64min 0
          • reasoning_tokensinteger· uint64min 0
          • sessionsinteger· uint64requiredmin 0
          • turnsinteger· uint64requiredmin 0
        • tracesany of

          Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not.

          Show 2 nested
          any of (2)
          Variant 1
          • by_sessionobject[]required
            Show 7 nested
            • depthinteger· uint32requiredmin 0
            • duration_msinteger· uint64requiredmin 0

              Elapsed window from the session's first observed span to its last.

            • error_span_countinteger· uint64requiredmin 0
            • parent_session_idstring
            • session_idstringrequired
            • span_countinteger· uint64requiredmin 0
            • trace_countinteger· uint64requiredmin 0
          • duration_msinteger· uint64requiredmin 0

            Elapsed window from the first observed span to the last observed span.

          • error_span_countinteger· uint64requiredmin 0
          • span_countinteger· uint64requiredmin 0
          • trace_countinteger· uint64requiredmin 0

            Distinct traces across the root session and all descendants.

          Variant 2
          valuenull
        Variant 2
        valuenull
      • outputunknown
      • pair_positioninteger· uint8min 0
      • passedboolean
      • rolestringrequiredenum control, treatment
      • run_idstringrequired
      • session_idstringrequired
      • started_atinteger· int64required
      • statusstringrequiredenum pending, running, completed, failed, cancelled
      • turn_idstring
    • treatment_aggregateobjectrequired
      Show 18 nested
      • benchmarked_runsinteger· uint32requiredmin 0
      • evaluated_runsinteger· uint32requiredmin 0
      • median_cost_usdnumber· double
      • median_error_span_countinteger· uint64min 0
      • median_function_call_errorsinteger· uint64min 0
      • median_function_callsinteger· uint64min 0
      • median_input_tokensinteger· uint64min 0
      • median_output_tokensinteger· uint64min 0
      • median_reasoning_tokensinteger· uint64min 0
      • median_scorenumber· double
      • median_span_countinteger· uint64min 0
      • median_total_tokensinteger· uint64min 0
      • median_trace_countinteger· uint64min 0
      • median_trace_duration_msinteger· uint64min 0
      • median_wall_time_msinteger· uint64min 0
      • pass_ratenumber· doublerequired
      • passedinteger· uint32requiredmin 0
      • runsinteger· uint32requiredmin 0
  • reportany of
    any of (2)
    Variant 1
    • completed_atinteger· int64required
    • controlobjectrequired
      Show 3 nested
      • labelstring
      • prompt_sha256stringrequired
      • system_prompt_sha256string
    • control_aggregateobjectrequired
      Show 18 nested
      • benchmarked_runsinteger· uint32requiredmin 0
      • evaluated_runsinteger· uint32requiredmin 0
      • median_cost_usdnumber· double
      • median_error_span_countinteger· uint64min 0
      • median_function_call_errorsinteger· uint64min 0
      • median_function_callsinteger· uint64min 0
      • median_input_tokensinteger· uint64min 0
      • median_output_tokensinteger· uint64min 0
      • median_reasoning_tokensinteger· uint64min 0
      • median_scorenumber· double
      • median_span_countinteger· uint64min 0
      • median_total_tokensinteger· uint64min 0
      • median_trace_countinteger· uint64min 0
      • median_trace_duration_msinteger· uint64min 0
      • median_wall_time_msinteger· uint64min 0
      • pass_ratenumber· doublerequired
      • passedinteger· uint32requiredmin 0
      • runsinteger· uint32requiredmin 0
    • created_atinteger· int64required
    • deltaall ofrequired

      Treatment minus control. Negative efficiency deltas are improvements.

      Show 1 nested
      all of (1)
      Variant 1
      • median_cost_usdnumber· double
      • median_error_span_countnumber· double
      • median_function_call_errorsnumber· double
      • median_function_callsnumber· double
      • median_reasoning_tokensnumber· double
      • median_scorenumber· double
      • median_span_countnumber· double
      • median_total_tokensnumber· double
      • median_trace_countnumber· double
      • median_trace_duration_msnumber· double
      • median_wall_time_msnumber· double
      • pass_ratenumber· doublerequired
    • dimensionstringrequiredenum prompt, system_prompt
    • effective_execution_orderstring[]required
    • eligibleboolean
    • evaluation_idstringrequired
    • evaluatorany of
      Show 2 nested
      any of (2)
      Variant 1
      • arguments_sha256stringrequired
      • function_idstringrequired
      Variant 2
      valuenull
    • execution_order_policystringrequiredenum balanced_control_first, balanced_treatment_first
    • limitsobjectrequired
      Show 2 nested
      • evaluationall of
        Show 1 nested
        all of (1)
        Variant 1
        • max_error_spansinteger· uint64min 0
        • max_function_call_errorsinteger· uint64min 0
      • executionall of
        Show 1 nested
        all of (1)
        Variant 1
        • invocation_timeout_secondsinteger· uint64min 0
        • max_cost_usdnumber· double
        • max_output_tokens_per_callinteger· uint64min 0
        • max_total_tokensinteger· uint64min 0
        • max_turnsinteger· uint32min 0
        • scenario_timeout_secondsinteger· uint64min 0
    • modelobjectrequired
      Show 6 nested
      • modeany of
        Show 2 nested
        any of (2)
        Variant 1
        valuestringenum ask, agent
        Variant 2
        valuenull
      • modelstringrequired
      • providerstring
      • provider_optionsobject
      • system_prompt_strategyall of
        Show 1 nested
        all of (1)
        Variant 1
        one of (3)
        Variant 1
        valuestringenum override
        Variant 2
        valuestringenum enrich
        Variant 3
        valuestringenum disabled
      • thinking_levelany of
        Show 2 nested
        any of (2)
        Variant 1
        valuestringenum minimal, low, medium, high, xhigh
        Variant 2
        valuenull
    • order_sensitivityobjectrequired
      Show 3 nested
      • controlobjectrequired
        Show 5 nested
        • differsbooleanrequired
        • first_passedinteger· uint32requiredmin 0
        • first_runsinteger· uint32requiredmin 0
        • second_passedinteger· uint32requiredmin 0
        • second_runsinteger· uint32requiredmin 0
      • detectedbooleanrequired
      • treatmentobjectrequired
        Show 5 nested
        • differsbooleanrequired
        • first_passedinteger· uint32requiredmin 0
        • first_runsinteger· uint32requiredmin 0
        • second_passedinteger· uint32requiredmin 0
        • second_runsinteger· uint32requiredmin 0
    • runsobject[]required
      Show 16 nested
      • benchmarkany of
        Show 2 nested
        any of (2)
        Variant 1
        • cache_read_tokensinteger· uint64min 0
        • cache_write_tokensinteger· uint64min 0
        • cost_usdnumber· double
        • error_span_countinteger· uint64min 0
        • function_call_errorsinteger· uint64requiredmin 0
        • function_callsinteger· uint64requiredmin 0
        • input_tokensinteger· uint64min 0
        • output_tokensinteger· uint64min 0
        • reasoning_tokensinteger· uint64min 0
        • sessionsinteger· uint64requiredmin 0
        • span_countinteger· uint64min 0
        • total_tokensinteger· uint64min 0
        • trace_countinteger· uint64min 0
        • trace_duration_msinteger· uint64min 0
        • turnsinteger· uint64requiredmin 0
        • wall_time_msinteger· uint64requiredmin 0
        Variant 2
        valuenull
      • completed_atinteger· int64
      • evaluationany of
        Show 2 nested
        any of (2)
        Variant 1
        • detailsunknown
        • passedbooleanrequired
        • reasonstring
        • scorenumber· double
        Variant 2
        valuenull
      • execution_positioninteger· uint32min 0
      • failuresobject[]
        Show 4 nested
        • codestring
        • function_idstring
        • messagestringrequired
        • phasestringrequiredenum setup, send, await, collect, evaluate, limit, …
      • iterationinteger· uint32requiredmin 0
      • metricsany of
        Show 2 nested
        any of (2)
        Variant 1
        • by_sessionobject[]required
          Show 12 nested
          • cache_read_tokensinteger· uint64min 0
          • cache_write_tokensinteger· uint64min 0
          • cost_usdnumber· double
          • depthinteger· uint32requiredmin 0
          • function_call_errorsinteger· uint64requiredmin 0
          • function_callsinteger· uint64requiredmin 0
          • input_tokensinteger· uint64min 0
          • output_tokensinteger· uint64min 0
          • parent_session_idstring
          • reasoning_tokensinteger· uint64min 0
          • session_idstringrequired
          • turnsinteger· uint64requiredmin 0
        • completebooleanrequired
        • root_session_idstringrequired
        • totalsobjectrequired
          Show 10 nested
          • cache_read_tokensinteger· uint64min 0
          • cache_write_tokensinteger· uint64min 0
          • cost_usdnumber· double
          • function_call_errorsinteger· uint64requiredmin 0
          • function_callsinteger· uint64requiredmin 0
          • input_tokensinteger· uint64min 0
          • output_tokensinteger· uint64min 0
          • reasoning_tokensinteger· uint64min 0
          • sessionsinteger· uint64requiredmin 0
          • turnsinteger· uint64requiredmin 0
        • tracesany of

          Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not.

          Show 2 nested
          any of (2)
          Variant 1
          • by_sessionobject[]required
            Show 7 nested
            • depthinteger· uint32requiredmin 0
            • duration_msinteger· uint64requiredmin 0

              Elapsed window from the session's first observed span to its last.

            • error_span_countinteger· uint64requiredmin 0
            • parent_session_idstring
            • session_idstringrequired
            • span_countinteger· uint64requiredmin 0
            • trace_countinteger· uint64requiredmin 0
          • duration_msinteger· uint64requiredmin 0

            Elapsed window from the first observed span to the last observed span.

          • error_span_countinteger· uint64requiredmin 0
          • span_countinteger· uint64requiredmin 0
          • trace_countinteger· uint64requiredmin 0

            Distinct traces across the root session and all descendants.

          Variant 2
          valuenull
        Variant 2
        valuenull
      • outputunknown
      • pair_positioninteger· uint8min 0
      • passedboolean
      • rolestringrequiredenum control, treatment
      • run_idstringrequired
      • session_idstringrequired
      • started_atinteger· int64required
      • statusstringrequiredenum pending, running, completed, failed, cancelled
      • turn_idstring
    • schema_versionstringrequired
    • shared_artifactsobjectrequired
      Show 4 nested
      • function_policy_sha256stringrequired
      • limits_sha256stringrequired
      • model_sha256stringrequired
      • output_sha256stringrequired
    • source_evaluation_idstring
    • treatmentobjectrequired
      Show 3 nested
      • labelstring
      • prompt_sha256stringrequired
      • system_prompt_sha256string
    • treatment_aggregateobjectrequired
      Show 18 nested
      • benchmarked_runsinteger· uint32requiredmin 0
      • evaluated_runsinteger· uint32requiredmin 0
      • median_cost_usdnumber· double
      • median_error_span_countinteger· uint64min 0
      • median_function_call_errorsinteger· uint64min 0
      • median_function_callsinteger· uint64min 0
      • median_input_tokensinteger· uint64min 0
      • median_output_tokensinteger· uint64min 0
      • median_reasoning_tokensinteger· uint64min 0
      • median_scorenumber· double
      • median_span_countinteger· uint64min 0
      • median_total_tokensinteger· uint64min 0
      • median_trace_countinteger· uint64min 0
      • median_trace_duration_msinteger· uint64min 0
      • median_wall_time_msinteger· uint64min 0
      • pass_ratenumber· doublerequired
      • passedinteger· uint32requiredmin 0
      • runsinteger· uint32requiredmin 0
    Variant 2
    valuenull
  • requestobjectrequired
    • controlobjectrequired
      Show 3 nested
      • labelstring
      • promptstringrequired
      • system_promptstring

        `null` disables the system prompt for this variant. An empty string keeps the harness behavior of resolving the provider or built-in prompt.

    • dimensionstringrequiredenum prompt, system_prompt
    • evaluatorany of
      Show 2 nested
      any of (2)
      Variant 1
      • argumentsunknown
      • function_idstringrequired
      Variant 2
      valuenull
    • execution_orderall of
      Show 1 nested
      all of (1)
      Variant 1
      valuestringenum balanced_control_first, balanced_treatment_first
    • functionsobjectrequired

      The fail-closed dispatch policy (harness.md § Functions). Absent on the send => every call denied (a plain chat loop).

      Show 3 nested
      • allowstring[]
      • denystring[]
      • exposeall of
        Show 1 nested
        all of (1)
        Variant 1
        valuestringenum agent_trigger, native
    • limitsobjectrequired
      Show 2 nested
      • evaluationall of
        Show 1 nested
        all of (1)
        Variant 1
        • max_error_spansinteger· uint64min 0
        • max_function_call_errorsinteger· uint64min 0
      • executionall of
        Show 1 nested
        all of (1)
        Variant 1
        • invocation_timeout_secondsinteger· uint64min 0
        • max_cost_usdnumber· double
        • max_output_tokens_per_callinteger· uint64min 0
        • max_total_tokensinteger· uint64min 0
        • max_turnsinteger· uint32min 0
        • scenario_timeout_secondsinteger· uint64min 0
    • metadataunknown
    • modelobjectrequired
      Show 6 nested
      • modeany of
        Show 2 nested
        any of (2)
        Variant 1
        valuestringenum ask, agent
        Variant 2
        valuenull
      • modelstringrequired
      • providerstring
      • provider_optionsobject
      • system_prompt_strategyall of
        Show 1 nested
        all of (1)
        Variant 1
        one of (3)
        Variant 1
        valuestringenum override
        Variant 2
        valuestringenum enrich
        Variant 3
        valuestringenum disabled
      • thinking_levelany of
        Show 2 nested
        any of (2)
        Variant 1
        valuestringenum minimal, low, medium, high, xhigh
        Variant 2
        valuenull
    • outputone ofrequired

      Free text by default; `json` constrains the final answer to a JSON value, validated against `schema` when supplied.

      Show 2 nested
      one of (2)
      Variant 1
      • typestringrequiredenum text
      Variant 2
      • schemaunknown
      • typestringrequiredenum json
    • runsinteger· uint32requiredmin 0
    • source_evaluation_idstring
    • treatmentobjectrequired
      Show 3 nested
      • labelstring
      • promptstringrequired
      • system_promptstring

        `null` disables the system prompt for this variant. An empty string keeps the harness behavior of resolving the provider or built-in prompt.

  • statusstringrequiredenum queued, running, completed, failed, cancelled
Variant 2
valuenull

eval::start

Function

Start a durable same-model A/B evaluation. Exactly one dimension may change: `prompt` or `system_prompt`. Returns evaluation_id immediately; use eval::status, eval::result, or the eval::completed trigger instead of polling in a tight loop.

Request
  • controlobjectrequired
    • labelstring
    • promptstringrequired
    • system_promptstring

      `null` disables the system prompt for this variant. An empty string keeps the harness behavior of resolving the provider or built-in prompt.

  • dimensionstringrequiredenum prompt, system_prompt
  • evaluatorany of
    any of (2)
    Variant 1
    • argumentsunknown
    • function_idstringrequired
    Variant 2
    valuenull
  • execution_orderany of
    any of (2)
    Variant 1
    valuestringenum balanced_control_first, balanced_treatment_first
    Variant 2
    valuenull
  • functionsany of

    Shared function policy for both variants. Omitted means deny all.

    any of (2)
    Variant 1
    • allowstring[]
    • denystring[]
    • exposeall of
      Show 1 nested
      all of (1)
      Variant 1
      valuestringenum agent_trigger, native
    Variant 2
    valuenull
  • limitsall of
    all of (1)
    Variant 1
    • evaluationall of
      Show 1 nested
      all of (1)
      Variant 1
      • max_error_spansinteger· uint64min 0
      • max_function_call_errorsinteger· uint64min 0
    • executionall of
      Show 1 nested
      all of (1)
      Variant 1
      • invocation_timeout_secondsinteger· uint64min 0
      • max_cost_usdnumber· double
      • max_output_tokens_per_callinteger· uint64min 0
      • max_total_tokensinteger· uint64min 0
      • max_turnsinteger· uint32min 0
      • scenario_timeout_secondsinteger· uint64min 0
  • metadataunknown

    Shared harness metadata, including an optional filesystem scope.

  • modelobjectrequired
    • modeany of
      any of (2)
      Variant 1
      valuestringenum ask, agent
      Variant 2
      valuenull
    • modelstringrequired
    • providerstring
    • provider_optionsobject
    • system_prompt_strategyall of
      all of (1)
      Variant 1
      one of (3)
      Variant 1
      valuestringenum override
      Variant 2
      valuestringenum enrich
      Variant 3
      valuestringenum disabled
    • thinking_levelany of
      any of (2)
      Variant 1
      valuestringenum minimal, low, medium, high, xhigh
      Variant 2
      valuenull
  • outputany of

    Shared output contract for both variants. Omitted means text.

    any of (2)
    Variant 1
    one of (2)
    Variant 1
    • typestringrequiredenum text
    Variant 2
    • schemaunknown
    • typestringrequiredenum json
    Variant 2
    valuenull
  • runsinteger· uint32min 0
  • treatmentobjectrequired
    • labelstring
    • promptstringrequired
    • system_promptstring

      `null` disables the system prompt for this variant. An empty string keeps the harness behavior of resolving the provider or built-in prompt.

Response
  • evaluation_idstringrequired
  • statusstringrequiredenum queued, running, completed, failed, cancelled

eval::status

Function

Read evaluation progress without loading the complete report.

Request
  • evaluation_idstringrequired
Response
any of (2)
Variant 1
  • activeany of
    any of (2)
    Variant 1
    • iterationinteger· uint32requiredmin 0
    • rolestringrequiredenum control, treatment
    • run_idstringrequired
    • session_idstringrequired
    • started_atinteger· int64required
    • turn_idstring
    Variant 2
    valuenull
  • completed_atinteger· int64
  • created_atinteger· int64required
  • errorstring
  • evaluation_idstringrequired
  • failed_runsinteger· uint32requiredmin 0
  • passed_runsinteger· uint32requiredmin 0
  • statusstringrequiredenum queued, running, completed, failed, cancelled
  • terminal_runsinteger· uint32requiredmin 0
  • total_runsinteger· uint32requiredmin 0
  • updated_atinteger· int64required
Variant 2
valuenull

eval::step

Function

Internal durable evaluation step.

Request
  • evaluation_idstringrequired
  • stepinteger· uint64requiredmin 0
Response
  • skippedbooleanrequired
  • statusstringrequiredenum queued, running, completed, failed, cancelled

eval::sweep

Function

Internal recovery and timeout sweep.

Request
  • scheduled_atinteger· int64
  • scheduled_timestring
Response
  • sweptinteger· uint64requiredmin 0

eval::ui-content

Function

Serve the eval worker's injected console UI assets (content function for its console:script / console:style triggers).

Request
  • pathstringrequired

    The asset path from the trigger config (e.g. `state/page.js`).

Response
  • contentstringrequired

    The asset source, verbatim.

  • content_typestringrequired

    MIME type the console should serve the asset with.

Triggers1

eval::completed

Trigger

An evaluation reached a terminal status.

Invocation
  • evaluation_idstring
Return
valueunknown

Search the registry

Find workers, kits and executables, or jump to a page.