eval

eval primarily compares 2–5 existing root sessions live. It reads the current session lifecycle and harness::metrics values, then displays objective deltas against a user-selected reference. The user decides whether the sessions are comparable; eval does not score, rank, validate equivalence, or persist a comparison. Prompt and system-prompt experiments remain available as an advanced surface.

Start an evaluation

{
  "dimension": "prompt",
  "model": {
    "model": "codex/gpt-5.6-luna",
    "provider": "openai-codex",
    "system_prompt_strategy": "override"
  },
  "control": {
    "label": "baseline",
    "prompt": "Reply with exactly OK.",
    "system_prompt": "Follow the user request exactly."
  },
  "treatment": {
    "label": "candidate",
    "prompt": "Return exactly the text OK.",
    "system_prompt": "Follow the user request exactly."
  },
  "evaluator": {
    "function_id": "eval::assert::exact",
    "arguments": {
      "expected": "OK"
    }
  }
}

Call eval::start with the request. It returns an evaluation_id immediately. Use eval::status for occasional progress checks and eval::result for the terminal report, or bind to the eval::completed trigger type.

One run per variant is scheduled by default. Their order alternates by pair to reduce order bias. A candidate is eligible only when every treatment run passes and its pass count does not regress against control. Efficiency metrics are descriptive and never select a winner automatically.

Public functions

• eval::compare-sessions — read 2–5 root sessions concurrently and return objective metrics plus arithmetic deltas against the selected reference.
• eval::start — validate, persist, and enqueue an evaluation.
• eval::list — list recent evaluations as lightweight summaries.
• eval::status — inspect progress without loading the full report.
• eval::result — read the normalized request and terminal comparison report.
• eval::cancel — cancel the active harness session and remaining runs.
• eval::delete — delete a terminal evaluation and its session indexes.
• eval::assert::exact — built-in deep JSON/string equality evaluator.

Evaluator functions receive the output, harness::metrics, run identity, and caller-supplied arguments. They return { passed, score?, reason?, details? }. Evaluators should be deterministic and idempotent because durable delivery is at-least-once.

Console UI

When the console worker is running, eval injects an eval page at #/ext/eval-benchmarks. The default Sessions tab lists visible root sessions, accepts active sessions, and renders a live matrix grouped by efficiency, reliability, orchestration, and context. Prompt experiments keeps the durable prompt/system-prompt workflow for advanced use.

The model picker reads the live router::models::list catalog and falls back to manual model/provider entry when the catalog is unavailable. Exact-value evaluation is built in; any evaluator function can also be selected by id with JSON arguments. Harness policies and output/metadata options remain collapsed until needed.

Boundaries

The worker intentionally does not implement an agent loop, model router, metrics collector, trace collector, test DSL, or LLM judge. Those concerns remain in the harness, engine, and user-provided evaluator functions.