judge

judge turns JSON state and Noul, Choice or Score questions into typed, validated answers for ticket triage, content checks and relevance judgments. It is a provider-neutral hub: every call is forwarded to a judge- worker that owns the model, credentials and transport. The default provider is judge-typesafe (TypeSafe JEV); other strategies plug in by registering the same three functions under their own judge- id.

Install

iii trigger compose::add worker=judge

Compose starts the hub with its judge-typesafe dependency, which in turn needs configuration. Use engine iii/v0.24.0-rc.2, the verified release used by this repository's CI. Wait for judge::evaluate to register before calling it.

Quickstart

Set the TypeSafe key in the Console under Settings → Workers → judge-typesafe or supply TYPESAFE_API_KEY in that worker's environment (see the provider README). Then ask whether a support ticket needs urgent attention:

iii trigger judge::evaluate --timeout-ms 65000 --json '{
  "timeout_ms": 60000,
  "evaluations": [{
    "id": "ticket-42",
    "state": {"ticket": "Production checkout is down for all customers."},
    "questions": {
      "urgent": {
        "type": "noul",
        "instructions": "Does this describe an urgent production outage?",
        "criteria": {"true": "A critical production workflow is blocked."}
      }
    }
  }]
}'

A successful response looks like this; scores, usage and timing are illustrative:

{
  "status": "ok", "model": "jev-1.13.0",
  "results": {"ticket-42": {
    "answers": {"urgent": {"type": "noul", "noul": 0.98}},
    "usage": {"input_tokens": 120, "output_tokens": 8}
  }},
  "stats": {"attempts": 1, "requests": 1, "questions": 1,
    "input_tokens": 120, "output_tokens": 8, "elapsed_ms": 250,
    "usage_complete": true}
}

Check status before reading answers. Errors return status: "error" and a code with no partial results; provider_unavailable means the selected judge- worker is not registered on the engine (or, for a local provider, its model failed to load). A bus invocation failure is handled separately. The hub adds 5 s of bus slack on top of your timeout_ms; the provider enforces the deadline itself.

See the mixed Noul/Choice/Score example, result and error handling, model listing and cancellation.

Providers

Settings → Workers → judge selects the default provider from the workers registered as judge- (seeded from JUDGE_PROVIDER, else typesafe). Six ship today: judge-typesafe (TypeSafe's hosted JEV), judge-openai (OpenAI's hosted Decisions API), judge-decider, judge-semif, judge-laya and judge-clef (open models running inside the worker); a request may name its own with a top-level provider. Between the two sits the calling session's provider: the console's composer (beside the model picker) stores it as the session's judge_provider metadata, the harness stamps it on every turn as the iii.judge.provider OTel baggage, and callers in that turn (function search, call reconciliation, browser::run) send it as provider; a request without one falls back to that baggage, then to the default. Each session routes on its own; no session changes another's judge or the default. A local provider keeps its model loaded while it is the default (all of them with Keep every local provider loaded, preload_all); a request naming any other one loads its model on first use, which that request may not outlast, and the model is released after 10 idle minutes. Picking a provider in the composer starts that load. The composer's judge picker is laid out like the model picker: each provider shows the model its settings name, Configure edits a provider's settings (or the hub's) inside the picker, and + adds a judge worker from the registry. A new provider is a worker that registers judge-::evaluate, ::models::list and ::cancel with the judge-contract types, marked metadata.internal: true so default discovery shows only the hub, and accepts request ids up to 512 bytes; the hub needs no change. See Configuration and Cancellation.

Chat mentions

Because the harness always runs the judge, the judge also delivers chat mentions to agents: a user's @kanban(id="…") (or any worker-defined mention) reaches the agent with a one-line summary of the item and a pre-verified call for its full details, and agents learn which mention names they may write in replies. See Chat mentions; JUDGE_MENTIONS=false turns it off.

The hub holds no credentials; its configuration entry (judge, or III_CONFIG_NAME) carries only the default provider and preload_all. For the full API, read reference.md; for the provider's build, configuration and tests, read judge-typesafe.