Skip to content
Registry

judge

v0.2.10

Provider-neutral evaluation hub (judge::evaluate, judge::cancel, and the internal judge::models::list) that forwards typed Noul, Choice and Score questions to the selected judge-<provider> worker; the default provider is a Console setting seeded from JUDGE_PROVIDER (typesafe). Also resolves worker-defined @<name>(id=…) chat mentions for agents (judge::mentions::resolve and a harness pre-generate hook).

license: Apache-2.0choiceevaluationjudgenoulrelevancescore

Install

bash
iii trigger compose::add worker=judge

Run in your project with the iii engine running. Exact versions are immutable and digest-pinned.

Functions8

judge::cancel

Function

Signal cancellation of an active evaluation or model listing started by the calling worker through the same provider. Returns whether a signal was accepted; does not roll back provider work.

Request
  • providerstring/^[a-z0-9-]{1,64}$/

    Provider worker suffix (judge-<provider>). Defaults to the calling session's provider (OTel baggage iii.judge.provider), then the hub's JUDGE_PROVIDER.

  • request_idstringrequired1…128 chars
Response
one of (2)
Variant 1
  • cancelledbooleanrequired
  • statusstringrequiredenum ok
Variant 2
  • codeone ofrequired
    one of (2)
    Variant 1
    valuestringenum invalid_request, missing_key, payload_too_large, deadline, attempt_timeout, cancelled, …
    Variant 2
    valuestringenum provider_unavailable
  • statusstringrequiredenum error

judge::configuration-id

Function

Returns this worker instance's effective configuration entry ID, without its value.

Request
Empty object
Response
  • idstringrequired

judge::evaluate

Function

Evaluate Noul, Choice and Score questions against arbitrary JSON state through the selected judge-<provider> worker. Results are atomic; stats retain known accepted usage. No credentials are accepted in the request.

Request
  • evaluationsobject[]required1…512 items
    • idstringrequired
    • questionsobjectrequired
    • stateany ofrequired

      Score levels require a description. Unlike instructions, Choice option descriptions and Noul criteria, the provider rejects a null level (HTTP422).

      any of (3)
      Variant 1
      valuestring
      Variant 2
      Empty object
      Variant 3
      itemsunknown[]
  • expires_at_unix_msinteger· uint64min 0
  • modelstring
  • optionsall of
    all of (1)
    Variant 1
    • attempt_timeout_msinteger· uint64min 1

      Bounds each network attempt separately from the whole-call deadline.

  • providerstring/^[a-z0-9-]{1,64}$/

    Provider worker suffix (judge-<provider>). Defaults to the calling session's provider (OTel baggage iii.judge.provider), then the hub's JUDGE_PROVIDER.

  • request_idstring1…128 chars
  • timeout_msinteger· uint64requiredmin 1
Response
one of (2)
Variant 1
  • modelstringrequired
  • resultsobjectrequired
  • statsobjectrequired

    Known accepted usage, never a billing estimate. Unreported counters or failure/cancellation make usage_complete false.

    • attemptsinteger· uintrequiredmin 0
    • elapsed_msinteger· uint64requiredmin 0
    • input_tokensinteger· uint64requiredmin 0
    • output_tokensinteger· uint64requiredmin 0
    • questionsinteger· uintrequiredmin 0
    • requestsinteger· uintrequiredmin 0
    • usage_completebooleanrequired
  • statusstringrequiredenum ok
Variant 2
  • codeone ofrequired
    one of (2)
    Variant 1
    valuestringenum invalid_request, missing_key, payload_too_large, deadline, attempt_timeout, cancelled, …
    Variant 2
    valuestringenum provider_unavailable
  • http_statusinteger· uint16min 0
  • provider_errorany of
    any of (2)
    Variant 1
    • detailunknown
    • messagestring
    • truncatedboolean
    Variant 2
    valuenull
  • retry_after_msinteger· uint64min 0
  • statsobjectrequired

    Known accepted usage, never a billing estimate. Unreported counters or failure/cancellation make usage_complete false.

    • attemptsinteger· uintrequiredmin 0
    • elapsed_msinteger· uint64requiredmin 0
    • input_tokensinteger· uint64requiredmin 0
    • output_tokensinteger· uint64requiredmin 0
    • questionsinteger· uintrequiredmin 0
    • requestsinteger· uintrequiredmin 0
    • usage_completebooleanrequired
  • statusstringrequiredenum error

judge::mentions::pre-generate

Function

Internal: harness pre-generate hook — tells agents which @<name>(id=…) mentions they may write and resolves the ones users wrote into summaries plus pre-verified details calls.

Request
  • generateany of
    any of (2)
    Variant 1
    • messagesunknown[]

      The window this step sends, earlier hook notices replayed in place.

    • toolsunknown[]

      The concrete functions this session may call (`{ name, … }`).

    Variant 2
    valuenull
  • session_idstring
  • stepinteger· uint64min 0
Response
  • annotationsobject
  • decisionstringrequired
  • mutationsany of
    any of (2)
    Variant 1
    • append_messagesunknown[]required
    Variant 2
    valuenull

judge::mentions::resolve

Function

Resolve @<name>(id="…") chat mentions (found in `text`, or listed) through the workers that provide them: each one's status, one-line summary and the call that returns its full details.

Request
  • mentionsobject[]

    Mentions to resolve, in addition to those in `text`.

    • idstringrequired
    • namestringrequired
  • textstring

    Text to find mentions in (markdown code is skipped).

Response
  • mentionsobject[]required
    • detailsany of
      any of (2)
      Variant 1
      • function_idstringrequired
      • payloadunknownrequired
      Variant 2
      valuenull
    • errorstring

      Why it did not resolve, when it errored.

    • idstringrequired
    • labelstring

      What the provider calls this kind of item ("Tickets").

    • namestringrequired
    • statusstringrequiredenum resolved, not_found, unknown_provider, error

      What became of one mention.

    • summarystring

      The agent-facing line, when resolved.

    • tokenstringrequired

      The canonical token (`@<name>(id="<id>")`).

judge::models::list

Function

List the selected provider's available model names, descriptions and release dates; performs no inference.

Request
  • expires_at_unix_msinteger· uint64min 0
  • optionsall of
    all of (1)
    Variant 1
    • attempt_timeout_msinteger· uint64min 1

      Bounds each network attempt separately from the whole-call deadline.

  • providerstring/^[a-z0-9-]{1,64}$/

    Provider worker suffix (judge-<provider>). Defaults to the calling session's provider (OTel baggage iii.judge.provider), then the hub's JUDGE_PROVIDER.

  • request_idstring1…128 chars
  • timeout_msinteger· uint64min 1
Response
one of (2)
Variant 1
  • modelsobject[]required
    • context_windowinteger· uint32min 0

      Tokens one evaluation row can hold (question, options and state), when the provider has a fixed window. Callers size their states from it.

    • descriptionstringrequired
    • max_optionsinteger· uint32min 0

      Most options one Choice question (or levels one Score) can offer, when the provider has a fixed limit below the contract's 255. Callers size their option groups from it.

    • namestringrequired
    • release_datestringrequired
  • statsobjectrequired

    Known accepted usage, never a billing estimate. Unreported counters or failure/cancellation make usage_complete false.

    • attemptsinteger· uintrequiredmin 0
    • elapsed_msinteger· uint64requiredmin 0
    • input_tokensinteger· uint64requiredmin 0
    • output_tokensinteger· uint64requiredmin 0
    • questionsinteger· uintrequiredmin 0
    • requestsinteger· uintrequiredmin 0
    • usage_completebooleanrequired
  • statusstringrequiredenum ok
Variant 2
  • codeone ofrequired
    one of (2)
    Variant 1
    valuestringenum invalid_request, missing_key, payload_too_large, deadline, attempt_timeout, cancelled, …
    Variant 2
    valuestringenum provider_unavailable
  • http_statusinteger· uint16min 0
  • provider_errorany of
    any of (2)
    Variant 1
    • detailunknown
    • messagestring
    • truncatedboolean
    Variant 2
    valuenull
  • retry_after_msinteger· uint64min 0
  • statsobjectrequired

    Known accepted usage, never a billing estimate. Unreported counters or failure/cancellation make usage_complete false.

    • attemptsinteger· uintrequiredmin 0
    • elapsed_msinteger· uint64requiredmin 0
    • input_tokensinteger· uint64requiredmin 0
    • output_tokensinteger· uint64requiredmin 0
    • questionsinteger· uintrequiredmin 0
    • requestsinteger· uintrequiredmin 0
    • usage_completebooleanrequired
  • statusstringrequiredenum error

judge::on-config-change

Function

Internal: reload judge configuration from authoritative storage.

Request
  • idstring

    Configuration id that changed.

Response
  • okbooleanrequired

judge::ui-content

Function

Serve the judge worker's injected console UI assets (content function for its console:script / console:style / console:module triggers).

Request
  • pathstringrequired

    The asset path from the trigger config (e.g. `state/page.js`).

Response
  • contentstringrequired

    The asset source, verbatim.

  • content_typestringrequired

    MIME type the console should serve the asset with.

Triggers0

No triggers registered

Search the registry

Find workers, kits and executables, or jump to a page.