Skip to content
Registry

judge

v0.2.4

Provider-neutral evaluation hub (judge::evaluate, judge::cancel, and the internal judge::models::list) that forwards typed Noul, Choice and Score questions to the selected judge-<provider> worker; the default provider is a Console setting seeded from JUDGE_PROVIDER (typesafe). Also resolves worker-defined @<name>(id=…) chat mentions for agents (judge::mentions::resolve and a harness pre-generate hook).

license: Apache-2.0choiceevaluationjudgenoulrelevancescore

Install

bash
iii trigger compose::add worker=judge@0.2.4

Run in your project with the iii engine running. Exact versions are immutable and digest-pinned.

Functions1

judge::evaluate

Function

Evaluate Noul, Choice and Score questions against arbitrary JSON state through the selected judge-<provider> worker. Results are atomic; stats retain known accepted usage. No credentials are accepted in the request.

Request
  • evaluationsobject[]required1…512 items
    • idstringrequired
    • questionsobjectrequired
    • stateany ofrequired

      Score levels require a description. Unlike instructions, Choice option descriptions and Noul criteria, the provider rejects a null level (HTTP422).

      any of (3)
      Variant 1
      valuestring
      Variant 2
      Empty object
      Variant 3
      itemsunknown[]
  • expires_at_unix_msinteger· uint64min 0
  • modelstring
  • optionsall of
    all of (1)
    Variant 1
    • attempt_timeout_msinteger· uint64min 1

      Bounds each network attempt separately from the whole-call deadline.

  • providerstring/^[a-z0-9-]{1,64}$/

    Provider worker suffix (judge-<provider>). Defaults to the calling session's provider (OTel baggage iii.judge.provider), then the hub's JUDGE_PROVIDER.

  • request_idstring1…128 chars
  • timeout_msinteger· uint64requiredmin 1
Response
one of (2)
Variant 1
  • modelstringrequired
  • resultsobjectrequired
  • statsobjectrequired

    Known accepted usage, never a billing estimate. Unreported counters or failure/cancellation make usage_complete false.

    • attemptsinteger· uintrequiredmin 0
    • elapsed_msinteger· uint64requiredmin 0
    • input_tokensinteger· uint64requiredmin 0
    • output_tokensinteger· uint64requiredmin 0
    • questionsinteger· uintrequiredmin 0
    • requestsinteger· uintrequiredmin 0
    • usage_completebooleanrequired
  • statusstringrequiredenum ok
Variant 2
  • codeone ofrequired
    one of (2)
    Variant 1
    valuestringenum invalid_request, missing_key, payload_too_large, deadline, attempt_timeout, cancelled, …
    Variant 2
    valuestringenum provider_unavailable
  • http_statusinteger· uint16min 0
  • provider_errorany of
    any of (2)
    Variant 1
    • detailunknown
    • messagestring
    • truncatedboolean
    Variant 2
    valuenull
  • retry_after_msinteger· uint64min 0
  • statsobjectrequired

    Known accepted usage, never a billing estimate. Unreported counters or failure/cancellation make usage_complete false.

    • attemptsinteger· uintrequiredmin 0
    • elapsed_msinteger· uint64requiredmin 0
    • input_tokensinteger· uint64requiredmin 0
    • output_tokensinteger· uint64requiredmin 0
    • questionsinteger· uintrequiredmin 0
    • requestsinteger· uintrequiredmin 0
    • usage_completebooleanrequired
  • statusstringrequiredenum error

Triggers0

No triggers registered

Search the registry

Find workers, kits and executables, or jump to a page.