judge-typesafe
v0.2.0TypeSafe evaluation (judge-typesafe::evaluate), model listing (judge-typesafe::models::list) and caller-scoped cancellation (judge-typesafe::cancel), with typed usage, provider errors, per-call retries and worker-owned credentials.
Install
bash
iii trigger compose::add worker=judge-typesafe@0.2.0Run in your project with the iii engine running. Exact versions are immutable and digest-pinned.
Agent-ready brief for v0.2.0
Install, config, dependencies, readme, skills and API reference, all in one place. Fetch as agent-context.md for an LLM to consume.
Full markdown
The same content rendered as discrete blocks below is exposed as a single markdown document at
/workers/judge-typesafe.md?version=0.2.0. Paste it into an LLM prompt or pipe it through curl from a worker.Install
iii trigger compose::add worker=judge-typesafe@0.2.0Configuration
iii-config.yaml
- api_key: null
max_request_bytes: 8388608
max_response_bytes: 8388608
max_timeout_ms: 300000
model: jev-latestDependencies
Dependency · 1Version
Readme
README.md
judge-typesafe
TypeSafe JEV provider for the judge hub. It turns JSON state and
Noul, Choice or Score questions into typed, validated answers, sharing TypeSafe
credentials and HTTP capacity across callers, with model listing and
caller-owned cancellation. Callers normally go through judge::evaluate; the
judge-typesafe::* functions below are the same contract addressed directly.
Install
iii trigger compose::add worker=judge-typesafeCompose starts the provider with its configuration dependency; installing
judge pulls it in automatically. Use engine
iii/v0.24.0-rc.2, the verified release with configuration::ensure used by
this repository's CI. Wait for JEV's functions to register before making a call.
Quickstart
In the Console, open Settings → Workers → Judge TypeSafe and set API key
(the form checks the saved key against judge-typesafe::models::list and offers
the answered catalog in the Default model select), or supply
TYPESAFE_API_KEY in the JEV service's environment before starting it. See
Configuration for precedence and reload behavior.
Ask whether a support ticket needs urgent attention:
iii trigger judge-typesafe::evaluate --timeout-ms 65000 --json '{
"timeout_ms": 60000,
"evaluations": [{
"id": "ticket-42",
"state": {"ticket": "Production checkout is down for all customers."},
"questions": {
"urgent": {
"type": "noul",
"instructions": "Does this describe an urgent production outage?",
"criteria": {"true": "A critical production workflow is blocked."}
}
}
}]
}'A successful response looks like this; scores, usage and timing are illustrative:
{
"status": "ok", "model": "jev-1.13.0",
"results": {"ticket-42": {
"answers": {"urgent": {"type": "noul", "noul": 0.98}},
"usage": {"input_tokens": 120, "output_tokens": 8}
}},
"stats": {"attempts": 1, "requests": 1, "questions": 1,
"input_tokens": 120, "output_tokens": 8, "elapsed_ms": 250,
"usage_complete": true}
}Check status before reading answers. A Noul value is between 0 and 1; choose
your own decision threshold. Errors return status: "error" and a code with no
partial results; a bus invocation failure is handled separately. The payload
budget covers the whole call, including queueing and retries; the longer CLI
timeout leaves room for transport overhead.
See the hub's mixed Noul/Choice/Score example,
result and error handling,
model listing and
cancellation. Through the hub, request_id
arrives here as judge_contract::MAX_PROVIDER_REQUEST_ID_BYTES).
Configuration
The Console entry defaults to judge-typesafe; set III_CONFIG_NAME=judge-typesafe-prod in the worker
environment to use judge-typesafe-prod. The form masks the API key and exposes these defaults:
api_key: null # Fall back to the worker's TYPESAFE_API_KEY.
model: jev-latest # Default unless a call supplies its own model.
max_request_bytes: 8388608 # Maximum JSON bytes per upstream evaluation.
max_response_bytes: 8388608 # Maximum bytes per upstream response.
max_timeout_ms: 300000 # Whole-call timeout ceiling in milliseconds.A nonblank configured key overrides the worker's environment key. Clearing it restores the fallback. All configuration fields hot-reload for new calls; in-flight calls keep their snapshot, and rejected reloads keep the last valid value. Changing the process environment requires a restart.
The configuration service persists values at ./config/
with its default filesystem adapter. Keep literal API keys out of committed
project configuration.
The form retains unknown values when editing other fields and shows errors
returned by the configuration service; a ${TYPESAFE_API_KEY} value expands in
the configuration service's environment instead of this process.
Consumer compatibility
Callers use the shared judge-contract request and response types. See
shared-contract compatibility
when updating existing consumers. JEV supplies evaluations; each caller owns its
thresholds, ranking decisions and fallback policy.
For the full API, read the hub's reference.md. For source builds, UI work and local test commands, read CONTRIBUTING.md.
API reference (JSON)
agent-api-reference.json
{
"functions": [
{
"description": "Signal cancellation of an active evaluation or model listing owned by the calling worker. Returns whether a signal was accepted; does not roll back provider work. Requires the same worker replica as the original call.",
"metadata": {},
"name": "judge-typesafe::cancel",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"request_id": {
"maxLength": 128,
"minLength": 1,
"type": "string"
}
},
"required": [
"request_id"
],
"title": "CancelRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ErrorCode": {
"oneOf": [
{
"enum": [
"invalid_request",
"missing_key",
"payload_too_large",
"deadline",
"attempt_timeout",
"cancelled",
"http",
"transport",
"invalid_response"
],
"type": "string"
},
{
"description": "The selected `judge-<provider>` worker is not registered on the engine.",
"enum": [
"provider_unavailable"
],
"type": "string"
}
]
}
},
"oneOf": [
{
"additionalProperties": false,
"properties": {
"cancelled": {
"type": "boolean"
},
"status": {
"enum": [
"ok"
],
"type": "string"
}
},
"required": [
"cancelled",
"status"
],
"type": "object"
},
{
"additionalProperties": false,
"properties": {
"code": {
"$ref": "#/definitions/ErrorCode"
},
"status": {
"enum": [
"error"
],
"type": "string"
}
},
"required": [
"code",
"status"
],
"type": "object"
}
],
"title": "CancelResponse"
}
},
{
"description": "Returns this worker instance's effective configuration entry ID, without its value.",
"metadata": {
"internal": true
},
"name": "judge-typesafe::configuration-id",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Empty user request. Ignore transport metadata such as `_caller_worker_id`, which the engine adds to routed calls; no payload field selects the entry.",
"title": "ConfigurationIdentityRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Only the entry ID is exposed; configuration values and secrets stay in the service.",
"properties": {
"id": {
"type": "string"
}
},
"required": [
"id"
],
"title": "ConfigurationIdentityResponse",
"type": "object"
}
},
{
"description": "Evaluate Noul, Choice and Score questions against arbitrary JSON state using JEV. Results are atomic; stats retain known accepted usage and mark unknown counters incomplete. No credentials are accepted in the request.",
"metadata": {},
"name": "judge-typesafe::evaluate",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"Content": {
"anyOf": [
{
"type": "string"
},
{
"additionalProperties": true,
"type": "object"
},
{
"items": true,
"type": "array"
},
{
"type": "null"
}
],
"description": "Structured provider descriptions; top-level numbers/bools are not descriptions."
},
"Evaluation": {
"additionalProperties": false,
"properties": {
"id": {
"type": "string"
},
"questions": {
"additionalProperties": {
"$ref": "#/definitions/Question"
},
"minProperties": 1,
"type": "object"
},
"state": {
"$ref": "#/definitions/ScoreLevel"
}
},
"required": [
"id",
"questions",
"state"
],
"type": "object"
},
"Question": {
"oneOf": [
{
"additionalProperties": false,
"properties": {
"criteria": {
"anyOf": [
{
"additionalProperties": false,
"properties": {
"false": {
"$ref": "#/definitions/Content"
},
"true": {
"$ref": "#/definitions/Content"
}
},
"type": "object"
},
{
"type": "null"
}
]
},
"instructions": {
"allOf": [
{
"$ref": "#/definitions/Content"
}
],
"default": null
},
"type": {
"enum": [
"noul"
],
"type": "string"
}
},
"required": [
"type"
],
"type": "object"
},
{
"additionalProperties": false,
"properties": {
"criteria": {
"additionalProperties": {
"$ref": "#/definitions/Content"
},
"maxProperties": 255,
"minProperties": 1,
"type": "object"
},
"instructions": {
"allOf": [
{
"$ref": "#/definitions/Content"
}
],
"default": null
},
"type": {
"enum": [
"choice"
],
"type": "string"
}
},
"required": [
"criteria",
"type"
],
"type": "object"
},
{
"additionalProperties": false,
"properties": {
"criteria": {
"items": {
"$ref": "#/definitions/ScoreLevel"
},
"maxItems": 10,
"minItems": 2,
"type": "array"
},
"instructions": {
"allOf": [
{
"$ref": "#/definitions/Content"
}
],
"default": null
},
"type": {
"enum": [
"score"
],
"type": "string"
}
},
"required": [
"criteria",
"type"
],
"type": "object"
}
]
},
"RequestOptions": {
"additionalProperties": false,
"properties": {
"attempt_timeout_ms": {
"description": "Bounds each network attempt separately from the whole-call deadline.",
"format": "uint64",
"minimum": 1,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"ScoreLevel": {
"anyOf": [
{
"type": "string"
},
{
"additionalProperties": true,
"type": "object"
},
{
"items": true,
"type": "array"
}
],
"description": "Score levels require a description. Unlike instructions, Choice option descriptions and Noul criteria, the provider rejects a null level (HTTP422)."
}
},
"properties": {
"evaluations": {
"items": {
"$ref": "#/definitions/Evaluation"
},
"maxItems": 512,
"minItems": 1,
"type": "array"
},
"expires_at_unix_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"model": {
"type": [
"string",
"null"
]
},
"options": {
"allOf": [
{
"$ref": "#/definitions/RequestOptions"
}
],
"default": {}
},
"request_id": {
"maxLength": 128,
"minLength": 1,
"type": [
"string",
"null"
]
},
"timeout_ms": {
"format": "uint64",
"minimum": 1,
"type": "integer"
}
},
"required": [
"evaluations",
"timeout_ms"
],
"title": "EvaluateRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"Answer": {
"oneOf": [
{
"additionalProperties": false,
"properties": {
"noul": {
"format": "double",
"type": "number"
},
"type": {
"enum": [
"noul"
],
"type": "string"
}
},
"required": [
"noul",
"type"
],
"type": "object"
},
{
"additionalProperties": false,
"properties": {
"choice": {
"type": "string"
},
"confidence": {
"format": "double",
"type": "number"
},
"probabilities": {
"additionalProperties": {
"format": "double",
"type": "number"
},
"type": "object"
},
"type": {
"enum": [
"choice"
],
"type": "string"
}
},
"required": [
"choice",
"confidence",
"probabilities",
"type"
],
"type": "object"
},
{
"additionalProperties": false,
"properties": {
"confidence": {
"format": "double",
"type": "number"
},
"legend": {
"additionalProperties": {
"$ref": "#/definitions/ScoreLevel"
},
"type": "object"
},
"probabilities": {
"additionalProperties": {
"format": "double",
"type": "number"
},
"type": "object"
},
"score": {
"format": "double",
"type": "number"
},
"type": {
"enum": [
"score"
],
"type": "string"
}
},
"required": [
"confidence",
"legend",
"probabilities",
"score",
"type"
],
"type": "object"
}
]
},
"ErrorCode": {
"oneOf": [
{
"enum": [
"invalid_request",
"missing_key",
"payload_too_large",
"deadline",
"attempt_timeout",
"cancelled",
"http",
"transport",
"invalid_response"
],
"type": "string"
},
{
"description": "The selected `judge-<provider>` worker is not registered on the engine.",
"enum": [
"provider_unavailable"
],
"type": "string"
}
]
},
"EvaluationResult": {
"additionalProperties": false,
"properties": {
"answers": {
"additionalProperties": {
"$ref": "#/definitions/Answer"
},
"type": "object"
},
"usage": {
"anyOf": [
{
"$ref": "#/definitions/Usage"
},
{
"type": "null"
}
]
}
},
"required": [
"answers"
],
"type": "object"
},
"ProviderError": {
"additionalProperties": false,
"description": "Bounded provider diagnostics, with credentials and supplied header values redacted.",
"properties": {
"detail": true,
"message": {
"type": [
"string",
"null"
]
},
"truncated": {
"default": false,
"type": "boolean"
}
},
"type": "object"
},
"ScoreLevel": {
"anyOf": [
{
"type": "string"
},
{
"additionalProperties": true,
"type": "object"
},
{
"items": true,
"type": "array"
}
],
"description": "Score levels require a description. Unlike instructions, Choice option descriptions and Noul criteria, the provider rejects a null level (HTTP422)."
},
"Stats": {
"additionalProperties": false,
"description": "Known accepted usage, never a billing estimate. Unreported counters or failure/cancellation make usage_complete false.",
"properties": {
"attempts": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"elapsed_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"questions": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"requests": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"usage_complete": {
"type": "boolean"
}
},
"required": [
"attempts",
"elapsed_ms",
"input_tokens",
"output_tokens",
"questions",
"requests",
"usage_complete"
],
"type": "object"
},
"Usage": {
"additionalProperties": false,
"description": "Each absent/null counter is unknown, independently of the other counter.",
"properties": {
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
}
},
"oneOf": [
{
"additionalProperties": false,
"properties": {
"model": {
"type": "string"
},
"results": {
"additionalProperties": {
"$ref": "#/definitions/EvaluationResult"
},
"type": "object"
},
"stats": {
"$ref": "#/definitions/Stats"
},
"status": {
"enum": [
"ok"
],
"type": "string"
}
},
"required": [
"model",
"results",
"stats",
"status"
],
"type": "object"
},
{
"additionalProperties": false,
"properties": {
"code": {
"$ref": "#/definitions/ErrorCode"
},
"http_status": {
"format": "uint16",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"provider_error": {
"anyOf": [
{
"$ref": "#/definitions/ProviderError"
},
{
"type": "null"
}
]
},
"retry_after_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stats": {
"$ref": "#/definitions/Stats"
},
"status": {
"enum": [
"error"
],
"type": "string"
}
},
"required": [
"code",
"stats",
"status"
],
"type": "object"
}
],
"title": "EvaluateResponse"
}
},
{
"description": "List the provider's available model names, descriptions and release dates. Shares evaluation credentials, transport limits and permits; performs no inference.",
"metadata": {},
"name": "judge-typesafe::models::list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"RequestOptions": {
"additionalProperties": false,
"properties": {
"attempt_timeout_ms": {
"description": "Bounds each network attempt separately from the whole-call deadline.",
"format": "uint64",
"minimum": 1,
"type": [
"integer",
"null"
]
}
},
"type": "object"
}
},
"properties": {
"expires_at_unix_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"options": {
"allOf": [
{
"$ref": "#/definitions/RequestOptions"
}
],
"default": {}
},
"request_id": {
"maxLength": 128,
"minLength": 1,
"type": [
"string",
"null"
]
},
"timeout_ms": {
"default": 30000,
"format": "uint64",
"minimum": 1,
"type": "integer"
}
},
"title": "ModelsRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ErrorCode": {
"oneOf": [
{
"enum": [
"invalid_request",
"missing_key",
"payload_too_large",
"deadline",
"attempt_timeout",
"cancelled",
"http",
"transport",
"invalid_response"
],
"type": "string"
},
{
"description": "The selected `judge-<provider>` worker is not registered on the engine.",
"enum": [
"provider_unavailable"
],
"type": "string"
}
]
},
"ModelCard": {
"properties": {
"description": {
"type": "string"
},
"name": {
"type": "string"
},
"release_date": {
"type": "string"
}
},
"required": [
"description",
"name",
"release_date"
],
"type": "object"
},
"ProviderError": {
"additionalProperties": false,
"description": "Bounded provider diagnostics, with credentials and supplied header values redacted.",
"properties": {
"detail": true,
"message": {
"type": [
"string",
"null"
]
},
"truncated": {
"default": false,
"type": "boolean"
}
},
"type": "object"
},
"Stats": {
"additionalProperties": false,
"description": "Known accepted usage, never a billing estimate. Unreported counters or failure/cancellation make usage_complete false.",
"properties": {
"attempts": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"elapsed_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"questions": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"requests": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"usage_complete": {
"type": "boolean"
}
},
"required": [
"attempts",
"elapsed_ms",
"input_tokens",
"output_tokens",
"questions",
"requests",
"usage_complete"
],
"type": "object"
}
},
"oneOf": [
{
"additionalProperties": false,
"properties": {
"models": {
"items": {
"$ref": "#/definitions/ModelCard"
},
"type": "array"
},
"stats": {
"$ref": "#/definitions/Stats"
},
"status": {
"enum": [
"ok"
],
"type": "string"
}
},
"required": [
"models",
"stats",
"status"
],
"type": "object"
},
{
"additionalProperties": false,
"properties": {
"code": {
"$ref": "#/definitions/ErrorCode"
},
"http_status": {
"format": "uint16",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"provider_error": {
"anyOf": [
{
"$ref": "#/definitions/ProviderError"
},
{
"type": "null"
}
]
},
"retry_after_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stats": {
"$ref": "#/definitions/Stats"
},
"status": {
"enum": [
"error"
],
"type": "string"
}
},
"required": [
"code",
"stats",
"status"
],
"type": "object"
}
],
"title": "ModelsResponse"
}
},
{
"description": "Internal: reload JEV configuration from authoritative storage.",
"metadata": {
"internal": true
},
"name": "judge-typesafe::on-config-change",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Trigger payload for `<worker>::on-config-change`. Advisory only: handlers re-fetch the authoritative value and ignore it, so a direct call can never inject config.",
"properties": {
"id": {
"default": null,
"description": "Configuration id that changed.",
"type": [
"string",
"null"
]
}
},
"title": "OnConfigChangeEvent",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Ack returned by the internal `<worker>::on-config-change` handler.",
"properties": {
"ok": {
"type": "boolean"
}
},
"required": [
"ok"
],
"title": "OnConfigChangeResponse",
"type": "object"
}
},
{
"description": "Serve the judge-typesafe worker's injected console UI assets (content function for its console:script / console:style triggers).",
"metadata": {
"internal": true
},
"name": "judge-typesafe::ui-content",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input of the content function: the console asks for one asset by path.",
"properties": {
"path": {
"description": "The asset path from the trigger config (e.g. `state/page.js`).",
"type": "string"
}
},
"required": [
"path"
],
"title": "UiContentInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Output of the content function.",
"properties": {
"content": {
"description": "The asset source, verbatim.",
"type": "string"
},
"content_type": {
"description": "MIME type the console should serve the asset with.",
"type": "string"
}
},
"required": [
"content",
"content_type"
],
"title": "UiContentResult",
"type": "object"
}
}
],
"triggers": []
}