# judge-typesafe > TypeSafe evaluation (judge-typesafe::evaluate), model listing (judge-typesafe::models::list) and caller-scoped cancellation (judge-typesafe::cancel), with typed usage, provider errors, per-call retries and worker-owned credentials. | field | value | |-------|-------| | version | 0.2.2 | | type | binary | | license | Apache-2.0 | | repo | https://github.com/iii-hq/workers | | supported_targets | x86_64-apple-darwin, aarch64-apple-darwin, x86_64-pc-windows-msvc, aarch64-pc-windows-msvc, x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu, x86_64-unknown-linux-musl, armv7-unknown-linux-gnueabihf | | author | iii | ## installation ```sh iii trigger compose::add worker=judge-typesafe@0.2.2 ``` ## configuration ```yaml - api_key: null max_request_bytes: 8388608 max_response_bytes: 8388608 max_timeout_ms: 300000 model: jev-latest ``` ## dependencies - `configuration` @ `latest` ## readme # judge-typesafe TypeSafe JEV provider for the [`judge`](../judge/) hub. It turns JSON state and Noul, Choice or Score questions into typed, validated answers, sharing TypeSafe credentials and HTTP capacity across callers, with model listing and caller-owned cancellation. Callers go through `judge::evaluate`; the `judge-typesafe::*` functions below are the same contract addressed directly. They are registered as internal, so default discovery (`engine::functions::list`, `directory::search_functions`) shows only the hub. ## Install ```bash iii trigger compose::add worker=judge-typesafe ``` Compose starts the provider with its `configuration` dependency; installing `judge` pulls it in automatically. Use engine **`iii/v0.24.0-rc.2`**, the verified release with `configuration::ensure` used by this repository's CI. Wait for JEV's functions to register before making a call. ## Quickstart In the Console, open **Settings → Workers → Judge TypeSafe** and set **API key** (the form checks the saved key against `judge-typesafe::models::list` and offers the answered catalog in the **Default model** select), preferably as a `secret://TYPESAFE_API_KEY` reference to the [`secrets`](../secrets/) worker, or supply `TYPESAFE_API_KEY` in the JEV service's environment before starting it. See [Configuration](#configuration) for precedence and reload behavior. Ask whether a support ticket needs urgent attention: ```bash iii trigger judge-typesafe::evaluate --timeout-ms 65000 --json '{ "timeout_ms": 60000, "evaluations": [{ "id": "ticket-42", "state": {"ticket": "Production checkout is down for all customers."}, "questions": { "urgent": { "type": "noul", "instructions": "Does this describe an urgent production outage?", "criteria": {"true": "A critical production workflow is blocked."} } } }] }' ``` A successful response looks like this; scores, usage and timing are illustrative: ```json { "status": "ok", "model": "jev-1.13.0", "results": {"ticket-42": { "answers": {"urgent": {"type": "noul", "noul": 0.98}}, "usage": {"input_tokens": 120, "output_tokens": 8} }}, "stats": {"attempts": 1, "requests": 1, "questions": 1, "input_tokens": 120, "output_tokens": 8, "elapsed_ms": 250, "usage_complete": true} } ``` Check `status` before reading answers. A Noul value is between 0 and 1; choose your own decision threshold. Errors return `status: "error"` and a `code` with no partial results; a bus invocation failure is handled separately. The payload budget covers the whole call, including queueing and retries; the longer CLI timeout leaves room for transport overhead. See the hub's [mixed Noul/Choice/Score example](../judge/reference.md#evaluate), [result and error handling](../judge/reference.md#handle-results-and-failures), [model listing](../judge/reference.md#list-models) and [cancellation](../judge/reference.md#cancellation). Through the hub, `request_id` arrives here as `:/`, which is why this worker accepts ids of up to 512 bytes (`judge_contract::MAX_PROVIDER_REQUEST_ID_BYTES`). ## Configuration The Console entry defaults to `judge-typesafe`; set `III_CONFIG_NAME=judge-typesafe-prod` in the worker environment to use `judge-typesafe-prod`. The form masks the API key and exposes these defaults: ```yaml api_key: null # Fall back to the worker's TYPESAFE_API_KEY; or secret://NAME. model: jev-latest # Default unless a call supplies its own model. max_request_bytes: 8388608 # Maximum JSON bytes per upstream evaluation. max_response_bytes: 8388608 # Maximum bytes per upstream response. max_timeout_ms: 300000 # Whole-call timeout ceiling in milliseconds. ``` A nonblank configured key overrides the worker's environment key. Clearing it restores the fallback. All configuration fields hot-reload for new calls; in-flight calls keep their snapshot, and rejected reloads keep the last valid value. Changing the process environment requires a restart. The `configuration` service persists values at `./config/.yaml` with its default filesystem adapter. Keep literal API keys out of committed project configuration: store the key in the `secrets` worker, list `judge-typesafe` in its `consumers`, and set `api_key: secret://TYPESAFE_API_KEY` (`NAME` matches `^[A-Za-z_][A-Za-z0-9_.-]{0,127}$`). The reference is resolved through `secrets::resolve` at call time, never sent upstream as a key, and kept only in memory. It is re-read when `secrets::changed` names it (bound at boot; the engine holds the binding until a secrets worker registers the type), after 5 minutes, and 10 seconds after a failure. While the secrets worker is unreachable, a key already resolved keeps being used. A reference that does not resolve returns `missing_key` with `provider_error.message` naming the fix (`secret NAME not found in the secrets worker`, `judge-typesafe is not allowed to read secret NAME; add judge-typesafe to the secret's consumers`, `secrets worker is not running`) and never falls back to `TYPESAFE_API_KEY`. The reference itself is not secret; never paste a key after `secret://`. The form retains unknown values when editing other fields and shows errors returned by the configuration service; a `${TYPESAFE_API_KEY}` value expands in the configuration service's environment instead of this process. ## Consumer compatibility Callers use the shared `judge-contract` request and response types. See [shared-contract compatibility](../judge/reference.md#limits-and-compatibility) when updating existing consumers. JEV supplies evaluations; each caller owns its thresholds, ranking decisions and fallback policy. For the full API, read the hub's [reference.md](../judge/reference.md). For source builds, UI work and local test commands, read [CONTRIBUTING.md](CONTRIBUTING.md). ## api reference ```json { "functions": [ { "description": "Signal cancellation of an active evaluation or model listing owned by the calling worker. Returns whether a signal was accepted; does not roll back provider work. Requires the same worker replica as the original call.", "metadata": { "internal": true }, "name": "judge-typesafe::cancel", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": false, "properties": { "request_id": { "maxLength": 128, "minLength": 1, "type": "string" } }, "required": [ "request_id" ], "title": "CancelRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "ErrorCode": { "oneOf": [ { "enum": [ "invalid_request", "missing_key", "payload_too_large", "deadline", "attempt_timeout", "cancelled", "http", "transport", "invalid_response" ], "type": "string" }, { "description": "The selected `judge-` worker is not registered on the engine, or (a local provider) its model failed to load.", "enum": [ "provider_unavailable" ], "type": "string" } ] } }, "oneOf": [ { "additionalProperties": false, "properties": { "cancelled": { "type": "boolean" }, "status": { "enum": [ "ok" ], "type": "string" } }, "required": [ "cancelled", "status" ], "type": "object" }, { "additionalProperties": false, "properties": { "code": { "$ref": "#/definitions/ErrorCode" }, "status": { "enum": [ "error" ], "type": "string" } }, "required": [ "code", "status" ], "type": "object" } ], "title": "CancelResponse" } }, { "description": "Returns this worker instance's effective configuration entry ID, without its value.", "metadata": { "internal": true }, "name": "judge-typesafe::configuration-id", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "description": "Empty user request. Ignore transport metadata such as `_caller_worker_id`, which the engine adds to routed calls; no payload field selects the entry.", "title": "ConfigurationIdentityRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "description": "Only the entry ID is exposed; configuration values and secrets stay in the service.", "properties": { "id": { "type": "string" } }, "required": [ "id" ], "title": "ConfigurationIdentityResponse", "type": "object" } }, { "description": "Evaluate Noul, Choice and Score questions against arbitrary JSON state using JEV. Results are atomic; stats retain known accepted usage and mark unknown counters incomplete. No credentials are accepted in the request.", "metadata": { "internal": true }, "name": "judge-typesafe::evaluate", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": false, "definitions": { "Content": { "anyOf": [ { "type": "string" }, { "additionalProperties": true, "type": "object" }, { "items": true, "type": "array" }, { "type": "null" } ], "description": "Structured provider descriptions; top-level numbers/bools are not descriptions." }, "Evaluation": { "additionalProperties": false, "properties": { "id": { "type": "string" }, "questions": { "additionalProperties": { "$ref": "#/definitions/Question" }, "minProperties": 1, "type": "object" }, "state": { "$ref": "#/definitions/ScoreLevel" } }, "required": [ "id", "questions", "state" ], "type": "object" }, "Question": { "oneOf": [ { "additionalProperties": false, "properties": { "criteria": { "anyOf": [ { "additionalProperties": false, "properties": { "false": { "$ref": "#/definitions/Content" }, "true": { "$ref": "#/definitions/Content" } }, "type": "object" }, { "type": "null" } ] }, "instructions": { "allOf": [ { "$ref": "#/definitions/Content" } ], "default": null }, "type": { "enum": [ "noul" ], "type": "string" } }, "required": [ "type" ], "type": "object" }, { "additionalProperties": false, "properties": { "criteria": { "additionalProperties": { "$ref": "#/definitions/Content" }, "maxProperties": 255, "minProperties": 1, "type": "object" }, "instructions": { "allOf": [ { "$ref": "#/definitions/Content" } ], "default": null }, "type": { "enum": [ "choice" ], "type": "string" } }, "required": [ "criteria", "type" ], "type": "object" }, { "additionalProperties": false, "properties": { "criteria": { "items": { "$ref": "#/definitions/ScoreLevel" }, "maxItems": 10, "minItems": 2, "type": "array" }, "instructions": { "allOf": [ { "$ref": "#/definitions/Content" } ], "default": null }, "type": { "enum": [ "score" ], "type": "string" } }, "required": [ "criteria", "type" ], "type": "object" } ] }, "RequestOptions": { "additionalProperties": false, "properties": { "attempt_timeout_ms": { "description": "Bounds each network attempt separately from the whole-call deadline.", "format": "uint64", "minimum": 1, "type": [ "integer", "null" ] } }, "type": "object" }, "ScoreLevel": { "anyOf": [ { "type": "string" }, { "additionalProperties": true, "type": "object" }, { "items": true, "type": "array" } ], "description": "Score levels require a description. Unlike instructions, Choice option descriptions and Noul criteria, the provider rejects a null level (HTTP422)." } }, "properties": { "evaluations": { "items": { "$ref": "#/definitions/Evaluation" }, "maxItems": 512, "minItems": 1, "type": "array" }, "expires_at_unix_ms": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "model": { "type": [ "string", "null" ] }, "options": { "allOf": [ { "$ref": "#/definitions/RequestOptions" } ], "default": {} }, "request_id": { "maxLength": 128, "minLength": 1, "type": [ "string", "null" ] }, "timeout_ms": { "format": "uint64", "minimum": 1, "type": "integer" } }, "required": [ "evaluations", "timeout_ms" ], "title": "EvaluateRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "Answer": { "oneOf": [ { "additionalProperties": false, "properties": { "noul": { "format": "double", "type": "number" }, "type": { "enum": [ "noul" ], "type": "string" } }, "required": [ "noul", "type" ], "type": "object" }, { "additionalProperties": false, "properties": { "choice": { "type": "string" }, "confidence": { "format": "double", "type": "number" }, "probabilities": { "additionalProperties": { "format": "double", "type": "number" }, "type": "object" }, "type": { "enum": [ "choice" ], "type": "string" } }, "required": [ "choice", "confidence", "probabilities", "type" ], "type": "object" }, { "additionalProperties": false, "properties": { "confidence": { "format": "double", "type": "number" }, "legend": { "additionalProperties": { "$ref": "#/definitions/ScoreLevel" }, "type": "object" }, "probabilities": { "additionalProperties": { "format": "double", "type": "number" }, "type": "object" }, "score": { "format": "double", "type": "number" }, "type": { "enum": [ "score" ], "type": "string" } }, "required": [ "confidence", "legend", "probabilities", "score", "type" ], "type": "object" } ] }, "ErrorCode": { "oneOf": [ { "enum": [ "invalid_request", "missing_key", "payload_too_large", "deadline", "attempt_timeout", "cancelled", "http", "transport", "invalid_response" ], "type": "string" }, { "description": "The selected `judge-` worker is not registered on the engine, or (a local provider) its model failed to load.", "enum": [ "provider_unavailable" ], "type": "string" } ] }, "EvaluationResult": { "additionalProperties": false, "properties": { "answers": { "additionalProperties": { "$ref": "#/definitions/Answer" }, "type": "object" }, "usage": { "anyOf": [ { "$ref": "#/definitions/Usage" }, { "type": "null" } ] } }, "required": [ "answers" ], "type": "object" }, "ProviderError": { "additionalProperties": false, "description": "Bounded provider diagnostics, with credentials and supplied header values redacted.", "properties": { "detail": true, "message": { "type": [ "string", "null" ] }, "truncated": { "default": false, "type": "boolean" } }, "type": "object" }, "ScoreLevel": { "anyOf": [ { "type": "string" }, { "additionalProperties": true, "type": "object" }, { "items": true, "type": "array" } ], "description": "Score levels require a description. Unlike instructions, Choice option descriptions and Noul criteria, the provider rejects a null level (HTTP422)." }, "Stats": { "additionalProperties": false, "description": "Known accepted usage, never a billing estimate. Unreported counters or failure/cancellation make usage_complete false.", "properties": { "attempts": { "format": "uint", "minimum": 0, "type": "integer" }, "elapsed_ms": { "format": "uint64", "minimum": 0, "type": "integer" }, "input_tokens": { "format": "uint64", "minimum": 0, "type": "integer" }, "output_tokens": { "format": "uint64", "minimum": 0, "type": "integer" }, "questions": { "format": "uint", "minimum": 0, "type": "integer" }, "requests": { "format": "uint", "minimum": 0, "type": "integer" }, "usage_complete": { "type": "boolean" } }, "required": [ "attempts", "elapsed_ms", "input_tokens", "output_tokens", "questions", "requests", "usage_complete" ], "type": "object" }, "Usage": { "additionalProperties": false, "description": "Each absent/null counter is unknown, independently of the other counter.", "properties": { "input_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "output_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] } }, "type": "object" } }, "oneOf": [ { "additionalProperties": false, "properties": { "model": { "type": "string" }, "results": { "additionalProperties": { "$ref": "#/definitions/EvaluationResult" }, "type": "object" }, "stats": { "$ref": "#/definitions/Stats" }, "status": { "enum": [ "ok" ], "type": "string" } }, "required": [ "model", "results", "stats", "status" ], "type": "object" }, { "additionalProperties": false, "properties": { "code": { "$ref": "#/definitions/ErrorCode" }, "http_status": { "format": "uint16", "minimum": 0, "type": [ "integer", "null" ] }, "provider_error": { "anyOf": [ { "$ref": "#/definitions/ProviderError" }, { "type": "null" } ] }, "retry_after_ms": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "stats": { "$ref": "#/definitions/Stats" }, "status": { "enum": [ "error" ], "type": "string" } }, "required": [ "code", "stats", "status" ], "type": "object" } ], "title": "EvaluateResponse" } }, { "description": "List the provider's available model names, descriptions and release dates. Shares evaluation credentials, transport limits and permits; performs no inference.", "metadata": { "internal": true }, "name": "judge-typesafe::models::list", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": false, "definitions": { "RequestOptions": { "additionalProperties": false, "properties": { "attempt_timeout_ms": { "description": "Bounds each network attempt separately from the whole-call deadline.", "format": "uint64", "minimum": 1, "type": [ "integer", "null" ] } }, "type": "object" } }, "properties": { "expires_at_unix_ms": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "options": { "allOf": [ { "$ref": "#/definitions/RequestOptions" } ], "default": {} }, "request_id": { "maxLength": 128, "minLength": 1, "type": [ "string", "null" ] }, "timeout_ms": { "default": 30000, "format": "uint64", "minimum": 1, "type": "integer" } }, "title": "ModelsRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "ErrorCode": { "oneOf": [ { "enum": [ "invalid_request", "missing_key", "payload_too_large", "deadline", "attempt_timeout", "cancelled", "http", "transport", "invalid_response" ], "type": "string" }, { "description": "The selected `judge-` worker is not registered on the engine, or (a local provider) its model failed to load.", "enum": [ "provider_unavailable" ], "type": "string" } ] }, "ModelCard": { "properties": { "context_window": { "description": "Tokens one evaluation row can hold (question, options and state), when the provider has a fixed window. Callers size their states from it.", "format": "uint32", "minimum": 0, "type": [ "integer", "null" ] }, "description": { "type": "string" }, "max_options": { "description": "Most options one Choice question (or levels one Score) can offer, when the provider has a fixed limit below the contract's 255. Callers size their option groups from it.", "format": "uint32", "minimum": 0, "type": [ "integer", "null" ] }, "name": { "type": "string" }, "release_date": { "type": "string" } }, "required": [ "description", "name", "release_date" ], "type": "object" }, "ProviderError": { "additionalProperties": false, "description": "Bounded provider diagnostics, with credentials and supplied header values redacted.", "properties": { "detail": true, "message": { "type": [ "string", "null" ] }, "truncated": { "default": false, "type": "boolean" } }, "type": "object" }, "Stats": { "additionalProperties": false, "description": "Known accepted usage, never a billing estimate. Unreported counters or failure/cancellation make usage_complete false.", "properties": { "attempts": { "format": "uint", "minimum": 0, "type": "integer" }, "elapsed_ms": { "format": "uint64", "minimum": 0, "type": "integer" }, "input_tokens": { "format": "uint64", "minimum": 0, "type": "integer" }, "output_tokens": { "format": "uint64", "minimum": 0, "type": "integer" }, "questions": { "format": "uint", "minimum": 0, "type": "integer" }, "requests": { "format": "uint", "minimum": 0, "type": "integer" }, "usage_complete": { "type": "boolean" } }, "required": [ "attempts", "elapsed_ms", "input_tokens", "output_tokens", "questions", "requests", "usage_complete" ], "type": "object" } }, "oneOf": [ { "additionalProperties": false, "properties": { "models": { "items": { "$ref": "#/definitions/ModelCard" }, "type": "array" }, "stats": { "$ref": "#/definitions/Stats" }, "status": { "enum": [ "ok" ], "type": "string" } }, "required": [ "models", "stats", "status" ], "type": "object" }, { "additionalProperties": false, "properties": { "code": { "$ref": "#/definitions/ErrorCode" }, "http_status": { "format": "uint16", "minimum": 0, "type": [ "integer", "null" ] }, "provider_error": { "anyOf": [ { "$ref": "#/definitions/ProviderError" }, { "type": "null" } ] }, "retry_after_ms": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "stats": { "$ref": "#/definitions/Stats" }, "status": { "enum": [ "error" ], "type": "string" } }, "required": [ "code", "stats", "status" ], "type": "object" } ], "title": "ModelsResponse" } }, { "description": "Internal: reload JEV configuration from authoritative storage.", "metadata": { "internal": true }, "name": "judge-typesafe::on-config-change", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "description": "Trigger payload for `::on-config-change`. Advisory only: handlers re-fetch the authoritative value and ignore it, so a direct call can never inject config.", "properties": { "id": { "default": null, "description": "Configuration id that changed.", "type": [ "string", "null" ] } }, "title": "OnConfigChangeEvent", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "description": "Ack returned by the internal `::on-config-change` handler.", "properties": { "ok": { "type": "boolean" } }, "required": [ "ok" ], "title": "OnConfigChangeResponse", "type": "object" } }, { "description": "Internal: drop a cached secret:// API key after the secrets worker changed it.", "metadata": { "internal": true }, "name": "judge-typesafe::on-secret-change", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "description": "Advisory `secrets::changed` payload; it never carries a value. Other fields (`action`, `fingerprint`, `updated_at`) are ignored.", "properties": { "name": { "default": null, "type": [ "string", "null" ] }, "ref": { "default": null, "type": [ "string", "null" ] } }, "title": "SecretChangedEvent", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "ok": { "type": "boolean" } }, "required": [ "ok" ], "title": "SecretChangedResponse", "type": "object" } }, { "description": "Serve the judge-typesafe worker's injected console UI assets (content function for its console:script / console:style / console:module triggers).", "metadata": { "internal": true }, "name": "judge-typesafe::ui-content", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "description": "Input of the content function: the console asks for one asset by path.", "properties": { "path": { "description": "The asset path from the trigger config (e.g. `state/page.js`).", "type": "string" } }, "required": [ "path" ], "title": "UiContentInput", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "description": "Output of the content function.", "properties": { "content": { "description": "The asset source, verbatim.", "type": "string" }, "content_type": { "description": "MIME type the console should serve the asset with.", "type": "string" } }, "required": [ "content", "content_type" ], "title": "UiContentResult", "type": "object" } } ], "triggers": [] } ```