judge-typesafe
v0.2.3TypeSafe evaluation (judge-typesafe::evaluate), model listing (judge-typesafe::models::list) and caller-scoped cancellation (judge-typesafe::cancel), with typed usage, provider errors, per-call retries and worker-owned credentials.
Install
bash
iii trigger compose::add worker=judge-typesafe@0.2.3Run in your project with the iii engine running. Exact versions are immutable and digest-pinned.
Agent-ready brief for v0.2.3
Install, config, dependencies, readme, skills and API reference, all in one place. Fetch as agent-context.md for an LLM to consume.
Full markdown
The same content rendered as discrete blocks below is exposed as a single markdown document at
/workers/judge-typesafe.md?version=0.2.3. Paste it into an LLM prompt or pipe it through curl from a worker.Install
iii trigger compose::add worker=judge-typesafe@0.2.3Configuration
iii-config.yaml
- api_key: null
max_request_bytes: 8388608
max_response_bytes: 8388608
max_timeout_ms: 300000
model: jev-latestDependencies
Dependency · 1Version
Readme
README.md
judge-typesafe
TypeSafe JEV provider for the judge hub. It turns JSON state and
Noul, Choice or Score questions into typed, validated answers, sharing TypeSafe
credentials and HTTP capacity across callers, with model listing and
caller-owned cancellation. Callers go through judge::evaluate; the
judge-typesafe::* functions below are the same contract addressed directly.
They are registered as internal, so default discovery
(engine::functions::list, directory::search_functions) shows only the hub.
Install
iii trigger compose::add worker=judge-typesafeCompose starts the provider with its configuration dependency; installing
judge pulls it in automatically. Use engine
iii/v0.24.0-rc.2, the verified release with configuration::ensure used by
this repository's CI. Wait for JEV's functions to register before making a call.
Quickstart
In the Console, open Settings → Workers → Judge TypeSafe and set API key
(the form checks the saved key against judge-typesafe::models::list and offers
the answered catalog in the Default model select), preferably as a
secret://TYPESAFE_API_KEY reference to the secrets worker, or
supply TYPESAFE_API_KEY in the JEV service's environment before starting it.
See Configuration for precedence and reload behavior.
Ask whether a support ticket needs urgent attention:
iii trigger judge-typesafe::evaluate --timeout-ms 65000 --json '{
"timeout_ms": 60000,
"evaluations": [{
"id": "ticket-42",
"state": {"ticket": "Production checkout is down for all customers."},
"questions": {
"urgent": {
"type": "noul",
"instructions": "Does this describe an urgent production outage?",
"criteria": {"true": "A critical production workflow is blocked."}
}
}
}]
}'A successful response looks like this; scores, usage and timing are illustrative:
{
"status": "ok", "model": "jev-1.13.0",
"results": {"ticket-42": {
"answers": {"urgent": {"type": "noul", "noul": 0.98}},
"usage": {"input_tokens": 120, "output_tokens": 8}
}},
"stats": {"attempts": 1, "requests": 1, "questions": 1,
"input_tokens": 120, "output_tokens": 8, "elapsed_ms": 250,
"usage_complete": true}
}Check status before reading answers. A Noul value is between 0 and 1; choose
your own decision threshold. Errors return status: "error" and a code with no
partial results; a bus invocation failure is handled separately. The payload
budget covers the whole call, including queueing and retries; the longer CLI
timeout leaves room for transport overhead.
See the hub's mixed Noul/Choice/Score example,
result and error handling,
model listing and
cancellation. Through the hub, request_id
arrives here as judge_contract::MAX_PROVIDER_REQUEST_ID_BYTES).
Configuration
The Console entry defaults to judge-typesafe; set III_CONFIG_NAME=judge-typesafe-prod in the worker
environment to use judge-typesafe-prod. The form masks the API key and exposes these defaults:
api_key: null # Fall back to the worker's TYPESAFE_API_KEY; or secret://NAME.
model: jev-latest # Default unless a call supplies its own model.
max_request_bytes: 8388608 # Maximum JSON bytes per upstream evaluation.
max_response_bytes: 8388608 # Maximum bytes per upstream response.
max_timeout_ms: 300000 # Whole-call timeout ceiling in milliseconds.A nonblank configured key overrides the worker's environment key. Clearing it restores the fallback. All configuration fields hot-reload for new calls; in-flight calls keep their snapshot, and rejected reloads keep the last valid value. Changing the process environment requires a restart.
The configuration service persists values at ./config/
with its default filesystem adapter. Keep literal API keys out of committed
project configuration: store the key in the secrets worker, list
judge-typesafe in its consumers, and set api_key: secret://TYPESAFE_API_KEY
(NAME matches ^[A-Za-z_][A-Za-z0-9_.-]{0,127}$). The reference is resolved
through secrets::resolve at call time, never sent upstream as a key, and kept
only in memory. It is re-read when secrets::changed names it (bound at boot; the
engine holds the binding until a secrets worker registers the type), after
5 minutes, and 10 seconds after a failure. While the secrets worker is
unreachable, a key already resolved keeps being used. A reference that does not
resolve returns missing_key with provider_error.message naming the fix
(secret NAME not found in the secrets worker, judge-typesafe is not allowed to read secret NAME; add judge-typesafe to the secret's consumers, secrets worker is not running) and never falls back to TYPESAFE_API_KEY. The reference itself is
not secret; never paste a key after secret://. Any other scheme:// value, such
as env://TYPESAFE_API_KEY, is not sent as a literal key either: it returns
missing_key with api_key uses an unsupported env:// reference; use secret://NAME, which echoes only the scheme, and also never falls back.
The form retains unknown values when editing other fields and shows errors
returned by the configuration service; a ${TYPESAFE_API_KEY} value expands in
the configuration service's environment instead of this process.
Consumer compatibility
Callers use the shared judge-contract request and response types. See
shared-contract compatibility
when updating existing consumers. JEV supplies evaluations; each caller owns its
thresholds, ranking decisions and fallback policy.
For the full API, read the hub's reference.md. For source builds, UI work and local test commands, read CONTRIBUTING.md.
API reference (JSON)
agent-api-reference.json
{
"functions": [
{
"description": "Signal cancellation of an active evaluation or model listing owned by the calling worker. Returns whether a signal was accepted; does not roll back provider work. Requires the same worker replica as the original call.",
"metadata": {
"internal": true
},
"name": "judge-typesafe::cancel",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"request_id": {
"maxLength": 128,
"minLength": 1,
"type": "string"
}
},
"required": [
"request_id"
],
"title": "CancelRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ErrorCode": {
"oneOf": [
{
"enum": [
"invalid_request",
"missing_key",
"payload_too_large",
"deadline",
"attempt_timeout",
"cancelled",
"http",
"transport",
"invalid_response"
],
"type": "string"
},
{
"description": "The selected `judge-<provider>` worker is not registered on the engine, or (a local provider) its model failed to load.",
"enum": [
"provider_unavailable"
],
"type": "string"
}
]
}
},
"oneOf": [
{
"additionalProperties": false,
"properties": {
"cancelled": {
"type": "boolean"
},
"status": {
"enum": [
"ok"
],
"type": "string"
}
},
"required": [
"cancelled",
"status"
],
"type": "object"
},
{
"additionalProperties": false,
"properties": {
"code": {
"$ref": "#/definitions/ErrorCode"
},
"status": {
"enum": [
"error"
],
"type": "string"
}
},
"required": [
"code",
"status"
],
"type": "object"
}
],
"title": "CancelResponse"
}
},
{
"description": "Returns this worker instance's effective configuration entry ID, without its value.",
"metadata": {
"internal": true
},
"name": "judge-typesafe::configuration-id",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Empty user request. Ignore transport metadata such as `_caller_worker_id`, which the engine adds to routed calls; no payload field selects the entry.",
"title": "ConfigurationIdentityRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Only the entry ID is exposed; configuration values and secrets stay in the service.",
"properties": {
"id": {
"type": "string"
}
},
"required": [
"id"
],
"title": "ConfigurationIdentityResponse",
"type": "object"
}
},
{
"description": "Evaluate Noul, Choice and Score questions against arbitrary JSON state using JEV. Results are atomic; stats retain known accepted usage and mark unknown counters incomplete. No credentials are accepted in the request.",
"metadata": {
"internal": true
},
"name": "judge-typesafe::evaluate",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"Content": {
"anyOf": [
{
"type": "string"
},
{
"additionalProperties": true,
"type": "object"
},
{
"items": true,
"type": "array"
},
{
"type": "null"
}
],
"description": "Structured provider descriptions; top-level numbers/bools are not descriptions."
},
"Evaluation": {
"additionalProperties": false,
"properties": {
"id": {
"type": "string"
},
"questions": {
"additionalProperties": {
"$ref": "#/definitions/Question"
},
"minProperties": 1,
"type": "object"
},
"state": {
"$ref": "#/definitions/ScoreLevel"
}
},
"required": [
"id",
"questions",
"state"
],
"type": "object"
},
"Question": {
"oneOf": [
{
"additionalProperties": false,
"properties": {
"criteria": {
"anyOf": [
{
"additionalProperties": false,
"properties": {
"false": {
"$ref": "#/definitions/Content"
},
"true": {
"$ref": "#/definitions/Content"
}
},
"type": "object"
},
{
"type": "null"
}
]
},
"instructions": {
"allOf": [
{
"$ref": "#/definitions/Content"
}
],
"default": null
},
"type": {
"enum": [
"noul"
],
"type": "string"
}
},
"required": [
"type"
],
"type": "object"
},
{
"additionalProperties": false,
"properties": {
"criteria": {
"additionalProperties": {
"$ref": "#/definitions/Content"
},
"maxProperties": 255,
"minProperties": 1,
"type": "object"
},
"instructions": {
"allOf": [
{
"$ref": "#/definitions/Content"
}
],
"default": null
},
"type": {
"enum": [
"choice"
],
"type": "string"
}
},
"required": [
"criteria",
"type"
],
"type": "object"
},
{
"additionalProperties": false,
"properties": {
"criteria": {
"items": {
"$ref": "#/definitions/ScoreLevel"
},
"maxItems": 10,
"minItems": 2,
"type": "array"
},
"instructions": {
"allOf": [
{
"$ref": "#/definitions/Content"
}
],
"default": null
},
"type": {
"enum": [
"score"
],
"type": "string"
}
},
"required": [
"criteria",
"type"
],
"type": "object"
}
]
},
"RequestOptions": {
"additionalProperties": false,
"properties": {
"attempt_timeout_ms": {
"description": "Bounds each network attempt separately from the whole-call deadline.",
"format": "uint64",
"minimum": 1,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"ScoreLevel": {
"anyOf": [
{
"type": "string"
},
{
"additionalProperties": true,
"type": "object"
},
{
"items": true,
"type": "array"
}
],
"description": "Score levels require a description. Unlike instructions, Choice option descriptions and Noul criteria, the provider rejects a null level (HTTP422)."
}
},
"properties": {
"evaluations": {
"items": {
"$ref": "#/definitions/Evaluation"
},
"maxItems": 512,
"minItems": 1,
"type": "array"
},
"expires_at_unix_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"model": {
"type": [
"string",
"null"
]
},
"options": {
"allOf": [
{
"$ref": "#/definitions/RequestOptions"
}
],
"default": {}
},
"request_id": {
"maxLength": 128,
"minLength": 1,
"type": [
"string",
"null"
]
},
"timeout_ms": {
"format": "uint64",
"minimum": 1,
"type": "integer"
}
},
"required": [
"evaluations",
"timeout_ms"
],
"title": "EvaluateRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"Answer": {
"oneOf": [
{
"additionalProperties": false,
"properties": {
"noul": {
"format": "double",
"type": "number"
},
"type": {
"enum": [
"noul"
],
"type": "string"
}
},
"required": [
"noul",
"type"
],
"type": "object"
},
{
"additionalProperties": false,
"properties": {
"choice": {
"type": "string"
},
"confidence": {
"format": "double",
"type": "number"
},
"probabilities": {
"additionalProperties": {
"format": "double",
"type": "number"
},
"type": "object"
},
"type": {
"enum": [
"choice"
],
"type": "string"
}
},
"required": [
"choice",
"confidence",
"probabilities",
"type"
],
"type": "object"
},
{
"additionalProperties": false,
"properties": {
"confidence": {
"format": "double",
"type": "number"
},
"legend": {
"additionalProperties": {
"$ref": "#/definitions/ScoreLevel"
},
"type": "object"
},
"probabilities": {
"additionalProperties": {
"format": "double",
"type": "number"
},
"type": "object"
},
"score": {
"format": "double",
"type": "number"
},
"type": {
"enum": [
"score"
],
"type": "string"
}
},
"required": [
"confidence",
"legend",
"probabilities",
"score",
"type"
],
"type": "object"
}
]
},
"ErrorCode": {
"oneOf": [
{
"enum": [
"invalid_request",
"missing_key",
"payload_too_large",
"deadline",
"attempt_timeout",
"cancelled",
"http",
"transport",
"invalid_response"
],
"type": "string"
},
{
"description": "The selected `judge-<provider>` worker is not registered on the engine, or (a local provider) its model failed to load.",
"enum": [
"provider_unavailable"
],
"type": "string"
}
]
},
"EvaluationResult": {
"additionalProperties": false,
"properties": {
"answers": {
"additionalProperties": {
"$ref": "#/definitions/Answer"
},
"type": "object"
},
"usage": {
"anyOf": [
{
"$ref": "#/definitions/Usage"
},
{
"type": "null"
}
]
}
},
"required": [
"answers"
],
"type": "object"
},
"ProviderError": {
"additionalProperties": false,
"description": "Bounded provider diagnostics, with credentials and supplied header values redacted.",
"properties": {
"detail": true,
"message": {
"type": [
"string",
"null"
]
},
"truncated": {
"default": false,
"type": "boolean"
}
},
"type": "object"
},
"ScoreLevel": {
"anyOf": [
{
"type": "string"
},
{
"additionalProperties": true,
"type": "object"
},
{
"items": true,
"type": "array"
}
],
"description": "Score levels require a description. Unlike instructions, Choice option descriptions and Noul criteria, the provider rejects a null level (HTTP422)."
},
"Stats": {
"additionalProperties": false,
"description": "Known accepted usage, never a billing estimate. Unreported counters or failure/cancellation make usage_complete false.",
"properties": {
"attempts": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"elapsed_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"questions": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"requests": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"usage_complete": {
"type": "boolean"
}
},
"required": [
"attempts",
"elapsed_ms",
"input_tokens",
"output_tokens",
"questions",
"requests",
"usage_complete"
],
"type": "object"
},
"Usage": {
"additionalProperties": false,
"description": "Each absent/null counter is unknown, independently of the other counter.",
"properties": {
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
}
},
"oneOf": [
{
"additionalProperties": false,
"properties": {
"model": {
"type": "string"
},
"results": {
"additionalProperties": {
"$ref": "#/definitions/EvaluationResult"
},
"type": "object"
},
"stats": {
"$ref": "#/definitions/Stats"
},
"status": {
"enum": [
"ok"
],
"type": "string"
}
},
"required": [
"model",
"results",
"stats",
"status"
],
"type": "object"
},
{
"additionalProperties": false,
"properties": {
"code": {
"$ref": "#/definitions/ErrorCode"
},
"http_status": {
"format": "uint16",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"provider_error": {
"anyOf": [
{
"$ref": "#/definitions/ProviderError"
},
{
"type": "null"
}
]
},
"retry_after_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stats": {
"$ref": "#/definitions/Stats"
},
"status": {
"enum": [
"error"
],
"type": "string"
}
},
"required": [
"code",
"stats",
"status"
],
"type": "object"
}
],
"title": "EvaluateResponse"
}
},
{
"description": "List the provider's available model names, descriptions and release dates. Shares evaluation credentials, transport limits and permits; performs no inference.",
"metadata": {
"internal": true
},
"name": "judge-typesafe::models::list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"RequestOptions": {
"additionalProperties": false,
"properties": {
"attempt_timeout_ms": {
"description": "Bounds each network attempt separately from the whole-call deadline.",
"format": "uint64",
"minimum": 1,
"type": [
"integer",
"null"
]
}
},
"type": "object"
}
},
"properties": {
"expires_at_unix_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"options": {
"allOf": [
{
"$ref": "#/definitions/RequestOptions"
}
],
"default": {}
},
"request_id": {
"maxLength": 128,
"minLength": 1,
"type": [
"string",
"null"
]
},
"timeout_ms": {
"default": 30000,
"format": "uint64",
"minimum": 1,
"type": "integer"
}
},
"title": "ModelsRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ErrorCode": {
"oneOf": [
{
"enum": [
"invalid_request",
"missing_key",
"payload_too_large",
"deadline",
"attempt_timeout",
"cancelled",
"http",
"transport",
"invalid_response"
],
"type": "string"
},
{
"description": "The selected `judge-<provider>` worker is not registered on the engine, or (a local provider) its model failed to load.",
"enum": [
"provider_unavailable"
],
"type": "string"
}
]
},
"ModelCard": {
"properties": {
"context_window": {
"description": "Tokens one evaluation row can hold (question, options and state), when the provider has a fixed window. Callers size their states from it.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"description": {
"type": "string"
},
"max_options": {
"description": "Most options one Choice question (or levels one Score) can offer, when the provider has a fixed limit below the contract's 255. Callers size their option groups from it.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"name": {
"type": "string"
},
"release_date": {
"type": "string"
}
},
"required": [
"description",
"name",
"release_date"
],
"type": "object"
},
"ProviderError": {
"additionalProperties": false,
"description": "Bounded provider diagnostics, with credentials and supplied header values redacted.",
"properties": {
"detail": true,
"message": {
"type": [
"string",
"null"
]
},
"truncated": {
"default": false,
"type": "boolean"
}
},
"type": "object"
},
"Stats": {
"additionalProperties": false,
"description": "Known accepted usage, never a billing estimate. Unreported counters or failure/cancellation make usage_complete false.",
"properties": {
"attempts": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"elapsed_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"questions": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"requests": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"usage_complete": {
"type": "boolean"
}
},
"required": [
"attempts",
"elapsed_ms",
"input_tokens",
"output_tokens",
"questions",
"requests",
"usage_complete"
],
"type": "object"
}
},
"oneOf": [
{
"additionalProperties": false,
"properties": {
"models": {
"items": {
"$ref": "#/definitions/ModelCard"
},
"type": "array"
},
"stats": {
"$ref": "#/definitions/Stats"
},
"status": {
"enum": [
"ok"
],
"type": "string"
}
},
"required": [
"models",
"stats",
"status"
],
"type": "object"
},
{
"additionalProperties": false,
"properties": {
"code": {
"$ref": "#/definitions/ErrorCode"
},
"http_status": {
"format": "uint16",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"provider_error": {
"anyOf": [
{
"$ref": "#/definitions/ProviderError"
},
{
"type": "null"
}
]
},
"retry_after_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stats": {
"$ref": "#/definitions/Stats"
},
"status": {
"enum": [
"error"
],
"type": "string"
}
},
"required": [
"code",
"stats",
"status"
],
"type": "object"
}
],
"title": "ModelsResponse"
}
},
{
"description": "Internal: reload JEV configuration from authoritative storage.",
"metadata": {
"internal": true
},
"name": "judge-typesafe::on-config-change",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Trigger payload for `<worker>::on-config-change`. Advisory only: handlers re-fetch the authoritative value and ignore it, so a direct call can never inject config.",
"properties": {
"id": {
"default": null,
"description": "Configuration id that changed.",
"type": [
"string",
"null"
]
}
},
"title": "OnConfigChangeEvent",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Ack returned by the internal `<worker>::on-config-change` handler.",
"properties": {
"ok": {
"type": "boolean"
}
},
"required": [
"ok"
],
"title": "OnConfigChangeResponse",
"type": "object"
}
},
{
"description": "Internal: drop a cached secret:// API key after the secrets worker changed it.",
"metadata": {
"internal": true
},
"name": "judge-typesafe::on-secret-change",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Advisory `secrets::changed` payload; it never carries a value. Other fields (`action`, `fingerprint`, `updated_at`) are ignored.",
"properties": {
"name": {
"default": null,
"type": [
"string",
"null"
]
},
"ref": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "SecretChangedEvent",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"ok": {
"type": "boolean"
}
},
"required": [
"ok"
],
"title": "SecretChangedResponse",
"type": "object"
}
},
{
"description": "Serve the judge-typesafe worker's injected console UI assets (content function for its console:script / console:style / console:module triggers).",
"metadata": {
"internal": true
},
"name": "judge-typesafe::ui-content",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input of the content function: the console asks for one asset by path.",
"properties": {
"path": {
"description": "The asset path from the trigger config (e.g. `state/page.js`).",
"type": "string"
}
},
"required": [
"path"
],
"title": "UiContentInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Output of the content function.",
"properties": {
"content": {
"description": "The asset source, verbatim.",
"type": "string"
},
"content_type": {
"description": "MIME type the console should serve the asset with.",
"type": "string"
}
},
"required": [
"content",
"content_type"
],
"title": "UiContentResult",
"type": "object"
}
}
],
"triggers": []
}