llm-router

One front door for every LLM provider. The router owns routing, the provider registry, credential resolution, the model catalog, streaming relay, retries, and a single failure contract — consumers call one chat surface and never talk to a provider directly.

llm-router is a standalone iii worker. Providers plug in as separate workers at runtime through a self-registration protocol (iii worker add provider-); the router never compiles against a provider, and removing a provider worker removes the provider.

Install

iii worker add llm-router

Quickstart

A consumer streams a turn by creating an iii channel, handing the router the channel's write endpoint, and reading frames from the read endpoint while router::chat runs. Any SDK works; Node shown:

import { createChannel } from 'iii-sdk';

const { reader, writerRef } = await createChannel(iii);
reader.onMessage((frame) => {
  const event = JSON.parse(frame); // AssistantMessageEvent
  if (event.type === 'text_delta') process.stdout.write(event.delta);
});

const res = await iii.trigger('router::chat', {
  writer_ref: writerRef, // direction "write"
  model: 'claude-sonnet-4',
  messages: [{ role: 'user', content: [{ type: 'text', text: 'Hello' }], timestamp: Date.now() }],
}, { timeout_ms: 320_000 }); // outer timeout ≥ the router's 300s stream budget
// res: { ok, provider, model, stop_reason, usage }

The streaming contract: every stream ends with exactly one terminal frame (done or error). When the router has to kill a stream itself (idle timeout, provider crash), it synthesizes the terminal frame and attaches the partial content, so consumers never hang on a half-open stream.

Functions

Consumer surface

Function	Purpose
router::chat	Stream a turn into the caller's channel; returns the turn summary.
router::complete	Non-streaming convenience over the same pipeline; returns the final message.
router::abort	Cancel an in-flight turn by request_id.
router::route	Read-only routing preview: {model, provider?} → {provider, candidates}, same rules and error codes as router::chat. Pin the result as the explicit provider on the chat call when you need the provider before streaming.
router::models::list	List catalog models, filterable by provider / capability.
router::models::get	Fetch one model record (null when unknown).
router::models::supports	Check one capability flag for one model.
router::provider::list	Registered providers with configured / available status.
router::system_prompt::get	Effective identity prompt for {provider?} (the configured default_provider when omitted): operator override when set → provider-declared → null. null also when the provider is unknown — callers fall back to their own default prompt.

Only the read surface is agent-callable (router::models::list / get / supports, router::provider::list); everything else is denied to in-run agents — see Security model.

Provider protocol

Token-gated after the first declare: the response to register carries a registration token, and every later protocol call must present it.

Function	Purpose
router::provider::register	Self-declaration at attach time; idempotent re-declare with the token.
router::provider::resolve	Per-request credential + endpoint resolution (config > env > none).
router::provider::update_credential	Persist a refreshed credential (OAuth write-back).
router::models::reconcile	Replace the provider's catalog slice in one write.

The provider worker itself exposes provider::::stream and, when it supports model discovery, provider::::refresh_models.

Configuration

All operator configuration lives in the engine's llm-router configuration entry — no env vars, no config file. The entry schema is composed at runtime from each registered provider's declaration:

{
  "default_provider": "anthropic",
  "providers": {
    "anthropic": {
      "api_key": "sk-…",
      "api_url": "https://api.anthropic.com/v1/messages",
      "max_tokens": 8192,
      "system_prompt": "optional override of the provider-declared identity prompt (unset = provider default)"
    }
  },
  "routing_heuristics": [{ "pattern": "^gpt-", "provider": "openai" }],
  "settings": {
    "stream_timeout_ms": 300000,
    "idle_timeout_ms": 120000,
    "retry_max": 2,
    "output_token_max": 32000
  }
}

Setting	Default	Meaning
stream_timeout_ms	300000	Hard budget for one streamed turn.
idle_timeout_ms	120000	Max silence between provider frames before the attempt is cut.
retry_max	2	Retries per turn for retryable failures before the first forwarded frame.
output_token_max	32000	Ceiling on max_output_tokens forwarded to providers.

Per-provider slices additionally accept a nullable system_prompt: when set (non-empty), it overrides the provider-declared identity prompt served by router::system_prompt::get; unset/null serves the provider's default. Read live per request — no restart, no debounce.

Pasting a key into a provider's slice is the whole onboarding flow: the router diffs the changed slice, debounces ~2 s, and kicks that provider's provider::::refresh_models discovery; discovered models land in the catalog via router::models::reconcile and show up in router::models::list within seconds — no restart.

Operational notes

• Env-var credential fallback resolves in the router's process. A provider's credential_env_var (e.g. ANTHROPIC_API_KEY) is read by the llm-router binary, not by the provider worker — launch the router with those variables set, or put keys in the entry. A key present only in another worker's environment shows up as configured: false.
• Registration-token recovery. Re-registering a provider id without its original token is rejected (anti-takeover). If a provider durably lost its token, delete the router's registry state (iii-state scope llm-router, key registry) and restart the affected providers to re-bind; pasted credentials in the configuration entry are unaffected.

Security model

The agent-callable surface is governed by the engine-wide permission policy (repo-root iii-permissions.yaml), not a per-worker file. In-run agents may only read the catalog and provider list:

• Allowed: router::models::list, router::models::get, router::models::supports, router::provider::list.
• Denied to agents: the chat/spend surface (router::chat, router::complete, router::abort, router::route), the whole provider/credential protocol (router::provider::resolve / register / update_credential, router::models::reconcile), and direct provider::* calls.

Worker-to-worker calls bypass the agent gate, so the harness, context-manager, and provider workers reach the full surface — only in-run agents are restricted. Provider credentials live in the configuration entry and are never readable back through any allowed function.

Events

The router registers three custom trigger types and fans out to every bound handler. Bind with the standard two-step pattern; the handler receives the payload verbatim (no envelope).

Trigger type	Fires when	Payload
router::models::changed	a provider reconciles its catalog slice	{ "provider": "", "count": }
router::provider::changed	the registry changes (declare / availability flip)	{ "provider": "", "op": "register" | "available" | "unavailable" }
router::ready	the router finishes booting; providers re-declare on it	{}

iii.registerFunction('my-worker::on-models-changed', async (payload) => {
  console.log('catalog changed:', payload); // { provider, count }
  return {};
});

iii.registerTrigger({
  type: 'router::models::changed',
  function_id: 'my-worker::on-models-changed',
  config: {},
});

Writing a provider worker

A provider worker must:

1. Register provider::::stream honouring the channel-writer contract: forward upstream output as AssistantMessageEvent frames into the writer_ref it receives, ending with one terminal frame.
2. Declare itself at startup via router::provider::register — retrying with backoff until acknowledged (covers provider-before-router boot order) — and re-declare on the router::ready event after a router restart.
3. Resolve credentials per request via router::provider::resolve; never read keys directly.
4. Treat closure of its stream channel as cancellation: abort the upstream request and stop writing frames.
5. Map upstream failures to the shared ErrorKind taxonomy on its error frames. Transport retries (429 / 5xx / connect) are the router's job, not the provider's.

Optionally, the declaration may carry a system_prompt — the identity prompt tailored to the provider's model family. The router stores it in the registry and serves it (unless the config slice sets an override) via router::system_prompt::get; the harness fetches it at turn creation.

The first real provider implementing this protocol is provider-anthropic/ — useful as a reference implementation alongside the scripted provider in the integration tests. provider-openai/ follows the same structure for the OpenAI Chat Completions API (native structured output, reasoning_effort).

Local development & testing

cargo test                       # unit suite, no engine needed
cargo test --test integration    # engine-backed suite; self-skips without an engine

The integration suite spawns a throwaway engine per test when iii is on PATH (or III_ENGINE_BIN points at a binary) and covers the chat relay, cancellation, abort, restart recovery, registration token gating, paste-a-key discovery, and event delivery end to end.

To run the worker locally against an engine:

cargo run -- --url ws://127.0.0.1:49134

--url defaults to ws://127.0.0.1:49134 and honours the III_WS_URL environment variable when the flag is not set. --config is accepted per the standard worker CLI but ignored with a warning — operator config lives in the engine's llm-router configuration entry (see Configuration above).