$worker
provider-llamacpp
v0.3.0llama.cpp server (llama-server) Chat Completions provider worker; implements provider::llamacpp::stream and provider::llamacpp::refresh_models behind llm-router.
iiiverifiedinstall
$iii worker add provider-llamacpp@0.3.0
- macOS: arm64 · x64
- Linux: arm64 · armv7 · x64
- Windows: arm64 · x64 · x86
repoiii-hq/workers
exact versions are immutable; binary and bundle artifacts are digest-pinned.
agent-ready brief for v0.3.0
install + config + dependencies + readme + api reference, all in one place. fetch as agent-context.md for an llm to consume.
full markdown
the same content rendered as discrete blocks below is exposed as a single markdown document at
/workers/provider-llamacpp.md?version=0.3.0. paste it into an llm prompt or pipe it through curl from a worker.install
install
$iii worker add provider-llamacpp@0.3.0
dependencies
readme
README.md
provider-llamacpp
llama.cpp server (llama-server)
Chat Completions provider worker behind llm-router.
Implements the provider protocol from
tech-specs/2026-06-agentic/llm-router.md: provider::llamacpp::stream
(SSE chunks → AssistantMessageEvent frames into a router-owned channel) and
provider::llamacpp::refresh_models (live GET /v1/models + GET /props →
router::models::reconcile).
Default upstream: http://127.0.0.1:8080/v1/chat/completions — llama-server's
own default bind address and port. Point api_url at any running
llama-server instance (local, LAN, or a remote box) to use it.
Embeddings
provider::llamacpp::embed serves batch text embeddings from the same configured server when llama-server runs with --embeddings and an embedding-capable model (e.g. a nomic-embed GGUF). One vector per input, order preserved; behind router::embed, this gives the memory worker fully local semantic recall with no cloud call.
Behavior
- Registration: self-declares via
router::provider::registerwith backoff until acked, and re-declares on therouter::readytrigger type. The declaration carries no models andcredential_env_var: LLAMACPP_API_KEY; the post-register refresh discovers the live catalog from the resolved server. - Identity binding: the router returns a
registration_tokenon first registration; it is persisted in iii-state (scopeprovider-llamacpp, keyregistration_token) and presented on every laterregister/resolve/reconcile. If that state is lost the router rejects re-registration — the operator must clear the binding on the router side. - Credentials are optional — the main difference from every other
provider here.
llama-serveronly requiresAuthorization: Bearerwhen started with--api-key; most local setups run with none at all. Streaming and discovery both resolve credentials viarouter::provider::resolveas usual, but a missing/blank credential is treated as "no key configured", not a configuration error: requests simply go out with noAuthorizationheader. If the server does have--api-keyset and ours is missing or wrong, the server's 401/403 surfaces as the normalauth_expirederror. - Catalog:
src/discovery.rsdiscovers the catalog live —GET /v1/modelslists every id the server serves (no "gpt-"-style family gate: llama.cpp serves arbitrary GGUF aliases, so every id is kept), enriched withGET /propsfor the runtime context size (n_ctx, the operator's--ctx-size— more accurate than/v1/models'meta.n_ctx_train, the model's trained max) and vision-modality support. No pricing (self-hosted). Multi-model router-mode (--models-dir,GET /models,/models/load) is out of scope for v1 — this targets the common single-loaded-model server. - Liveness:
pingat least every 30s of upstream silence; a failed channel write (caller gone /router::abort) drops the SSE receiver and aborts the in-flight HTTP request. - Errors: 401/403 →
auth_expired(only reachable when--api-keyis set),context_length_exceeded/message-sniffed prompt-overflow phrasing →context_overflow, 5xx/network →transient, other 4xx →permanent. No transport retries here — the router owns retry policy. - Structured output: real schema-constrained decoding, unlike
json_object-only providers —response_format: {"type": "json_schema", "schema": {...}}(llama.cpp nests the schema directly, not under an extra OpenAI-stylejson_schemawrapper key), or{"type": "json_object"}with no schema. Every discovered model advertisessupports_structured_output: true. - Reasoning: llama.cpp has no dedicated per-request reasoning switch. Its
only lever is the
enable_thinkingchat-template kwarg, so a requestedthinking_levelis mapped best-effort ontochat_template_kwargs: {"enable_thinking": …}(any level →true, absent →false) — which reasoning GGUFs conventionally gate their thinking channel on. It is effective only if the model's chat template references that key; otherwise reasoning stays whatever the server's--reasoning-formatflag and template dictate, and a report-and-continue warning notes the mapping is best-effort. When the server runs with--reasoning-format deepseek, chain-of-thought streams asreasoning_contentdeltas, which this worker surfaces asthinkingblocks on the channel (src/sse.rs). - Tool calling: requires the server be started with
--jinjaand a chat template that supports tool calls; tool schemas ride as the standard OpenAI{"type":"function","function":{...}}envelope. Every discovered model optimistically advertisessupports_tools: true— llama.cpp silently ignores tools a template can't use, so this never breaks non-tool turns. - Prompt caching:
cache_promptreuse is a server-side default in llama.cpp, not something this provider requests explicitly.
Tests
cargo test # unit (pure modules + TCP stubs)
III_ENGINE_BIN=$(which iii) cargo test --test integration -- --test-threads=1The integration suite spawns a real engine, the real router (path dep), this provider, and a local stub upstream — no external servers required.
Running
The binary takes the standard worker CLI flags: --url (engine WebSocket,
default ws://127.0.0.1:49134, falls back to the III_WS_URL environment
variable), --manifest (print the registry manifest and exit), and
--config (accepted but ignored with a warning — provider config comes
from the llm-router configuration entry).
Point it at a real llama-server:
llama-server -m /path/to/model.gguf --jinja --port 8080
cargo run -- --url ws://127.0.0.1:49134api reference (json)
agent-api-reference.json
{
"functions": [
{
"description": "Cancel the in-flight upstream stream for a request_id (router::abort fan-out), stopping billed generation immediately.",
"metadata": {
"internal": true
},
"name": "provider::llamacpp::abort",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input of a provider's `provider::<id>::abort`: actively cancel the in-flight upstream stream for `request_id` (the router's `request_id`, delivered to the provider as `resolution_key`) so billed generation stops immediately instead of waiting for the provider to notice the closed channel on its next write.",
"properties": {
"request_id": {
"type": "string"
}
},
"required": [
"request_id"
],
"title": "ProviderAbortRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Output of `provider::<id>::abort`. `aborted: false` means the request was unknown — already finished, never started, or aborted before (idempotent).",
"properties": {
"aborted": {
"type": "boolean"
}
},
"required": [
"aborted"
],
"title": "ProviderAbortResponse",
"type": "object"
}
},
{
"description": "Batch text embeddings via the configured llama-server's /v1/embeddings (requires --embeddings and an embedding-capable model). One vector per input, order preserved. Fully local; credential only when the server runs with --api-key.",
"metadata": {
"internal": true
},
"name": "provider::llamacpp::embed",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"input": {
"description": "Texts to embed, one vector returned per input, order preserved.",
"items": {
"type": "string"
},
"maxItems": 512,
"minItems": 1,
"type": "array"
},
"model": {
"default": null,
"description": "Model name passed through to the server. llama-server embeds with its loaded model regardless; the field is echoed for parity with the other providers.",
"type": [
"string",
"null"
]
}
},
"required": [
"input"
],
"title": "EmbedRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"embeddings": {
"description": "One embedding per input, in input order.",
"items": {
"items": {
"format": "float",
"type": "number"
},
"type": "array"
},
"type": "array"
},
"model": {
"type": "string"
}
},
"required": [
"embeddings",
"model"
],
"title": "EmbedResponse",
"type": "object"
}
},
{
"description": "Internal: router::ready subscriber that re-declares this provider and refreshes its catalog.",
"metadata": {},
"name": "provider::llamacpp::on_router_ready",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Event delivered to a provider's `provider::<id>::on_router_ready` (the `router::ready` trigger payload, currently `{}`). Unknown fields are ignored.",
"title": "RouterReadyEvent",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Ack returned by a provider's `provider::<id>::on_router_ready`.",
"properties": {
"ok": {
"type": "boolean"
}
},
"required": [
"ok"
],
"title": "ProviderReadyAck",
"type": "object"
}
},
{
"description": "Discover the resolved llama.cpp server's live model catalog (GET /v1/models + /props) and reconcile it through the router; returns the model count written.",
"metadata": {},
"name": "provider::llamacpp::refresh_models",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input of a provider's `provider::<id>::refresh_models` — takes no arguments. A struct (not `Value`) keeps the request schema concrete; unknown fields (e.g. the engine-injected `_caller_worker_id`) are ignored.",
"title": "RefreshModelsRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Output of `provider::<id>::refresh_models`.",
"properties": {
"count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"ok": {
"type": "boolean"
}
},
"required": [
"count",
"ok"
],
"title": "RefreshModelsResponse",
"type": "object"
}
},
{
"description": "Stream a llama.cpp server chat completion: resolve credentials (optional — most local servers run with no --api-key), call the upstream Chat Completions API, and relay AssistantMessageEvent frames to writer_ref.",
"metadata": {},
"name": "provider::llamacpp::stream",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"AgentFunction": {
"description": "Function invocation schema — what a provider sees as a `tools` array entry (README § Function invocation schema; adapter boundary). These describe iii functions exposed to the model, not provider-native tools.",
"properties": {
"description": {
"type": "string"
},
"execution_mode": {
"type": [
"string",
"null"
]
},
"label": {
"type": [
"string",
"null"
]
},
"name": {
"type": "string"
},
"parameters": true
},
"required": [
"description",
"name",
"parameters"
],
"type": "object"
},
"AgentMessage": {
"anyOf": [
{
"$ref": "#/definitions/AssistantMessage"
},
{
"$ref": "#/definitions/FunctionResultMessage"
},
{
"$ref": "#/definitions/CustomMessage"
},
{
"$ref": "#/definitions/UserMessage"
}
],
"description": "The canonical transcript message union. Untagged: the single-variant role tags disambiguate deserialization."
},
"AssistantMessage": {
"properties": {
"content": {
"items": {
"$ref": "#/definitions/ContentBlock"
},
"type": "array"
},
"error_kind": {
"anyOf": [
{
"$ref": "#/definitions/ErrorKind"
},
{
"type": "null"
}
]
},
"error_message": {
"type": [
"string",
"null"
]
},
"model": {
"type": "string"
},
"native_stop_reason": {
"type": [
"string",
"null"
]
},
"provider": {
"type": "string"
},
"role": {
"$ref": "#/definitions/AssistantRoleTag"
},
"stop_reason": {
"$ref": "#/definitions/StopReason"
},
"timestamp": {
"format": "int64",
"type": "integer"
},
"usage": {
"anyOf": [
{
"$ref": "#/definitions/Usage"
},
{
"type": "null"
}
]
},
"warnings": {
"items": {
"type": "string"
},
"type": [
"array",
"null"
]
}
},
"required": [
"content",
"model",
"provider",
"role",
"stop_reason",
"timestamp"
],
"type": "object"
},
"AssistantRoleTag": {
"enum": [
"assistant"
],
"type": "string"
},
"ChannelDirection": {
"enum": [
"read",
"write"
],
"type": "string"
},
"ContentBlock": {
"description": "Content blocks — the atomic units of message content (README § Content blocks).",
"oneOf": [
{
"properties": {
"text": {
"type": "string"
},
"type": {
"enum": [
"text"
],
"type": "string"
}
},
"required": [
"text",
"type"
],
"type": "object"
},
{
"properties": {
"data": {
"type": "string"
},
"mime": {
"type": "string"
},
"type": {
"enum": [
"image"
],
"type": "string"
}
},
"required": [
"data",
"mime",
"type"
],
"type": "object"
},
{
"properties": {
"signature": {
"type": [
"string",
"null"
]
},
"text": {
"type": "string"
},
"type": {
"enum": [
"thinking"
],
"type": "string"
}
},
"required": [
"text",
"type"
],
"type": "object"
},
{
"description": "Opaque redacted thinking payload — replayed verbatim on the Anthropic wire.",
"properties": {
"data": {
"type": "string"
},
"type": {
"enum": [
"redacted_thinking"
],
"type": "string"
}
},
"required": [
"data",
"type"
],
"type": "object"
},
{
"properties": {
"arguments": true,
"function_id": {
"type": "string"
},
"id": {
"type": "string"
},
"type": {
"enum": [
"function_call"
],
"type": "string"
}
},
"required": [
"arguments",
"function_id",
"id",
"type"
],
"type": "object"
},
{
"properties": {
"content": {
"items": {
"$ref": "#/definitions/ContentBlock"
},
"type": "array"
},
"function_call_id": {
"type": "string"
},
"is_error": {
"type": [
"boolean",
"null"
]
},
"type": {
"enum": [
"function_result"
],
"type": "string"
}
},
"required": [
"content",
"function_call_id",
"type"
],
"type": "object"
}
]
},
"CustomMessage": {
"properties": {
"content": {
"items": {
"$ref": "#/definitions/ContentBlock"
},
"type": "array"
},
"custom_type": {
"type": "string"
},
"details": true,
"display": {
"type": [
"string",
"null"
]
},
"role": {
"$ref": "#/definitions/CustomRoleTag"
},
"timestamp": {
"format": "int64",
"type": "integer"
}
},
"required": [
"content",
"custom_type",
"role",
"timestamp"
],
"type": "object"
},
"CustomRoleTag": {
"enum": [
"custom"
],
"type": "string"
},
"ErrorKind": {
"enum": [
"auth_expired",
"rate_limited",
"context_overflow",
"transient",
"permanent"
],
"type": "string"
},
"FunctionResultMessage": {
"properties": {
"content": {
"items": {
"$ref": "#/definitions/ContentBlock"
},
"type": "array"
},
"details": true,
"function_call_id": {
"type": "string"
},
"function_id": {
"type": "string"
},
"is_error": {
"type": "boolean"
},
"role": {
"$ref": "#/definitions/FunctionResultRoleTag"
},
"timestamp": {
"format": "int64",
"type": "integer"
}
},
"required": [
"content",
"details",
"function_call_id",
"function_id",
"is_error",
"role",
"timestamp"
],
"type": "object"
},
"FunctionResultRoleTag": {
"enum": [
"function_result"
],
"type": "string"
},
"Model": {
"description": "The capability record (README § Model descriptor).",
"properties": {
"context_window": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"display_name": {
"type": [
"string",
"null"
]
},
"id": {
"type": "string"
},
"input_limit": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"pricing": {
"anyOf": [
{
"$ref": "#/definitions/Pricing"
},
{
"type": "null"
}
]
},
"provider": {
"type": "string"
},
"reasoning_efforts": {
"items": {
"$ref": "#/definitions/ReasoningEffort"
},
"type": [
"array",
"null"
]
},
"supports_cache": {
"type": [
"boolean",
"null"
]
},
"supports_structured_output": {
"type": [
"boolean",
"null"
]
},
"supports_thinking": {
"type": [
"boolean",
"null"
]
},
"supports_tools": {
"type": [
"boolean",
"null"
]
},
"supports_vision": {
"type": [
"boolean",
"null"
]
},
"supports_xhigh": {
"type": [
"boolean",
"null"
]
},
"thinking_budgets": {
"additionalProperties": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"type": [
"object",
"null"
]
}
},
"required": [
"context_window",
"id",
"max_output_tokens",
"provider"
],
"type": "object"
},
"Pricing": {
"properties": {
"cache_read": {
"format": "double",
"type": [
"number",
"null"
]
},
"cache_write": {
"format": "double",
"type": [
"number",
"null"
]
},
"input": {
"format": "double",
"type": [
"number",
"null"
]
},
"output": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"type": "object"
},
"ReasoningEffort": {
"description": "One provider-native reasoning effort advertised for a specific model.\n\nValues intentionally remain strings: provider catalogs can add efforts without requiring a router-wide enum release first.",
"properties": {
"description": {
"type": [
"string",
"null"
]
},
"effort": {
"type": "string"
}
},
"required": [
"effort"
],
"type": "object"
},
"ResponseFormat": {
"properties": {
"schema": true,
"type": {
"type": "string"
}
},
"required": [
"type"
],
"type": "object"
},
"StopReason": {
"enum": [
"end",
"length",
"function_call",
"aborted",
"error"
],
"type": "string"
},
"StreamChannelRef": {
"properties": {
"access_key": {
"type": "string"
},
"channel_id": {
"type": "string"
},
"direction": {
"$ref": "#/definitions/ChannelDirection"
}
},
"required": [
"access_key",
"channel_id",
"direction"
],
"type": "object"
},
"ThinkingLevel": {
"description": "\"minimal\" requests the lowest reasoning effort and needs only `thinking` support; levels map to provider-native knobs via `Model::thinking_budgets`.",
"enum": [
"minimal",
"low",
"medium",
"high",
"xhigh"
],
"type": "string"
},
"Usage": {
"properties": {
"cache_read": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"input": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"reasoning": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"UserMessage": {
"properties": {
"content": {
"items": {
"$ref": "#/definitions/ContentBlock"
},
"type": "array"
},
"role": {
"$ref": "#/definitions/UserRoleTag"
},
"timestamp": {
"format": "int64",
"type": "integer"
}
},
"required": [
"content",
"role",
"timestamp"
],
"type": "object"
},
"UserRoleTag": {
"description": "Single-variant role tags: exact-match on deserialize, correct wire string on serialize, and they let `AgentMessage` be an untagged union.",
"enum": [
"user"
],
"type": "string"
}
},
"description": "Input of a provider worker's `provider::<id>::stream` iii function — what the router forwards per attempt. (No `PartialEq`: `iii_sdk::StreamChannelRef` doesn't implement it.)",
"properties": {
"max_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"messages": {
"items": {
"$ref": "#/definitions/AgentMessage"
},
"type": "array"
},
"model": {
"type": "string"
},
"model_meta": {
"anyOf": [
{
"$ref": "#/definitions/Model"
},
{
"type": "null"
}
]
},
"provider_options": true,
"resolution_key": {
"type": [
"string",
"null"
]
},
"response_format": {
"anyOf": [
{
"$ref": "#/definitions/ResponseFormat"
},
{
"type": "null"
}
]
},
"system_prompt": {
"type": [
"string",
"null"
]
},
"thinking_level": {
"anyOf": [
{
"$ref": "#/definitions/ThinkingLevel"
},
{
"type": "null"
}
]
},
"tools": {
"items": {
"$ref": "#/definitions/AgentFunction"
},
"type": [
"array",
"null"
]
},
"writer_ref": {
"$ref": "#/definitions/StreamChannelRef"
}
},
"required": [
"messages",
"model",
"writer_ref"
],
"title": "ProviderStreamInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Output of a provider's `provider::<id>::stream` (spec § stream contract): the function streams frames to `writer_ref` and returns this ack.",
"properties": {
"ok": {
"type": "boolean"
}
},
"required": [
"ok"
],
"title": "ProviderStreamOutput",
"type": "object"
}
}
],
"triggers": []
}