skip to content
iii / worker
$worker

provider-llamacpp

v0.3.1

llama.cpp server (llama-server) Chat Completions provider worker; implements provider::llamacpp::stream and provider::llamacpp::refresh_models behind llm-router.

iiiverified
27 installs5 in 7d0 today
install
$iii worker add provider-llamacpp
binarylicense: Apache-2.0llmllama.cpplocalopen-sourceprovider
  • macOS: arm64 · x64
  • Linux: arm64 · armv7 · x64
  • Windows: arm64 · x64 · x86
6functions0triggers0skills2depsv0.3.1version
repoiii-hq/workers

exact versions are immutable; binary and bundle artifacts are digest-pinned.

functions

6

provider::llamacpp::abort

function

Cancel the in-flight upstream stream for a request_id (router::abort fan-out), stopping billed generation immediately.

request
  • request_idstringrequired
response
  • abortedbooleanrequired

provider::llamacpp::count_tokens

function

Count prompt tokens for {model, system_prompt?, tools?, messages} through the llama-server's own count endpoint, using the loaded model's tokenizer; never runs the model and costs nothing.

request
  • messagesany of[]required

    Wire agent messages, the same shape `provider::llamacpp::stream` accepts. Must be non-empty.

    any of (4)
    variant 1
    • contentone of[]required
      … expand 6 nested
      one of (6)
      variant 1
      • textstringrequired
      • typestringrequiredenum: text
      variant 2
      • datastringrequired
      • mimestringrequired
      • typestringrequiredenum: image
      variant 3
      • signaturestring
      • textstringrequired
      • typestringrequiredenum: thinking
      variant 4
      • datastringrequired
      • typestringrequiredenum: redacted_thinking
      variant 5
      • argumentsunknownrequired
      • function_idstringrequired
      • idstringrequired
      • typestringrequiredenum: function_call
      variant 6
      • contentunknown[]required
      • function_call_idstringrequired
      • is_errorboolean
      • typestringrequiredenum: function_result
    • error_kindany of
      … expand 2 nested
      any of (2)
      variant 1
      valuestringenum: auth_expired, rate_limited, context_overflow, transient, permanent
      variant 2
      valuenull
    • error_messagestring
    • modelstringrequired
    • native_stop_reasonstring
    • providerstringrequired
    • rolestringrequiredenum: assistant
    • stop_reasonstringrequiredenum: end, length, function_call, aborted, error
    • timestampinteger· int64required
    • usageany of
      … expand 2 nested
      any of (2)
      variant 1
      • cache_readinteger· uint64min 0
      • cache_writeinteger· uint64min 0
      • cost_usdnumber· double
      • inputinteger· uint64min 0
      • outputinteger· uint64min 0
      • reasoninginteger· uint64min 0
      variant 2
      valuenull
    • warningsstring[]
    variant 2
    • contentone of[]required
      … expand 6 nested
      one of (6)
      variant 1
      • textstringrequired
      • typestringrequiredenum: text
      variant 2
      • datastringrequired
      • mimestringrequired
      • typestringrequiredenum: image
      variant 3
      • signaturestring
      • textstringrequired
      • typestringrequiredenum: thinking
      variant 4
      • datastringrequired
      • typestringrequiredenum: redacted_thinking
      variant 5
      • argumentsunknownrequired
      • function_idstringrequired
      • idstringrequired
      • typestringrequiredenum: function_call
      variant 6
      • contentunknown[]required
      • function_call_idstringrequired
      • is_errorboolean
      • typestringrequiredenum: function_result
    • detailsunknownrequired
    • function_call_idstringrequired
    • function_idstringrequired
    • is_errorbooleanrequired
    • rolestringrequiredenum: function_result
    • timestampinteger· int64required
    variant 3
    • contentone of[]required
      … expand 6 nested
      one of (6)
      variant 1
      • textstringrequired
      • typestringrequiredenum: text
      variant 2
      • datastringrequired
      • mimestringrequired
      • typestringrequiredenum: image
      variant 3
      • signaturestring
      • textstringrequired
      • typestringrequiredenum: thinking
      variant 4
      • datastringrequired
      • typestringrequiredenum: redacted_thinking
      variant 5
      • argumentsunknownrequired
      • function_idstringrequired
      • idstringrequired
      • typestringrequiredenum: function_call
      variant 6
      • contentunknown[]required
      • function_call_idstringrequired
      • is_errorboolean
      • typestringrequiredenum: function_result
    • custom_typestringrequired
    • detailsunknown
    • displaystring
    • rolestringrequiredenum: custom
    • timestampinteger· int64required
    variant 4
    • contentone of[]required
      … expand 6 nested
      one of (6)
      variant 1
      • textstringrequired
      • typestringrequiredenum: text
      variant 2
      • datastringrequired
      • mimestringrequired
      • typestringrequiredenum: image
      variant 3
      • signaturestring
      • textstringrequired
      • typestringrequiredenum: thinking
      variant 4
      • datastringrequired
      • typestringrequiredenum: redacted_thinking
      variant 5
      • argumentsunknownrequired
      • function_idstringrequired
      • idstringrequired
      • typestringrequiredenum: function_call
      variant 6
      • contentunknown[]required
      • function_call_idstringrequired
      • is_errorboolean
      • typestringrequiredenum: function_result
    • rolestringrequiredenum: user

      Single-variant role tags: exact-match on deserialize, correct wire string on serialize, and they let `AgentMessage` be an untagged union.

    • timestampinteger· int64required
  • modelstringrequired

    Model id the prompt targets. `llama-server` serves one model at a time and counts with whichever it loaded, so this is carried for the reply rather than to select a tokenizer.

  • system_promptstring

    System prompt counted as the leading wire message when present.

  • toolsobject[]

    Function invocation schemas; mapped to the wire `tools` array.

    • descriptionstringrequired
    • execution_modestring
    • labelstring
    • namestringrequired
    • parametersunknownrequired
response
  • estimatorstringrequired

    Always `metered`: the server that holds the model produced the count.

  • modelstringrequired
  • tokensinteger· uint64requiredmin 0

    Prompt tokens the server counted with the loaded model's tokenizer.

provider::llamacpp::embed

function

Batch text embeddings via the configured llama-server's /v1/embeddings (requires --embeddings and an embedding-capable model). One vector per input, order preserved. Fully local; credential only when the server runs with --api-key.

request
  • inputstring[]required1…512 items

    Texts to embed, one vector returned per input, order preserved.

  • modelstring

    Model name passed through to the server. llama-server embeds with its loaded model regardless; the field is echoed for parity with the other providers.

response
  • embeddingsnumber[][]required

    One embedding per input, in input order.

  • modelstringrequired

provider::llamacpp::on_router_ready

function

Internal: router::ready subscriber that re-declares this provider and refreshes its catalog.

request
empty object
response
  • okbooleanrequired

provider::llamacpp::refresh_models

function

Discover the resolved llama.cpp server's live model catalog (GET /v1/models + /props) and reconcile it through the router; returns the model count written.

request
empty object
response
  • countinteger· uintrequiredmin 0
  • okbooleanrequired

provider::llamacpp::stream

function

Stream a llama.cpp server chat completion: resolve credentials (optional — most local servers run with no --api-key), call the upstream Chat Completions API, and relay AssistantMessageEvent frames to writer_ref.

request
  • max_output_tokensinteger· uint64min 0
  • messagesany of[]required
    any of (4)
    variant 1
    • contentone of[]required
      … expand 6 nested
      one of (6)
      variant 1
      • textstringrequired
      • typestringrequiredenum: text
      variant 2
      • datastringrequired
      • mimestringrequired
      • typestringrequiredenum: image
      variant 3
      • signaturestring
      • textstringrequired
      • typestringrequiredenum: thinking
      variant 4
      • datastringrequired
      • typestringrequiredenum: redacted_thinking
      variant 5
      • argumentsunknownrequired
      • function_idstringrequired
      • idstringrequired
      • typestringrequiredenum: function_call
      variant 6
      • contentunknown[]required
      • function_call_idstringrequired
      • is_errorboolean
      • typestringrequiredenum: function_result
    • error_kindany of
      … expand 2 nested
      any of (2)
      variant 1
      valuestringenum: auth_expired, rate_limited, context_overflow, transient, permanent
      variant 2
      valuenull
    • error_messagestring
    • modelstringrequired
    • native_stop_reasonstring
    • providerstringrequired
    • rolestringrequiredenum: assistant
    • stop_reasonstringrequiredenum: end, length, function_call, aborted, error
    • timestampinteger· int64required
    • usageany of
      … expand 2 nested
      any of (2)
      variant 1
      • cache_readinteger· uint64min 0
      • cache_writeinteger· uint64min 0
      • cost_usdnumber· double
      • inputinteger· uint64min 0
      • outputinteger· uint64min 0
      • reasoninginteger· uint64min 0
      variant 2
      valuenull
    • warningsstring[]
    variant 2
    • contentone of[]required
      … expand 6 nested
      one of (6)
      variant 1
      • textstringrequired
      • typestringrequiredenum: text
      variant 2
      • datastringrequired
      • mimestringrequired
      • typestringrequiredenum: image
      variant 3
      • signaturestring
      • textstringrequired
      • typestringrequiredenum: thinking
      variant 4
      • datastringrequired
      • typestringrequiredenum: redacted_thinking
      variant 5
      • argumentsunknownrequired
      • function_idstringrequired
      • idstringrequired
      • typestringrequiredenum: function_call
      variant 6
      • contentunknown[]required
      • function_call_idstringrequired
      • is_errorboolean
      • typestringrequiredenum: function_result
    • detailsunknownrequired
    • function_call_idstringrequired
    • function_idstringrequired
    • is_errorbooleanrequired
    • rolestringrequiredenum: function_result
    • timestampinteger· int64required
    variant 3
    • contentone of[]required
      … expand 6 nested
      one of (6)
      variant 1
      • textstringrequired
      • typestringrequiredenum: text
      variant 2
      • datastringrequired
      • mimestringrequired
      • typestringrequiredenum: image
      variant 3
      • signaturestring
      • textstringrequired
      • typestringrequiredenum: thinking
      variant 4
      • datastringrequired
      • typestringrequiredenum: redacted_thinking
      variant 5
      • argumentsunknownrequired
      • function_idstringrequired
      • idstringrequired
      • typestringrequiredenum: function_call
      variant 6
      • contentunknown[]required
      • function_call_idstringrequired
      • is_errorboolean
      • typestringrequiredenum: function_result
    • custom_typestringrequired
    • detailsunknown
    • displaystring
    • rolestringrequiredenum: custom
    • timestampinteger· int64required
    variant 4
    • contentone of[]required
      … expand 6 nested
      one of (6)
      variant 1
      • textstringrequired
      • typestringrequiredenum: text
      variant 2
      • datastringrequired
      • mimestringrequired
      • typestringrequiredenum: image
      variant 3
      • signaturestring
      • textstringrequired
      • typestringrequiredenum: thinking
      variant 4
      • datastringrequired
      • typestringrequiredenum: redacted_thinking
      variant 5
      • argumentsunknownrequired
      • function_idstringrequired
      • idstringrequired
      • typestringrequiredenum: function_call
      variant 6
      • contentunknown[]required
      • function_call_idstringrequired
      • is_errorboolean
      • typestringrequiredenum: function_result
    • rolestringrequiredenum: user

      Single-variant role tags: exact-match on deserialize, correct wire string on serialize, and they let `AgentMessage` be an untagged union.

    • timestampinteger· int64required
  • modelstringrequired
  • model_metaany of
    any of (2)
    variant 1
    • context_windowinteger· uint64requiredmin 0
    • display_namestring
    • idstringrequired
    • input_limitinteger· uint64min 0
    • max_output_tokensinteger· uint64requiredmin 0
    • pricingany of
      … expand 2 nested
      any of (2)
      variant 1
      • cache_readnumber· double
      • cache_writenumber· double
      • inputnumber· double
      • outputnumber· double
      variant 2
      valuenull
    • providerstringrequired
    • reasoning_effortsobject[]
      … expand 2 nested
      • descriptionstring
      • effortstringrequired
    • supports_cacheboolean
    • supports_structured_outputboolean
    • supports_thinkingboolean
    • supports_toolsboolean
    • supports_visionboolean
    • supports_xhighboolean
    • thinking_budgetsobject
    variant 2
    valuenull
  • provider_optionsunknown
  • resolution_keystring
  • response_formatany of
    any of (2)
    variant 1
    • schemaunknown
    • typestringrequired
    variant 2
    valuenull
  • system_promptstring
  • thinking_levelany of
    any of (2)
    variant 1
    valuestringenum: minimal, low, medium, high, xhigh
    variant 2
    valuenull
  • toolsobject[]
    • descriptionstringrequired
    • execution_modestring
    • labelstring
    • namestringrequired
    • parametersunknownrequired
  • writer_refobjectrequired
    • access_keystringrequired
    • channel_idstringrequired
    • directionstringrequiredenum: read, write
response
  • okbooleanrequired

triggers

0
no triggers registered