context-manager

Turns a raw conversation history plus a target model into a model-ready context: a system prompt and an ordered message list that fits inside the model's usable token budget. It owns what the model sees this turn — token counting, function-result pruning, and history compaction (summarisation) — and nothing else. It is stateless with respect to conversation storage: pass messages in, get results back, persist them yourself (or with session-manager). Summarisation and model-limit lookups go through llm-router when installed; token counting and pruning work standalone.

Install

iii trigger compose::add worker=context-manager

iii trigger compose::add declares the worker in worker-compose.yaml and starts it as part of the Compose project. Runtime configuration lives in the configuration worker (enabled by default in the engine): at boot the worker registers its schema and fetches the authoritative value, and it hot-reloads on change — every field, with no restart (summarizer_timeout_ms is read per call; a lease_dir change rebuilds and swaps the lease store). An optional --config .yaml only seeds the first registration.

Quickstart

Build a budgeted context before every model call:

use iii_sdk::{register_worker, InitOptions, TriggerRequest};
use serde_json::json;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let iii = register_worker("ws://localhost:49134", InitOptions::default());

    let result = iii.trigger(TriggerRequest {
        function_id: "context::assemble".into(),
        payload: json!({
            "messages": [
                { "role": "user", "content": [{ "type": "text", "text": "hi" }], "timestamp": 1 }
            ],
            // Inline limits keep the call standalone; pass only `id`/`provider`
            // to resolve limits through llm-router instead.
            "model": { "id": "claude-sonnet-4",
                       "limits": { "context_window": 200000, "max_output_tokens": 8000 } },
            "system_prompt": "You are a helpful assistant."
        }),
        action: None,
        timeout_ms: Some(30_000),
    }).await?;

    // -> { system_prompt, messages, token_count, usable, model_resolved,
    //      applied: { pruned, pruned_tokens, capped_parts, capped_tokens,
    //                 compacted, summary?, tail_start_index? } }
    println!("{result:#?}");
    Ok(())
}

The compaction round trip: when applied.compacted is true, persist applied.summary and the boundary your storage maps applied.tail_start_index to. On later calls pass only the post-compaction window as messages plus the stored summary as options.previous_summary — the summary is rendered into the system prompt under # Conversation summary, and any further compaction updates it instead of starting over. Callers that skip persistence stay correct at the cost of one summariser call per over-budget request.

Before token accounting, context::assemble normalizes images in its cloned model-facing view. A known catalog model receives placeholders unless it explicitly declares vision support. For a known vision model, tool images age after the next assistant response and user images age when a later user turn begins after a response. Inline limits and unresolved models keep images. Live images cost a fixed 4,096-token heuristic budget; base64 bytes are never counted as text. The caller's transcript is unchanged.

Before pruning or compaction runs, assemble unconditionally caps any message-level or inline function result over max_result_tokens (default 20000; 0 disables the pass) to a bounded […result capped: was ~N tokens; middle omitted; re-call {function_id} with narrower arguments if the omitted middle is needed] head+tail view. Text uses provider-visible newline separators and each image costs the fixed 4,096-token allowance — see Configuration below.

Every successful context::assemble response satisfies token_count <= usable. Callers should include the complete tools array and set options.request_overhead_tokens for response-format and provider-specific request fields. If ordinary pruning and compaction are insufficient, assembly replaces oversized function results in its model-facing copy with bounded references to the full result retained in the session transcript. Requests that still cannot fit fail with context/overflow; callers must not issue a provider request in that case.

The other three functions: context::count-tokens (estimate messages + tools + system prompt vs a model), context::prune (replace verbose function outputs with [output of {function_id} pruned: was ~N tokens; re-call it if still needed] placeholders, no LLM involved), and context::compact (summarise the head, keep a recent tail verbatim — returns ok | busy | empty | overflow).

Configuration

reserved_tokens_cap: 20000     # default reserve = min(cap, pct% of context window)
reserved_pct: 10
tail_turns: 2                  # user+assistant pairs kept verbatim by compaction
protect_recent_tokens: 40000   # newest function-output tokens never pruned
min_free_tokens: 20000         # skip pruning when it would free less
max_output_chars: 2000         # outputs at or under this size are never pruned
max_result_tokens: 20000       # per-result ceiling for assemble's unconditional cap pass; 0 disables
lease_ttl_secs: 300            # compaction mutual-exclusion lease TTL
allow_fallback_limits: true    # conservative 8192/1024 when limits can't resolve
summarizer_timeout_ms: 320000  # outer budget for one router::chat summariser call

Other details (and the defaults' single source of truth) live in src/config.rs.

Local development & testing

cargo test                          # unit + BDD (engine scenarios soft-skip)
cargo test --test bdd -- --tags @pure    # no engine required
UPDATE_GOLDENS=1 cargo test --test schemas   # regenerate wire-schema goldens