judge-laya

laya provider for the judge hub, running inside the worker: no API key, no Python, no external service. laya is a typed-decision model (ModernBERT-large 421M for English, mmBERT-base 322M for 100+ languages, Apache-2.0) that answers Noul, Choice and Score questions over JSON state in one encoder pass per question. The encoder runs through llama.cpp (the runtime shared with judge-semif) on the CPU, on Metal (macOS) or on Vulkan (Linux x86_64: AMD, NVIDIA and Intel GPUs), picked automatically at start; laya's decision head runs in candle on the CPU.

Install

iii trigger compose::add worker=judge-laya

The functions register at start; the checkpoints load on demand. While laya is the judge hub's default provider (provider: laya under Settings → Workers → judge) they load at once and stay loaded. Otherwise the first call that names "provider": "laya" (or comes from a session that picked it) loads them, and they are released (VRAM included) after 10 minutes without calls; a call that cannot wait for the load answers deadline while the load goes on for the next one. The first load downloads the checkpoint (843 MB for laya, 644 MB for laya-multilingual) from the Hugging Face Hub into the hf-hub cache ($HF_HOME, default ~/.cache/huggingface) and converts its encoder once to the GGUF llama.cpp loads, under $HF_HOME/judge-laya/ (keyed by model and revision; under a second for laya, 791 MB). To keep the checkpoints loaded while another provider is the default, turn on Keep every local provider loaded (preload_all) in the judge settings. A hub build that does not expose judge::configuration-id keeps the checkpoints loaded from the start. Air-gapped installs point III_LAYA_CHECKPOINT_DIR at a directory holding model.safetensors, encoder/config.json, rl_agent_config.json and tokenizer.json (plus an optional pre-converted encoder.gguf; without it the encoder is converted into the temporary directory); III_LAYA_ENCODER_GGUF swaps only the GGUF of the default model, and cargo run --example convert_encoder converts one ahead of time.

The GGUF holds the checkpoint's own (fine-tuned) encoder.* tensors, renamed to llama.cpp's modern-bert layout by the worker (src/gguf.rs): its tensors match llama.cpp's convert_hf_to_gguf.py byte for byte on all three checkpoints (tests/gguf.rs checks the tiny one against that converter's output). Matrices stay f16: Q8_0 moved laya's calibrated probabilities by up to 0.04 and flipped one fixture answer at 512 tokens.

Hardware selection

• macOS: Metal is linked in; Apple Silicon GPUs run the encoder.
• Linux x86_64: llama.cpp's backends are modules loaded at start from the binary's directory. The Vulkan module loads wherever a Vulkan loader and driver exist (libvulkan.so.1); without them it is skipped and the CPU runs the encoder. The package ships libllama, libggml, libggml-base, the CPU variants and the Vulkan module beside the binary (the same layout as judge-semif).
• Linux aarch64: CPU, statically linked.

gpu_layers: 0 keeps the encoder on the CPU even when a GPU is present. The chosen device is logged at start as selected inference device and shown in the settings form.

Make it the default under Settings → Workers → judge, then call the hub:

iii trigger judge::evaluate --timeout-ms 65000 --json '{
  "timeout_ms": 60000, "provider": "laya",
  "evaluations": [{
    "id": "ticket-4411",
    "state": {"subject": "Duplicate charge on invoice #4411",
              "body": "We were billed twice for March. Refund the duplicate today or we cancel."},
    "questions": {
      "department": {"type": "choice", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages"}},
      "urgency": {"type": "score", "criteria": ["not urgent", "soon", "critical deadline or blocking issue"]},
      "churn_risk": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"}
    }
  }]
}'

Answers follow the shared contract exactly: noul is P(true); choice carries the option probabilities and confidence; score carries the level distribution, the probability-weighted score, confidence and the legend. confidence is TypeSafe's (judge_contract::confidence), read from the probabilities after the checkpoint's per-bucket temperature: (n·p_max − 1) / (n − 1) for a choice, 1 − the expected distance from the likeliest level over the levels' mean distance from the middle for a score. usage.input_tokens counts encoder tokens, output_tokens is always 0 and usage_complete is true.

Measured on an i9-14900K and an RX 6900 XT (8 threads, 100–180-token rows): one question answers in about 0.12 s on the CPU and 0.04–0.05 s on Vulkan, against 0.22–0.32 s with the previous candle encoder. Rows that fill the 512-token window cost more; laya's bidirectional encoder reads state and question together, so nothing is shared between questions (no prefix reuse). Keep timeout_ms honest for a big state with a hundred questions, or route such callers to a hosted provider. RAYON_NUM_THREADS in the worker environment overrides the threads setting for the head.

Configuration

Settings → Workers → judge-laya:

Field	Default	Applied
model	laya	next start (laya-multilingual for non-English, laya-typed-decisions for laya's four workflows)
preload	[]	next start (extra checkpoints, ~1.7 GB of RAM each)
auto_route	false	new calls
auto_task_detection	false	new calls
shortlist_k	off	new calls
revision	main	next start
threads	min(8, logical cores)	next start (RAYON_NUM_THREADS in the environment wins for the head)
gpu_layers	all layers when a GPU backend and device exist	next start (0 = CPU)
batch_questions	16	new calls
max_request_bytes	8388608	new calls
max_timeout_ms	300000	new calls

The form shows which checkpoint the running worker actually loaded (through judge-laya::models::list, one card per loaded checkpoint with its context_window) and warns when the selection needs a restart.

Routing and shortlist (laya 0.3.5 Router and predict_shortlist)

A request naming model uses that checkpoint, which must be model or in preload (invalid_request otherwise). Without it, each evaluation picks its own checkpoint: with auto_task_detection, question ids that exactly form one of laya's four typed-decisions workflows (agent_trace_observability, customer_service, invoice_processing, security_incidents) go to laya-typed-decisions; with auto_route, the state's script and language (laya's dependency-free detector, ported with its fixtures) send non-English states to laya-multilingual and English ones to laya. A target that is not loaded falls back to the default. The response model is the checkpoint every evaluation used, or the default when they differ; one forward never mixes checkpoints.

shortlist_k enables laya's opt-in embedding shortlist: a choice question with more options than k embeds instructions + state and every rendered option with the same encoder (mean-pooled states, cosine), keeps the k closest labels for the decision head and reports the others with probability 0. Kept options stay in the contract's key order, where laya reorders them by rank.

Encoding notes

State and structured criteria are serialized exactly as laya's Python json.dumps (", " and ": " separators), because the checkpoint was trained on that spelling. Choice options are encoded in the contract's key order, and JSON object keys (state, criteria) arrive sorted through the bus, so the same logical request can tokenize differently than in Python laya, whose dicts keep insertion order. laya is order-sensitive (one ticket flipped from billing 0.57 to sales 0.69 when its four options were reordered), so answers here are deterministic but not byte-identical to Python's; the port itself matches Python's logits to 1e-4 on identical token sequences for the English and multilingual checkpoints. Sequences are capped at the checkpoint's window (512 tokens for laya, 1024 for laya-multilingual and laya-typed-decisions; option text ≤48 tokens each, header ≤192 or ≤256): a request whose options cannot all fit answers payload_too_large, longer states are truncated on the right (logged as state truncated to the checkpoint window; the model then answers without the dropped part). Requests naming a model that is not loaded answer invalid_request.

For the full API, read the hub's reference.

Building

llama.cpp is compiled from source through crates/llama-runtime: cmake, a C++ compiler and libclang (for bindgen) are required; if libclang lives outside the default search path set LIBCLANG_PATH (and BINDGEN_EXTRA_CLANG_ARGS=-I/include when its builtin headers are not found). Linux x86_64 builds also need the Vulkan loader headers, the SPIR-V headers and glslc (Ubuntu: libvulkan-dev spirv-headers glslc) to compile the Vulkan module. The build copies the modules and libraries beside the binary, so target/release has the published layout; the release catalog ships them as the artifact's companions. Windows is not published yet.