iii-observability
v0.19.7-next.1OpenTelemetry-based traces, metrics, logs, alerts, and sampling.
exact versions are immutable; binary and bundle artifacts are digest-pinned.
full markdown
/workers/iii-observability.md?version=0.19.7-next.1. paste it into an llm prompt or pipe it through curl from a worker.install
configuration
- enabled: true
exporter: memory
logs_console_output: true
logs_enabled: true
logs_exporter: memory
logs_max_count: 1000
logs_retention_seconds: 3600
memory_max_spans: 1000000
metrics_enabled: true
metrics_exporter: memory
metrics_max_count: 10000
metrics_retention_seconds: 3600
sampling_ratio: 1
service_name: iii
service_version: ${SERVICE_VERSION:__III_ENGINE_VERSION__}dependencies
readme
iii-observability
Full OpenTelemetry observability for III Engine: distributed tracing, structured logs, performance metrics, alert rules, and trace sampling — all queryable via built-in functions.
Install
iii worker add iii-observabilityResolves from the worker registry at workers.iii.dev.
Skills
Install the iii-observability agent skill for Claude Code, Cursor, and 30+ other agents:
npx skills add iii-hq/iii --full-depth --skill iii-observabilitySample Configuration
- name: iii-observability
config:
enabled: true
service_name: my-service
service_version: 1.0.0
exporter: memory
metrics_enabled: true
logs_enabled: true
memory_max_spans: 1000
sampling_ratio: 1.0
alerts:
- name: high-error-rate
metric: iii.invocations.error
threshold: 10
operator: ">"
window_seconds: 60
action:
type: logConfigure
The full configuration surface is registered with the builtin configuration
worker under the id iii-observability. The stored entry is the runtime
source of truth; the config.yaml block is seed-only — it populates the
entry on the very first boot and is ignored afterwards. To change a setting
after first boot, edit the entry (console, or configuration::set { "id": "iii-observability", "value": { ... } }); editing config.yaml
alone has no effect anymore.
With the default file-backed adapter the entry persists at
./data/configuration/iii-observability.yaml and is read again at every
engine start — before logging/tracing init — so even restart-tier fields
edited at runtime apply on the next start. ${VAR:default} placeholders work
in string fields and are expanded on read.
Values are validated against the JSON schema at configuration::set time
(unknown fields rejected, ratios bounded to 0..=1, counts ≥ 1). Two
caveats:
- Alert
operatorsymbols (>,<, ...) are accepted inconfig.yamlonly; remote edits must use the canonical names the schema advertises (greaterthan,lessthan, ...). - After a schema tightening, a previously-stored out-of-range value makes
the boot-time schema refresh fail with
SCHEMA_INVALID(warn-and-continue); reads still work and out-of-range values are clamped on read.
Hot Reload
configuration:updated events are applied per field tier:
| Tier | Fields | Effect |
|---|---|---|
| Live | logs_console_output, logs_sampling_ratio, logs_enabled (ingest gate), enabled (ingest gate) |
Immediate — read per use |
| Limits | memory_max_spans, logs_max_count, metrics_max_count, metrics_retention_seconds |
Immediate — enforced on the next insert / 60s sweep |
| Swap | sampling_ratio, sampling.*, alerts, collapse_spans, level |
Immediate — compiled artifact rebuilt and swapped (alert states of surviving rules keep cooldown/firing continuity) |
| Task rebuild | logs_exporter, logs_batch_size, logs_flush_interval_ms, logs_retention_seconds, and logs_enabled on a false→true transition |
The background task restarts with the new settings. A logs_enabled false→true toggle revives the log store and respawns the log-trigger subscriber, OTLP logs exporter, and retention task — so the log trigger fan-out and OTLP log export reactivate without an engine restart |
| Restart-only | exporter, endpoint (trace and logs exporters), service_name/service_version/service_namespace (trace resource and logs exporter identity), format, metrics_enabled, metrics_exporter, enabled (pipeline construction) |
Logged as a warning; applied at the next engine start via the persisted entry. endpoint/service_name/service_version are restart-tier for all signals so logs and traces always move to the new collector/identity together, never split mid-edit |
Known limitation: an engine config-file reload that destroys and recreates this worker shuts down the OTLP trace/metric providers without rebuilding them (they are process-global set-once state) — OTLP export then requires an engine restart. Memory-backed stores are unaffected.
Configuration
| Field | Type | Description |
|---|---|---|
enabled |
boolean | Whether OpenTelemetry tracing export is enabled. Defaults to false. Env: OTEL_ENABLED. |
service_name |
string | Service name in traces and metrics. Defaults to "iii". Env: OTEL_SERVICE_NAME. |
service_version |
string | Service version (service.version OTEL attribute). Env: SERVICE_VERSION. |
service_namespace |
string | Service namespace (service.namespace OTEL attribute). Env: SERVICE_NAMESPACE. |
exporter |
string | Trace exporter: memory, otlp, or both. Use both when traces should remain queryable in iii while also being exported. Defaults to otlp. Env: OTEL_EXPORTER_TYPE. |
endpoint |
string | OTLP collector base endpoint. Defaults to "http://localhost:4317". Env: OTEL_EXPORTER_OTLP_ENDPOINT. |
sampling_ratio |
number | Global trace sampling ratio (0.0–1.0). Defaults to 1.0. Env: OTEL_TRACES_SAMPLER_ARG. |
memory_max_spans |
number | Max spans to keep in memory. Defaults to 1000. Env: OTEL_MEMORY_MAX_SPANS. |
metrics_enabled |
boolean | Whether metrics collection is enabled. Defaults to false. Env: OTEL_METRICS_ENABLED. |
metrics_exporter |
string | Metrics exporter: memory or otlp. Defaults to memory. Env: OTEL_METRICS_EXPORTER. |
metrics_retention_seconds |
number | How long to retain metrics in memory. Defaults to 3600. Env: OTEL_METRICS_RETENTION_SECONDS. |
metrics_max_count |
number | Max metric data points in memory. Defaults to 10000. Env: OTEL_METRICS_MAX_COUNT. |
logs_enabled |
boolean | Whether structured log storage is enabled. |
logs_exporter |
string | Logs exporter: memory, otlp, or both. Use both when logs should remain queryable in iii while also being exported. Defaults to memory. Env: OTEL_LOGS_EXPORTER. |
logs_max_count |
number | Max log entries in memory. Defaults to 1000. |
logs_retention_seconds |
number | How long to retain logs in memory. Defaults to 3600. |
logs_sampling_ratio |
number | Fraction of logs to retain (0.0–1.0). Defaults to 1.0. |
logs_console_output |
boolean | Print ingested logs to the console. Defaults to true. |
level |
string | Minimum log level: trace, debug, info, warn, error. Defaults to info. |
format |
string | Log output format: default or json. Defaults to default. |
alerts |
AlertRule[] | Alert rules evaluated against metrics. |
OTLP Transport
Traces and metrics export over OTLP/gRPC by default. https:// endpoints use TLS with system roots,
while http:// endpoints use cleartext transport.
To export traces and metrics with OTLP/HTTP protobuf instead of gRPC, set the standard OpenTelemetry protocol environment variable before starting the engine:
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobufSignal-specific protocol variables override the global value:
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_METRICS_PROTOCOL=http/protobufWhen HTTP/protobuf is selected, iii treats endpoint as the collector base URL and appends the
signal path when needed:
- traces:
/v1/traces - metrics:
/v1/metrics
The logs exporter sends OTLP logs over HTTP and posts to /v1/logs.
Collectors that require authentication or routing headers can use the standard OTLP headers environment variables:
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer $OTLP_TOKEN"Use signal-specific headers when one signal needs different values:
OTEL_EXPORTER_OTLP_TRACES_HEADERSOTEL_EXPORTER_OTLP_METRICS_HEADERSOTEL_EXPORTER_OTLP_LOGS_HEADERS
The logs exporter reads OTEL_EXPORTER_OTLP_LOGS_HEADERS first and falls back to
OTEL_EXPORTER_OTLP_HEADERS. Keep tokens in environment variables or a secret manager; do not
commit them to config files.
To keep the iii console useful while exporting to an external collector, use exporter: both for
traces and logs_exporter: both for logs.
Alert Rule Fields
| Field | Type | Description |
|---|---|---|
name |
string | Required. Unique alert rule name. |
metric |
string | Required. Metric name to monitor (e.g., iii.invocations.error). |
threshold |
number | Required. Threshold value. |
operator |
string | Comparison operator: >, >=, <, <=, ==, !=. Defaults to >. |
window_seconds |
number | Time window in seconds for metric evaluation. Defaults to 60. |
cooldown_seconds |
number | Minimum interval between alert fires. Defaults to 60. |
enabled |
boolean | Whether the alert rule is active. Defaults to true. |
action |
AlertAction | { "type": "log" }, { "type": "webhook", "url": "..." }, or { "type": "function", "path": "..." }. |
Advanced Sampling
sampling:
default: 1.0
parent_based: true
rules:
- operation: "api.*"
rate: 0.1
rate_limit:
max_traces_per_second: 100Functions
Logging
| Function | Description |
|---|---|
engine::log::info |
Log an informational message. |
engine::log::warn |
Log a warning message. |
engine::log::error |
Log an error message. |
engine::log::debug |
Log a debug message. |
engine::log::trace |
Log a trace-level message. |
All logging functions accept: message (string, required), data (object), trace_id (string),
span_id (string), service_name (string).
Logs API
| Function | Description |
|---|---|
engine::logs::list |
Query stored log entries. Filters: start_time, end_time, trace_id, span_id, severity_min, severity_text, offset, limit. |
engine::logs::clear |
Clear all stored log entries from memory. |
Traces API
| Function | Description |
|---|---|
engine::traces::list |
List stored spans. Filters: trace_id, service_name, name, status, min_duration_ms, max_duration_ms, start_time, end_time, sort_by, sort_order, attributes, include_internal, offset, limit. |
engine::traces::tree |
Retrieve a trace as a hierarchical span tree. Parameters: trace_id (required). |
engine::traces::clear |
Clear all stored trace spans from memory. |
Metrics API
| Function | Description |
|---|---|
engine::metrics::list |
List metrics with aggregated statistics. Returns engine counters (invocations, workers, performance), SDK metrics, and optional time-bucketed aggregations. |
engine::rollups::list |
List metric rollup aggregations (1-minute, 5-minute, 1-hour windows). |
Other APIs
| Function | Description |
|---|---|
engine::baggage::get |
Get a baggage value from the current trace context. |
engine::baggage::set |
Set a baggage value in the current trace context. |
engine::baggage::get_all |
Get all baggage key-value pairs. |
engine::sampling::rules |
List all active sampling rules. |
engine::health::check |
Check engine health status. Returns status, components, timestamp, version. |
engine::alerts::list |
List all configured alert rules and their current state. |
engine::alerts::evaluate |
Manually trigger evaluation of all alert rules. |
Trigger Type: log
Register a function to react to log entries as they are produced.
| Config Field | Type | Description |
|---|---|---|
level |
string | Log level to subscribe to: info, warn, error, debug, or trace. When omitted, fires for all levels. |
Sample Code
const fn = iii.registerFunction("monitoring::onError", async (logEntry) => {
await sendAlert({
message: logEntry.body,
severity: logEntry.severity_text,
traceId: logEntry.trace_id,
});
return {};
});
iii.registerTrigger({
type: "log",
function_id: fn.id,
config: { level: "error" },
});Log entry payload fields: timestamp_unix_nano, observed_timestamp_unix_nano, severity_number,
severity_text, body, attributes, trace_id, span_id, resource, service_name,
instrumentation_scope_name, instrumentation_scope_version.
Trigger Type: trace
Register a function to react to span activity in the in-memory trace store, so any client — a worker
or the web console — can refresh reactively instead of polling. The trigger is a coalesced "traces
changed" tick, not a per-span feed: span activity is debounced (~300ms) and the handler receives
the distinct affected trace ids for the window. Re-read details via engine::traces::list /
engine::traces::tree. Requires the memory exporter (exporter: memory or both); with the
OTLP-only exporter there is no in-memory store and trace triggers stay dormant.
Engine-internal spans and the trigger's own delivery spans are excluded from firing it —
delivering a trigger via engine.call is itself instrumented as a span, so without this exclusion
the trigger would re-fire on its own output (an unbounded feedback loop). This is why a span trigger
differs from the log trigger, whose delivery produces spans, not logs.
| Config Field | Type | Description |
|---|---|---|
service_name |
string | Only fire for activity from this service. When omitted, fires for any service. Compared case-insensitively. |
status |
string | Only fire when a span with this status (ok, error, or unset) landed in the window. When omitted, fires for any status. Compared case-insensitively. |
Both filters are ANDed; omit both to fire on any span activity.
Sample Code
const fn = iii.registerFunction("devtools::onTracesChanged", async ({ trace_ids }) => {
// A "refetch soon" beat — re-read the traces you care about.
await refreshTraceViews(trace_ids);
return {};
});
iii.registerTrigger({
type: "trace",
function_id: fn.id,
config: {}, // or { status: 'error' }
});Handler payload: { "trace_ids": string[] } — the distinct trace ids with span activity in the
coalesced window.
api reference (json)
{
"functions": [
{
"description": "Force an immediate alert-rule evaluation against current metrics and return any triggered_alerts, bypassing the periodic tick. Returns evaluated:false when the alert manager is not initialized; produces nothing without configured rules.",
"metadata": {},
"name": "engine::alerts::evaluate",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "AlertsEvaluateInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"evaluated": {
"type": "boolean"
},
"message": {
"type": [
"string",
"null"
]
},
"timestamp": {
"format": "int64",
"type": "integer"
},
"triggered_alerts": {
"description": "Alerts triggered by this evaluation pass (present when the manager is initialized).",
"items": true,
"type": [
"array",
"null"
]
}
},
"required": [
"evaluated",
"timestamp"
],
"title": "AlertsEvaluateResult",
"type": "object"
}
},
{
"description": "List configured alert rules with their current evaluated states and a firing_count. Returns empty when no alert rules are configured or the alert manager is not initialized.",
"metadata": {},
"name": "engine::alerts::list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "AlertsListInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"AlertAction": {
"description": "Alert action type",
"oneOf": [
{
"description": "Log the alert (default)",
"properties": {
"type": {
"enum": [
"log"
],
"type": "string"
}
},
"required": [
"type"
],
"type": "object"
},
{
"description": "Send webhook notification to the specified URL",
"properties": {
"type": {
"enum": [
"webhook"
],
"type": "string"
},
"url": {
"description": "The webhook URL to send the alert to",
"type": "string"
}
},
"required": [
"type",
"url"
],
"type": "object"
},
{
"description": "Invoke a function at the specified path",
"properties": {
"path": {
"description": "The function path to invoke",
"type": "string"
},
"type": {
"enum": [
"function"
],
"type": "string"
}
},
"required": [
"path",
"type"
],
"type": "object"
}
]
},
"AlertOperator": {
"description": "Comparison operator for alert thresholds NOTE: the symbol aliases (`>`, `>=`, ...) are serde-only conveniences for config.yaml. The generated JSON Schema advertises the canonical lowercase names, and `configuration::set` validates against the schema — so remote edits must use the canonical names.",
"enum": [
"greaterthan",
"greaterthanorequal",
"lessthan",
"lessthanorequal",
"equal",
"notequal"
],
"type": "string"
},
"AlertRule": {
"additionalProperties": false,
"description": "Single alert rule configuration",
"properties": {
"action": {
"allOf": [
{
"$ref": "#/definitions/AlertAction"
}
],
"default": {
"type": "log"
},
"description": "Action to take when alert triggers (Log, Webhook { url }, Function { path })"
},
"cooldown_seconds": {
"default": 60,
"description": "Minimum interval between alert triggers in seconds (default: 60)",
"format": "uint64",
"minimum": 1,
"type": "integer"
},
"enabled": {
"default": true,
"description": "Whether the alert is enabled (default: true)",
"type": "boolean"
},
"metric": {
"description": "Metric name to monitor (e.g., \"iii.invocations.error\")",
"type": "string"
},
"name": {
"description": "Name of the alert (for identification)",
"type": "string"
},
"operator": {
"allOf": [
{
"$ref": "#/definitions/AlertOperator"
}
],
"default": "greaterthan",
"description": "Comparison operator (>, >=, <, <=, ==, !=)"
},
"threshold": {
"description": "Threshold value for the alert",
"format": "double",
"type": "number"
},
"window_seconds": {
"default": 60,
"description": "Time window in seconds to evaluate the metric (default: 60)",
"format": "uint64",
"minimum": 1,
"type": "integer"
}
},
"required": [
"metric",
"name",
"threshold"
],
"type": "object"
}
},
"properties": {
"alerts": {
"description": "Current evaluated alert states (serialized).",
"items": true,
"type": "array"
},
"firing_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"message": {
"type": [
"string",
"null"
]
},
"rules": {
"description": "Configured alert rules (present when the alert manager is initialized).",
"items": {
"$ref": "#/definitions/AlertRule"
},
"type": [
"array",
"null"
]
},
"timestamp": {
"format": "int64",
"type": "integer"
}
},
"required": [
"alerts",
"firing_count",
"timestamp"
],
"title": "AlertsListResult",
"type": "object"
}
},
{
"description": "Read one baggage entry by key from the current OTEL context, returning { value } (null if unset). Diagnostic only: reads ambient process context, not per-invocation baggage.",
"metadata": {},
"name": "engine::baggage::get",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input for baggage.get function",
"properties": {
"key": {
"description": "The baggage key to retrieve",
"type": "string"
}
},
"required": [
"key"
],
"title": "BaggageGetInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"value": {
"type": [
"string",
"null"
]
}
},
"title": "BaggageGetResult",
"type": "object"
}
},
{
"description": "Read all baggage entries from the current OTEL context as a { baggage } map. Diagnostic only: reflects ambient process context, not per-invocation baggage.",
"metadata": {},
"name": "engine::baggage::get_all",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input for baggage.getAll function (empty)",
"title": "BaggageGetAllInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"baggage": {
"additionalProperties": {
"type": "string"
},
"type": "object"
}
},
"required": [
"baggage"
],
"title": "BaggageGetAllResult",
"type": "object"
}
},
{
"description": "Set a baggage key/value on a fresh OTEL context for verification only; the new context is not propagated to the caller or global state. Use SDK-level headers for real propagation.",
"metadata": {},
"name": "engine::baggage::set",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input for baggage.set function",
"properties": {
"key": {
"description": "The baggage key to set",
"type": "string"
},
"value": {
"description": "The baggage value to set",
"type": "string"
}
},
"required": [
"key",
"value"
],
"title": "BaggageSetInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"note": {
"type": "string"
},
"success": {
"type": "boolean"
}
},
"required": [
"note",
"success"
],
"title": "BaggageSetResult",
"type": "object"
}
},
{
"description": "Report observability subsystem health: per-component status (otel, metrics, logs, spans) marked healthy with counts or disabled, plus engine version. Always succeeds regardless of configuration.",
"metadata": {},
"name": "engine::health::check",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "HealthCheckInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"HealthComponentsView": {
"properties": {
"logs": true,
"metrics": true,
"otel": {
"description": "Each component is `{ status: \"healthy\"|\"disabled\", details: <object|null> }`."
},
"spans": true
},
"required": [
"logs",
"metrics",
"otel",
"spans"
],
"type": "object"
}
},
"properties": {
"components": {
"$ref": "#/definitions/HealthComponentsView"
},
"status": {
"type": "string"
},
"timestamp": {
"format": "int64",
"type": "integer"
},
"version": {
"type": "string"
}
},
"required": [
"components",
"status",
"timestamp",
"version"
],
"title": "HealthCheckResult",
"type": "object"
}
},
{
"description": "Record a DEBUG-level OTEL log (severity 5) with optional trace/span correlation and structured data. No-op when logs_enabled is false or dropped by logs_sampling_ratio.",
"metadata": {},
"name": "engine::log::debug",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input for OTEL log functions (log.info, log.warn, log.error)",
"properties": {
"data": {
"description": "Additional structured data/attributes"
},
"message": {
"description": "The log message",
"type": "string"
},
"service_name": {
"description": "Service name (defaults to function name if not provided)",
"type": [
"string",
"null"
]
},
"span_id": {
"description": "Optional span ID for correlation",
"type": [
"string",
"null"
]
},
"trace_id": {
"description": "Optional trace ID for correlation",
"type": [
"string",
"null"
]
}
},
"required": [
"message"
],
"title": "OtelLogInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "Record an ERROR-level OTEL log (severity 17) with optional trace/span correlation and structured data. No-op when logs_enabled is false or dropped by logs_sampling_ratio.",
"metadata": {},
"name": "engine::log::error",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input for OTEL log functions (log.info, log.warn, log.error)",
"properties": {
"data": {
"description": "Additional structured data/attributes"
},
"message": {
"description": "The log message",
"type": "string"
},
"service_name": {
"description": "Service name (defaults to function name if not provided)",
"type": [
"string",
"null"
]
},
"span_id": {
"description": "Optional span ID for correlation",
"type": [
"string",
"null"
]
},
"trace_id": {
"description": "Optional trace ID for correlation",
"type": [
"string",
"null"
]
}
},
"required": [
"message"
],
"title": "OtelLogInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "Record an INFO-level OTEL log (severity 9) with optional trace/span correlation and structured data. No-op when logs_enabled is false or dropped by logs_sampling_ratio.",
"metadata": {},
"name": "engine::log::info",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input for OTEL log functions (log.info, log.warn, log.error)",
"properties": {
"data": {
"description": "Additional structured data/attributes"
},
"message": {
"description": "The log message",
"type": "string"
},
"service_name": {
"description": "Service name (defaults to function name if not provided)",
"type": [
"string",
"null"
]
},
"span_id": {
"description": "Optional span ID for correlation",
"type": [
"string",
"null"
]
},
"trace_id": {
"description": "Optional trace ID for correlation",
"type": [
"string",
"null"
]
}
},
"required": [
"message"
],
"title": "OtelLogInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "Record a TRACE-level OTEL log (severity 1) with optional trace/span correlation and structured data. No-op when logs_enabled is false or dropped by logs_sampling_ratio.",
"metadata": {},
"name": "engine::log::trace",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input for OTEL log functions (log.info, log.warn, log.error)",
"properties": {
"data": {
"description": "Additional structured data/attributes"
},
"message": {
"description": "The log message",
"type": "string"
},
"service_name": {
"description": "Service name (defaults to function name if not provided)",
"type": [
"string",
"null"
]
},
"span_id": {
"description": "Optional span ID for correlation",
"type": [
"string",
"null"
]
},
"trace_id": {
"description": "Optional trace ID for correlation",
"type": [
"string",
"null"
]
}
},
"required": [
"message"
],
"title": "OtelLogInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "Record a WARN-level OTEL log (severity 13) with optional trace/span correlation and structured data. No-op when logs_enabled is false or dropped by logs_sampling_ratio.",
"metadata": {},
"name": "engine::log::warn",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input for OTEL log functions (log.info, log.warn, log.error)",
"properties": {
"data": {
"description": "Additional structured data/attributes"
},
"message": {
"description": "The log message",
"type": "string"
},
"service_name": {
"description": "Service name (defaults to function name if not provided)",
"type": [
"string",
"null"
]
},
"span_id": {
"description": "Optional span ID for correlation",
"type": [
"string",
"null"
]
},
"trace_id": {
"description": "Optional trace ID for correlation",
"type": [
"string",
"null"
]
}
},
"required": [
"message"
],
"title": "OtelLogInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "Drop every stored OTEL log, returning { success: true }. Succeeds as a no-op when log storage was never initialized (logs_enabled false).",
"metadata": {},
"name": "engine::logs::clear",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "LogsClearInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"message": {
"type": [
"string",
"null"
]
},
"success": {
"type": "boolean"
}
},
"required": [
"success"
],
"title": "LogsClearResult",
"type": "object"
}
},
{
"description": "List stored OTEL logs filtered by trace/span id, severity, and time range, with pagination and a total count. Returns an empty result when logs_enabled is false (no log storage).",
"metadata": {},
"name": "engine::logs::list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"end_time": {
"description": "End time in Unix timestamp milliseconds",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"limit": {
"description": "Maximum number of logs to return",
"format": "uint",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"offset": {
"description": "Pagination offset (default: 0)",
"format": "uint",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"severity_min": {
"description": "Minimum severity number (1-24, higher = more severe)",
"format": "int32",
"type": [
"integer",
"null"
]
},
"severity_text": {
"description": "Filter by severity text (e.g., \"ERROR\", \"WARN\", \"INFO\")",
"type": [
"string",
"null"
]
},
"span_id": {
"description": "Filter by span ID",
"type": [
"string",
"null"
]
},
"start_time": {
"description": "Start time in Unix timestamp milliseconds",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"trace_id": {
"description": "Filter by trace ID",
"type": [
"string",
"null"
]
}
},
"title": "LogsListInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"LogsListQuery": {
"properties": {
"end_time": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"limit": {
"format": "uint",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"offset": {
"format": "uint",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"severity_min": {
"format": "int32",
"type": [
"integer",
"null"
]
},
"severity_text": {
"type": [
"string",
"null"
]
},
"span_id": {
"type": [
"string",
"null"
]
},
"start_time": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"trace_id": {
"type": [
"string",
"null"
]
}
},
"type": "object"
}
},
"properties": {
"logs": {
"description": "Stored OTEL log records (serialized).",
"items": true,
"type": "array"
},
"query": {
"anyOf": [
{
"$ref": "#/definitions/LogsListQuery"
},
{
"type": "null"
}
],
"description": "Echo of the applied filters (present only when storage exists)."
},
"timestamp": {
"format": "int64",
"type": "integer"
},
"total": {
"description": "Total matching logs before pagination.",
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"logs",
"timestamp",
"total"
],
"title": "LogsListResult",
"type": "object"
}
},
{
"description": "Return engine invocation/worker counters and span-derived latency percentiles, plus stored SDK metrics filtered by name/time and optionally aggregated by interval. engine_metrics is always present; sdk_metrics is empty when no metric storage exists.",
"metadata": {},
"name": "engine::metrics::list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"aggregate_interval": {
"description": "Aggregate interval in seconds",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"end_time": {
"description": "End time in Unix timestamp milliseconds",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"metric_name": {
"description": "Filter by metric name",
"type": [
"string",
"null"
]
},
"start_time": {
"description": "Start time in Unix timestamp milliseconds",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"title": "MetricsListInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"EngineMetricsView": {
"properties": {
"invocations": {
"$ref": "#/definitions/InvocationsView"
},
"performance": {
"$ref": "#/definitions/PerformanceView"
},
"workers": {
"$ref": "#/definitions/WorkersView"
}
},
"required": [
"invocations",
"performance",
"workers"
],
"type": "object"
},
"InvocationsView": {
"properties": {
"by_function": {
"additionalProperties": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"description": "Per-function invocation counts.",
"type": "object"
},
"deferred": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"error": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"success": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"total": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"by_function",
"deferred",
"error",
"success",
"total"
],
"type": "object"
},
"MetricsListQuery": {
"properties": {
"aggregate_interval": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"end_time": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"metric_name": {
"type": [
"string",
"null"
]
},
"start_time": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"PerformanceView": {
"properties": {
"avg_duration_ms": {
"format": "double",
"type": "number"
},
"max_duration_ms": {
"format": "double",
"type": "number"
},
"min_duration_ms": {
"format": "double",
"type": "number"
},
"p50_duration_ms": {
"format": "double",
"type": "number"
},
"p95_duration_ms": {
"format": "double",
"type": "number"
},
"p99_duration_ms": {
"format": "double",
"type": "number"
}
},
"required": [
"avg_duration_ms",
"max_duration_ms",
"min_duration_ms",
"p50_duration_ms",
"p95_duration_ms",
"p99_duration_ms"
],
"type": "object"
},
"WorkersView": {
"properties": {
"active": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"deaths": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"spawns": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"active",
"deaths",
"spawns"
],
"type": "object"
}
},
"properties": {
"aggregated_metrics": {
"description": "Time-bucketed aggregates (present only when an aggregate_interval was requested and produced data).",
"items": true,
"type": "array"
},
"engine_metrics": {
"$ref": "#/definitions/EngineMetricsView"
},
"query": {
"anyOf": [
{
"$ref": "#/definitions/MetricsListQuery"
},
{
"type": "null"
}
],
"description": "Echo of the applied query filters (present only when a filter was supplied)."
},
"sdk_metrics": {
"description": "Stored SDK metric points (serialized).",
"items": true,
"type": "array"
},
"timestamp": {
"format": "int64",
"type": "integer"
}
},
"required": [
"aggregated_metrics",
"engine_metrics",
"sdk_metrics",
"timestamp"
],
"title": "MetricsListResult",
"type": "object"
}
},
{
"description": "Return pre-aggregated metric rollups and histograms for a level (0=1m, 1=5m, 2=1h) over a time range (default last hour), optionally by metric name. Falls back to on-the-fly aggregation over metric storage when no rollup storage exists.",
"metadata": {},
"name": "engine::rollups::list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"end_time": {
"description": "End time in Unix timestamp milliseconds",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"level": {
"description": "Rollup level index (0 = 1 min, 1 = 5 min, 2 = 1 hour)",
"format": "uint",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"metric_name": {
"description": "Filter by metric name",
"type": [
"string",
"null"
]
},
"start_time": {
"description": "Start time in Unix timestamp milliseconds",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"title": "RollupsListInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"RollupsListQuery": {
"properties": {
"end_time": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"metric_name": {
"type": [
"string",
"null"
]
},
"start_time": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
}
},
"properties": {
"histogram_rollups": {
"description": "Pre-aggregated histogram rollups (serialized).",
"items": true,
"type": "array"
},
"level": {
"format": "uint",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"message": {
"type": [
"string",
"null"
]
},
"query": {
"anyOf": [
{
"$ref": "#/definitions/RollupsListQuery"
},
{
"type": "null"
}
]
},
"rollups": {
"description": "Pre-aggregated metric rollups (serialized).",
"items": true,
"type": "array"
},
"source": {
"description": "\"on_the_fly\" when computed live because no rollup storage exists.",
"type": [
"string",
"null"
]
},
"timestamp": {
"format": "int64",
"type": "integer"
}
},
"required": [
"histogram_rollups",
"rollups",
"timestamp"
],
"title": "RollupsListResult",
"type": "object"
}
},
{
"description": "Report the active trace sampling config (default ratio, per-operation/service rules, parent_based) and the logs sampling_ratio, read from live config. Defaults to ratio 1.0 with no rules when sampling is unconfigured.",
"metadata": {},
"name": "engine::sampling::rules",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "LogsClearInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"SamplingLogsView": {
"properties": {
"sampling_ratio": {
"format": "double",
"type": "number"
}
},
"required": [
"sampling_ratio"
],
"type": "object"
},
"SamplingRuleView": {
"properties": {
"operation": {
"type": [
"string",
"null"
]
},
"rate": {
"format": "double",
"type": "number"
},
"service": {
"type": [
"string",
"null"
]
}
},
"required": [
"rate"
],
"type": "object"
},
"SamplingTracesView": {
"properties": {
"default_ratio": {
"format": "double",
"type": "number"
},
"parent_based": {
"type": "boolean"
},
"rules": {
"items": {
"$ref": "#/definitions/SamplingRuleView"
},
"type": "array"
}
},
"required": [
"default_ratio",
"parent_based",
"rules"
],
"type": "object"
}
},
"properties": {
"logs": {
"$ref": "#/definitions/SamplingLogsView"
},
"timestamp": {
"format": "int64",
"type": "integer"
},
"traces": {
"$ref": "#/definitions/SamplingTracesView"
}
},
"required": [
"logs",
"timestamp",
"traces"
],
"title": "SamplingRulesResult",
"type": "object"
}
},
{
"description": "Drop every span from the in-memory trace store, returning { success: true }. Requires exporter memory or both, else fails memory_exporter_not_enabled.",
"metadata": {},
"name": "engine::traces::clear",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "TracesClearInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"success": {
"type": "boolean"
}
},
"required": [
"success"
],
"title": "OkResult",
"type": "object"
}
},
{
"description": "Aggregate stored spans by one attribute into groups (trace_ids, span_count, duration, error_count), newest-first, capped at limit (default 100); skips spans lacking the attribute and engine-internal spans unless include_internal. Requires exporter memory or both.",
"metadata": {},
"name": "engine::traces::group_by",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"attribute": {
"description": "Span attribute key to group by. Spans without this attribute are skipped.",
"type": "string"
},
"include_internal": {
"default": null,
"description": "Include engine-internal spans. Defaults to false, matching `traces::list`.",
"type": [
"boolean",
"null"
]
},
"limit": {
"default": null,
"description": "Max groups returned after sorting by `first_seen_ms` descending. Default 100.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"since_ms": {
"default": null,
"description": "Earliest end_time (ms since epoch) to include.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"attribute"
],
"title": "TracesGroupByInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"TraceGroup": {
"properties": {
"duration_ms": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"error_count": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"first_seen_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"last_seen_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"span_count": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"trace_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"value": {
"type": "string"
}
},
"required": [
"duration_ms",
"error_count",
"first_seen_ms",
"last_seen_ms",
"span_count",
"trace_ids",
"value"
],
"type": "object"
}
},
"properties": {
"groups": {
"items": {
"$ref": "#/definitions/TraceGroup"
},
"type": "array"
}
},
"required": [
"groups"
],
"title": "TracesGroupByResult",
"type": "object"
}
},
{
"description": "List root spans of stored traces with filtering (service, name, status, duration, time, attributes), pagination, and sort; hides engine-internal spans unless include_internal. Requires exporter memory or both, else fails memory_exporter_not_enabled.",
"metadata": {},
"name": "engine::traces::list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"attributes": {
"description": "Filter by span attributes (array of [key, value] pairs, AND logic, exact match)",
"items": {
"items": {
"type": "string"
},
"type": "array"
},
"type": [
"array",
"null"
]
},
"end_time": {
"description": "End time in unix timestamp milliseconds (include spans overlapping before this)",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"include_internal": {
"default": null,
"description": "Include internal engine traces (engine.* functions). Defaults to false.",
"type": [
"boolean",
"null"
]
},
"limit": {
"description": "Pagination limit (default: 100)",
"format": "uint",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_duration_ms": {
"description": "Maximum span duration in milliseconds (sub-ms precision)",
"format": "double",
"type": [
"number",
"null"
]
},
"min_duration_ms": {
"description": "Minimum span duration in milliseconds (sub-ms precision)",
"format": "double",
"type": [
"number",
"null"
]
},
"name": {
"description": "Filter by span name (case-insensitive substring match)",
"type": [
"string",
"null"
]
},
"offset": {
"description": "Pagination offset (default: 0)",
"format": "uint",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"search_all_spans": {
"default": null,
"description": "Search across all spans in each trace, not just root spans. When true and a `name` filter is set, traces are matched if ANY span in the trace matches the name filter. Defaults to false.",
"type": [
"boolean",
"null"
]
},
"service_name": {
"description": "Filter by service name (case-insensitive substring match)",
"type": [
"string",
"null"
]
},
"sort_by": {
"description": "Sort field: \"start_time\" | \"duration\" (alias \"duration_ms\") | \"service_name\" | \"name\" (default: \"start_time\"). Unknown values fall back to \"start_time\".",
"type": [
"string",
"null"
]
},
"sort_order": {
"description": "Sort order: \"asc\" | \"desc\" (default: \"asc\")",
"type": [
"string",
"null"
]
},
"start_time": {
"description": "Start time in unix timestamp milliseconds (include spans overlapping after this)",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"status": {
"description": "Filter by status (case-insensitive substring match)",
"type": [
"string",
"null"
]
},
"trace_id": {
"description": "Filter by specific trace ID",
"type": [
"string",
"null"
]
}
},
"title": "TracesListInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"limit": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"offset": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"spans": {
"description": "Stored spans (each a serialized span record).",
"items": true,
"type": "array"
},
"total": {
"description": "Total matching spans before pagination.",
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"limit",
"offset",
"spans",
"total"
],
"title": "TracesListResult",
"type": "object"
}
},
{
"description": "Build the nested span tree for one trace_id as { roots }, pruning no-op trigger wrappers and collapsing configured pass-through spans. Requires exporter memory or both, else fails memory_exporter_not_enabled.",
"metadata": {},
"name": "engine::traces::tree",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"trace_id": {
"description": "Trace ID to build the tree for",
"type": "string"
}
},
"required": [
"trace_id"
],
"title": "TracesTreeInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"roots": {
"description": "Root span-tree nodes (each a serialized, flattened span with nested children).",
"items": true,
"type": "array"
}
},
"required": [
"roots"
],
"title": "TracesTreeResult",
"type": "object"
}
},
{
"description": "Internal: re-apply the iii-observability configuration when the authoritative configuration entry changes.",
"metadata": {
"internal": true
},
"name": "iii-observability::on-config-change",
"request_schema": {},
"response_schema": {}
}
],
"triggers": []
}