iii-observability
v0.19.4-next.1OpenTelemetry-based traces, metrics, logs, alerts, and sampling.
exact versions are immutable; binary and bundle artifacts are digest-pinned.
full markdown
/workers/iii-observability.md?version=0.19.4-next.1. paste it into an llm prompt or pipe it through curl from a worker.install
configuration
- enabled: true
exporter: memory
logs_console_output: true
logs_enabled: true
logs_exporter: memory
logs_max_count: 1000
logs_retention_seconds: 3600
memory_max_spans: 1000000
metrics_enabled: true
metrics_exporter: memory
metrics_max_count: 10000
metrics_retention_seconds: 3600
sampling_ratio: 1
service_name: iii
service_version: ${SERVICE_VERSION:__III_ENGINE_VERSION__}dependencies
readme
iii-observability
Full OpenTelemetry observability for III Engine: distributed tracing, structured logs, performance metrics, alert rules, and trace sampling — all queryable via built-in functions.
Install
iii worker add iii-observabilityResolves from the worker registry at workers.iii.dev.
Skills
Install the iii-observability agent skill for Claude Code, Cursor, and 30+ other agents:
npx skills add iii-hq/iii --full-depth --skill iii-observabilitySample Configuration
- name: iii-observability
config:
enabled: true
service_name: my-service
service_version: 1.0.0
exporter: memory
metrics_enabled: true
logs_enabled: true
memory_max_spans: 1000
sampling_ratio: 1.0
alerts:
- name: high-error-rate
metric: iii.invocations.error
threshold: 10
operator: ">"
window_seconds: 60
action:
type: logConfiguration
| Field | Type | Description |
|---|---|---|
enabled |
boolean | Whether OpenTelemetry tracing export is enabled. Defaults to false. Env: OTEL_ENABLED. |
service_name |
string | Service name in traces and metrics. Defaults to "iii". Env: OTEL_SERVICE_NAME. |
service_version |
string | Service version (service.version OTEL attribute). Env: SERVICE_VERSION. |
service_namespace |
string | Service namespace (service.namespace OTEL attribute). Env: SERVICE_NAMESPACE. |
exporter |
string | Trace exporter: memory, otlp, or both. Use both when traces should remain queryable in iii while also being exported. Defaults to otlp. Env: OTEL_EXPORTER_TYPE. |
endpoint |
string | OTLP collector base endpoint. Defaults to "http://localhost:4317". Env: OTEL_EXPORTER_OTLP_ENDPOINT. |
sampling_ratio |
number | Global trace sampling ratio (0.0–1.0). Defaults to 1.0. Env: OTEL_TRACES_SAMPLER_ARG. |
memory_max_spans |
number | Max spans to keep in memory. Defaults to 1000. Env: OTEL_MEMORY_MAX_SPANS. |
metrics_enabled |
boolean | Whether metrics collection is enabled. Defaults to false. Env: OTEL_METRICS_ENABLED. |
metrics_exporter |
string | Metrics exporter: memory or otlp. Defaults to memory. Env: OTEL_METRICS_EXPORTER. |
metrics_retention_seconds |
number | How long to retain metrics in memory. Defaults to 3600. Env: OTEL_METRICS_RETENTION_SECONDS. |
metrics_max_count |
number | Max metric data points in memory. Defaults to 10000. Env: OTEL_METRICS_MAX_COUNT. |
logs_enabled |
boolean | Whether structured log storage is enabled. |
logs_exporter |
string | Logs exporter: memory, otlp, or both. Use both when logs should remain queryable in iii while also being exported. Defaults to memory. Env: OTEL_LOGS_EXPORTER. |
logs_max_count |
number | Max log entries in memory. Defaults to 1000. |
logs_retention_seconds |
number | How long to retain logs in memory. Defaults to 3600. |
logs_sampling_ratio |
number | Fraction of logs to retain (0.0–1.0). Defaults to 1.0. |
logs_console_output |
boolean | Print ingested logs to the console. Defaults to true. |
level |
string | Minimum log level: trace, debug, info, warn, error. Defaults to info. |
format |
string | Log output format: default or json. Defaults to default. |
alerts |
AlertRule[] | Alert rules evaluated against metrics. |
OTLP Transport
Traces and metrics export over OTLP/gRPC by default. https:// endpoints use TLS with system roots,
while http:// endpoints use cleartext transport.
To export traces and metrics with OTLP/HTTP protobuf instead of gRPC, set the standard OpenTelemetry protocol environment variable before starting the engine:
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobufSignal-specific protocol variables override the global value:
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_METRICS_PROTOCOL=http/protobufWhen HTTP/protobuf is selected, iii treats endpoint as the collector base URL and appends the
signal path when needed:
- traces:
/v1/traces - metrics:
/v1/metrics
The logs exporter sends OTLP logs over HTTP and posts to /v1/logs.
Collectors that require authentication or routing headers can use the standard OTLP headers environment variables:
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer $OTLP_TOKEN"Use signal-specific headers when one signal needs different values:
OTEL_EXPORTER_OTLP_TRACES_HEADERSOTEL_EXPORTER_OTLP_METRICS_HEADERSOTEL_EXPORTER_OTLP_LOGS_HEADERS
The logs exporter reads OTEL_EXPORTER_OTLP_LOGS_HEADERS first and falls back to
OTEL_EXPORTER_OTLP_HEADERS. Keep tokens in environment variables or a secret manager; do not
commit them to config files.
To keep the iii console useful while exporting to an external collector, use exporter: both for
traces and logs_exporter: both for logs.
Alert Rule Fields
| Field | Type | Description |
|---|---|---|
name |
string | Required. Unique alert rule name. |
metric |
string | Required. Metric name to monitor (e.g., iii.invocations.error). |
threshold |
number | Required. Threshold value. |
operator |
string | Comparison operator: >, >=, <, <=, ==, !=. Defaults to >. |
window_seconds |
number | Time window in seconds for metric evaluation. Defaults to 60. |
cooldown_seconds |
number | Minimum interval between alert fires. Defaults to 60. |
enabled |
boolean | Whether the alert rule is active. Defaults to true. |
action |
AlertAction | { "type": "log" }, { "type": "webhook", "url": "..." }, or { "type": "function", "path": "..." }. |
Advanced Sampling
sampling:
default: 1.0
parent_based: true
rules:
- operation: "api.*"
rate: 0.1
rate_limit:
max_traces_per_second: 100Functions
Logging
| Function | Description |
|---|---|
engine::log::info |
Log an informational message. |
engine::log::warn |
Log a warning message. |
engine::log::error |
Log an error message. |
engine::log::debug |
Log a debug message. |
engine::log::trace |
Log a trace-level message. |
All logging functions accept: message (string, required), data (object), trace_id (string),
span_id (string), service_name (string).
Logs API
| Function | Description |
|---|---|
engine::logs::list |
Query stored log entries. Filters: start_time, end_time, trace_id, span_id, severity_min, severity_text, offset, limit. |
engine::logs::clear |
Clear all stored log entries from memory. |
Traces API
| Function | Description |
|---|---|
engine::traces::list |
List stored spans. Filters: trace_id, service_name, name, status, min_duration_ms, max_duration_ms, start_time, end_time, sort_by, sort_order, attributes, include_internal, offset, limit. |
engine::traces::tree |
Retrieve a trace as a hierarchical span tree. Parameters: trace_id (required). |
engine::traces::clear |
Clear all stored trace spans from memory. |
Metrics API
| Function | Description |
|---|---|
engine::metrics::list |
List metrics with aggregated statistics. Returns engine counters (invocations, workers, performance), SDK metrics, and optional time-bucketed aggregations. |
engine::rollups::list |
List metric rollup aggregations (1-minute, 5-minute, 1-hour windows). |
Other APIs
| Function | Description |
|---|---|
engine::baggage::get |
Get a baggage value from the current trace context. |
engine::baggage::set |
Set a baggage value in the current trace context. |
engine::baggage::get_all |
Get all baggage key-value pairs. |
engine::sampling::rules |
List all active sampling rules. |
engine::health::check |
Check engine health status. Returns status, components, timestamp, version. |
engine::alerts::list |
List all configured alert rules and their current state. |
engine::alerts::evaluate |
Manually trigger evaluation of all alert rules. |
Trigger Type: log
Register a function to react to log entries as they are produced.
| Config Field | Type | Description |
|---|---|---|
level |
string | Log level to subscribe to: info, warn, error, debug, or trace. When omitted, fires for all levels. |
Sample Code
const fn = iii.registerFunction("monitoring::onError", async (logEntry) => {
await sendAlert({
message: logEntry.body,
severity: logEntry.severity_text,
traceId: logEntry.trace_id,
});
return {};
});
iii.registerTrigger({
type: "log",
function_id: fn.id,
config: { level: "error" },
});Log entry payload fields: timestamp_unix_nano, observed_timestamp_unix_nano, severity_number,
severity_text, body, attributes, trace_id, span_id, resource, service_name,
instrumentation_scope_name, instrumentation_scope_version.
Trigger Type: trace
Register a function to react to span activity in the in-memory trace store, so any client — a worker
or the web console — can refresh reactively instead of polling. The trigger is a coalesced "traces
changed" tick, not a per-span feed: span activity is debounced (~300ms) and the handler receives
the distinct affected trace ids for the window. Re-read details via engine::traces::list /
engine::traces::tree. Requires the memory exporter (exporter: memory or both); with the
OTLP-only exporter there is no in-memory store and trace triggers stay dormant.
Engine-internal spans and the trigger's own delivery spans are excluded from firing it —
delivering a trigger via engine.call is itself instrumented as a span, so without this exclusion
the trigger would re-fire on its own output (an unbounded feedback loop). This is why a span trigger
differs from the log trigger, whose delivery produces spans, not logs.
| Config Field | Type | Description |
|---|---|---|
service_name |
string | Only fire for activity from this service. When omitted, fires for any service. Compared case-insensitively. |
status |
string | Only fire when a span with this status (ok, error, or unset) landed in the window. When omitted, fires for any status. Compared case-insensitively. |
Both filters are ANDed; omit both to fire on any span activity.
Sample Code
const fn = iii.registerFunction("devtools::onTracesChanged", async ({ trace_ids }) => {
// A "refetch soon" beat — re-read the traces you care about.
await refreshTraceViews(trace_ids);
return {};
});
iii.registerTrigger({
type: "trace",
function_id: fn.id,
config: {}, // or { status: 'error' }
});Handler payload: { "trace_ids": string[] } — the distinct trace ids with span activity in the
coalesced window.
api reference (json)
{
"functions": [
{
"description": "Manually trigger alert evaluation",
"metadata": {},
"name": "engine::alerts::evaluate",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "AlertsEvaluateInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "List current alert states",
"metadata": {},
"name": "engine::alerts::list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "AlertsListInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "Get a baggage item value from the current context",
"metadata": {},
"name": "engine::baggage::get",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input for baggage.get function",
"properties": {
"key": {
"description": "The baggage key to retrieve",
"type": "string"
}
},
"required": [
"key"
],
"title": "BaggageGetInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "Get all baggage items from the current context",
"metadata": {},
"name": "engine::baggage::get_all",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input for baggage.getAll function (empty)",
"title": "BaggageGetAllInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "Set a baggage item value (returns new context, does not modify global)",
"metadata": {},
"name": "engine::baggage::set",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input for baggage.set function",
"properties": {
"key": {
"description": "The baggage key to set",
"type": "string"
},
"value": {
"description": "The baggage value to set",
"type": "string"
}
},
"required": [
"key",
"value"
],
"title": "BaggageSetInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "Check system health status",
"metadata": {},
"name": "engine::health::check",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "HealthCheckInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "Log a debug message using OTEL",
"metadata": {},
"name": "engine::log::debug",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input for OTEL log functions (log.info, log.warn, log.error)",
"properties": {
"data": {
"description": "Additional structured data/attributes"
},
"message": {
"description": "The log message",
"type": "string"
},
"service_name": {
"description": "Service name (defaults to function name if not provided)",
"type": [
"string",
"null"
]
},
"span_id": {
"description": "Optional span ID for correlation",
"type": [
"string",
"null"
]
},
"trace_id": {
"description": "Optional trace ID for correlation",
"type": [
"string",
"null"
]
}
},
"required": [
"message"
],
"title": "OtelLogInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "Log an error message using OTEL",
"metadata": {},
"name": "engine::log::error",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input for OTEL log functions (log.info, log.warn, log.error)",
"properties": {
"data": {
"description": "Additional structured data/attributes"
},
"message": {
"description": "The log message",
"type": "string"
},
"service_name": {
"description": "Service name (defaults to function name if not provided)",
"type": [
"string",
"null"
]
},
"span_id": {
"description": "Optional span ID for correlation",
"type": [
"string",
"null"
]
},
"trace_id": {
"description": "Optional trace ID for correlation",
"type": [
"string",
"null"
]
}
},
"required": [
"message"
],
"title": "OtelLogInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "Log an info message using OTEL",
"metadata": {},
"name": "engine::log::info",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input for OTEL log functions (log.info, log.warn, log.error)",
"properties": {
"data": {
"description": "Additional structured data/attributes"
},
"message": {
"description": "The log message",
"type": "string"
},
"service_name": {
"description": "Service name (defaults to function name if not provided)",
"type": [
"string",
"null"
]
},
"span_id": {
"description": "Optional span ID for correlation",
"type": [
"string",
"null"
]
},
"trace_id": {
"description": "Optional trace ID for correlation",
"type": [
"string",
"null"
]
}
},
"required": [
"message"
],
"title": "OtelLogInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "Log a trace-level message using OTEL",
"metadata": {},
"name": "engine::log::trace",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input for OTEL log functions (log.info, log.warn, log.error)",
"properties": {
"data": {
"description": "Additional structured data/attributes"
},
"message": {
"description": "The log message",
"type": "string"
},
"service_name": {
"description": "Service name (defaults to function name if not provided)",
"type": [
"string",
"null"
]
},
"span_id": {
"description": "Optional span ID for correlation",
"type": [
"string",
"null"
]
},
"trace_id": {
"description": "Optional trace ID for correlation",
"type": [
"string",
"null"
]
}
},
"required": [
"message"
],
"title": "OtelLogInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "Log a warning message using OTEL",
"metadata": {},
"name": "engine::log::warn",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input for OTEL log functions (log.info, log.warn, log.error)",
"properties": {
"data": {
"description": "Additional structured data/attributes"
},
"message": {
"description": "The log message",
"type": "string"
},
"service_name": {
"description": "Service name (defaults to function name if not provided)",
"type": [
"string",
"null"
]
},
"span_id": {
"description": "Optional span ID for correlation",
"type": [
"string",
"null"
]
},
"trace_id": {
"description": "Optional trace ID for correlation",
"type": [
"string",
"null"
]
}
},
"required": [
"message"
],
"title": "OtelLogInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "Clear all stored OTEL logs",
"metadata": {},
"name": "engine::logs::clear",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "LogsClearInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "List stored OTEL logs",
"metadata": {},
"name": "engine::logs::list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"end_time": {
"description": "End time in Unix timestamp milliseconds",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"limit": {
"description": "Maximum number of logs to return",
"format": "uint",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"offset": {
"description": "Pagination offset (default: 0)",
"format": "uint",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"severity_min": {
"description": "Minimum severity number (1-24, higher = more severe)",
"format": "int32",
"type": [
"integer",
"null"
]
},
"severity_text": {
"description": "Filter by severity text (e.g., \"ERROR\", \"WARN\", \"INFO\")",
"type": [
"string",
"null"
]
},
"span_id": {
"description": "Filter by span ID",
"type": [
"string",
"null"
]
},
"start_time": {
"description": "Start time in Unix timestamp milliseconds",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"trace_id": {
"description": "Filter by trace ID",
"type": [
"string",
"null"
]
}
},
"title": "LogsListInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "List current metrics values",
"metadata": {},
"name": "engine::metrics::list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"aggregate_interval": {
"description": "Aggregate interval in seconds",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"end_time": {
"description": "End time in Unix timestamp milliseconds",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"metric_name": {
"description": "Filter by metric name",
"type": [
"string",
"null"
]
},
"start_time": {
"description": "Start time in Unix timestamp milliseconds",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"title": "MetricsListInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "Get pre-aggregated metrics rollups",
"metadata": {},
"name": "engine::rollups::list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"end_time": {
"description": "End time in Unix timestamp milliseconds",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"level": {
"description": "Rollup level index (0 = 1 min, 1 = 5 min, 2 = 1 hour)",
"format": "uint",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"metric_name": {
"description": "Filter by metric name",
"type": [
"string",
"null"
]
},
"start_time": {
"description": "Start time in Unix timestamp milliseconds",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"title": "RollupsListInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "Get active sampling rules configuration",
"metadata": {},
"name": "engine::sampling::rules",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "LogsClearInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "Clear all stored traces (only available when exporter is 'memory' or 'both')",
"metadata": {},
"name": "engine::traces::clear",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "TracesClearInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "Group stored spans by an attribute value (only available when exporter is 'memory' or 'both')",
"metadata": {},
"name": "engine::traces::group_by",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"attribute": {
"description": "Span attribute key to group by. Spans without this attribute are skipped.",
"type": "string"
},
"include_internal": {
"default": null,
"description": "Include engine-internal spans. Defaults to false, matching `traces::list`.",
"type": [
"boolean",
"null"
]
},
"limit": {
"default": null,
"description": "Max groups returned after sorting by `first_seen_ms` descending. Default 100.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"since_ms": {
"default": null,
"description": "Earliest end_time (ms since epoch) to include.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"attribute"
],
"title": "TracesGroupByInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "List stored traces (only available when exporter is 'memory' or 'both')",
"metadata": {},
"name": "engine::traces::list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"attributes": {
"description": "Filter by span attributes (array of [key, value] pairs, AND logic, exact match)",
"items": {
"items": {
"type": "string"
},
"type": "array"
},
"type": [
"array",
"null"
]
},
"end_time": {
"description": "End time in unix timestamp milliseconds (include spans overlapping before this)",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"include_internal": {
"default": null,
"description": "Include internal engine traces (engine.* functions). Defaults to false.",
"type": [
"boolean",
"null"
]
},
"limit": {
"description": "Pagination limit (default: 100)",
"format": "uint",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_duration_ms": {
"description": "Maximum span duration in milliseconds (sub-ms precision)",
"format": "double",
"type": [
"number",
"null"
]
},
"min_duration_ms": {
"description": "Minimum span duration in milliseconds (sub-ms precision)",
"format": "double",
"type": [
"number",
"null"
]
},
"name": {
"description": "Filter by span name (case-insensitive substring match)",
"type": [
"string",
"null"
]
},
"offset": {
"description": "Pagination offset (default: 0)",
"format": "uint",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"search_all_spans": {
"default": null,
"description": "Search across all spans in each trace, not just root spans. When true and a `name` filter is set, traces are matched if ANY span in the trace matches the name filter. Defaults to false.",
"type": [
"boolean",
"null"
]
},
"service_name": {
"description": "Filter by service name (case-insensitive substring match)",
"type": [
"string",
"null"
]
},
"sort_by": {
"description": "Sort field: \"start_time\" | \"duration\" (alias \"duration_ms\") | \"service_name\" | \"name\" (default: \"start_time\"). Unknown values fall back to \"start_time\".",
"type": [
"string",
"null"
]
},
"sort_order": {
"description": "Sort order: \"asc\" | \"desc\" (default: \"asc\")",
"type": [
"string",
"null"
]
},
"start_time": {
"description": "Start time in unix timestamp milliseconds (include spans overlapping after this)",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"status": {
"description": "Filter by status (case-insensitive substring match)",
"type": [
"string",
"null"
]
},
"trace_id": {
"description": "Filter by specific trace ID",
"type": [
"string",
"null"
]
}
},
"title": "TracesListInput",
"type": "object"
},
"response_schema": {}
},
{
"description": "Get trace tree with nested children (only available when exporter is 'memory' or 'both')",
"metadata": {},
"name": "engine::traces::tree",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"trace_id": {
"description": "Trace ID to build the tree for",
"type": "string"
}
},
"required": [
"trace_id"
],
"title": "TracesTreeInput",
"type": "object"
},
"response_schema": {}
}
],
"triggers": []
}