skip to content
$worker

iii-observability

v0.20.0-alpha.1

OpenTelemetry-based traces, metrics, logs, alerts, and sampling.

iiiverified
8,665 installs64 in 7d0 today
install
$iii worker add iii-observability@0.20.0-alpha.1

exact versions are immutable; binary and bundle artifacts are digest-pinned.

engine module
baked into the iii engine; no separate install required.
agent-ready brief for v0.20.0-alpha.1
install + config + dependencies + readme + api reference, all in one place. fetch as agent-context.md for an llm to consume.
the same content rendered as discrete blocks below is exposed as a single markdown document at /workers/iii-observability.md?version=0.20.0-alpha.1. paste it into an llm prompt or pipe it through curl from a worker.

install

install
$iii worker add iii-observability@0.20.0-alpha.1

configuration

iii-config.yaml
- enabled: true
  exporter: memory
  logs_console_output: true
  logs_enabled: true
  logs_exporter: memory
  logs_max_count: 1000
  logs_retention_seconds: 3600
  memory_max_spans: 1000000
  metrics_enabled: true
  metrics_exporter: memory
  metrics_max_count: 10000
  metrics_retention_seconds: 3600
  sampling_ratio: 1
  service_name: iii
  service_version: ${SERVICE_VERSION:__III_ENGINE_VERSION__}

dependencies

no dependencies for v0.20.0-alpha.1

readme

README.md

iii-observability

Full OpenTelemetry observability for III Engine: distributed tracing, structured logs, performance metrics, alert rules, and trace sampling — all queryable via built-in functions.

Install

iii worker add iii-observability

Resolves from the worker registry at workers.iii.dev.

Skills

Install the iii-observability agent skill for Claude Code, Cursor, and 30+ other agents:

npx skills add iii-hq/iii --full-depth --skill iii-observability

Sample Configuration

- name: iii-observability
  config:
    enabled: true
    service_name: my-service
    service_version: 1.0.0
    exporter: memory
    metrics_enabled: true
    logs_enabled: true
    memory_max_spans: 1000
    sampling_ratio: 1.0
    alerts:
      - name: high-error-rate
        metric: iii.invocations.error
        threshold: 10
        operator: ">"
        window_seconds: 60
        action:
          type: log

Configure

The full configuration surface is registered with the builtin configuration worker under the id iii-observability. The stored entry is the runtime source of truth; the config.yaml block is seed-only — it populates the entry on the very first boot and is ignored afterwards. To change a setting after first boot, edit the entry (console, or configuration::set { "id": "iii-observability", "value": { ... } }); editing config.yaml alone has no effect anymore.

With the default file-backed adapter the entry persists at ./config/iii-observability.yaml and is read again at every engine start — before logging/tracing init — so even restart-tier fields edited at runtime apply on the next start. ${VAR:default} placeholders work in string fields and are expanded on read.

Values are validated against the JSON schema at configuration::set time (unknown fields rejected, ratios bounded to 0..=1, counts ≥ 1). Two caveats:

  • Alert operator symbols (>, <, ...) are accepted in config.yaml only; remote edits must use the canonical names the schema advertises (greaterthan, lessthan, ...).
  • After a schema tightening, a previously-stored out-of-range value makes the boot-time schema refresh fail with SCHEMA_INVALID (warn-and-continue); reads still work and out-of-range values are clamped on read.

Hot Reload

configuration:updated events are applied per field tier:

Tier Fields Effect
Live logs_console_output, logs_sampling_ratio, logs_enabled (ingest gate), enabled (ingest gate) Immediate — read per use
Limits memory_max_spans, logs_max_count, metrics_max_count, metrics_retention_seconds Immediate — enforced on the next insert / 60s sweep
Swap sampling_ratio, sampling.*, alerts, collapse_spans, level Immediate — compiled artifact rebuilt and swapped (alert states of surviving rules keep cooldown/firing continuity)
Task rebuild logs_exporter, logs_batch_size, logs_flush_interval_ms, logs_retention_seconds, and logs_enabled on a false→true transition The background task restarts with the new settings. A logs_enabled false→true toggle revives the log store and respawns the log-trigger subscriber, OTLP logs exporter, and retention task — so the log trigger fan-out and OTLP log export reactivate without an engine restart
Restart-only exporter, endpoint (trace and logs exporters), service_name/service_version/service_namespace (trace resource and logs exporter identity), format, metrics_enabled, metrics_exporter, enabled (pipeline construction) Logged as a warning; applied at the next engine start via the persisted entry. endpoint/service_name/service_version are restart-tier for all signals so logs and traces always move to the new collector/identity together, never split mid-edit

Known limitation: an engine config-file reload that destroys and recreates this worker shuts down the OTLP trace/metric providers without rebuilding them (they are process-global set-once state) — OTLP export then requires an engine restart. Memory-backed stores are unaffected.

Configuration

Field Type Description
enabled boolean Whether OpenTelemetry tracing export is enabled. Defaults to false. Env: OTEL_ENABLED.
service_name string Service name in traces and metrics. Defaults to "iii". Env: OTEL_SERVICE_NAME.
service_version string Service version (service.version OTEL attribute). Env: SERVICE_VERSION.
service_namespace string Service namespace (service.namespace OTEL attribute). Env: SERVICE_NAMESPACE.
exporter string Trace exporter: memory, otlp, or both. Use both when traces should remain queryable in iii while also being exported. Defaults to otlp. Env: OTEL_EXPORTER_TYPE.
endpoint string OTLP collector base endpoint. Defaults to "http://localhost:4317". Env: OTEL_EXPORTER_OTLP_ENDPOINT.
sampling_ratio number Global trace sampling ratio (0.01.0). Defaults to 1.0. Env: OTEL_TRACES_SAMPLER_ARG.
memory_max_spans number Max spans to keep in memory. Defaults to 1000. Env: OTEL_MEMORY_MAX_SPANS.
metrics_enabled boolean Whether metrics collection is enabled. Defaults to false. Env: OTEL_METRICS_ENABLED.
metrics_exporter string Metrics exporter: memory or otlp. Defaults to memory. Env: OTEL_METRICS_EXPORTER.
metrics_retention_seconds number How long to retain metrics in memory. Defaults to 3600. Env: OTEL_METRICS_RETENTION_SECONDS.
metrics_max_count number Max metric data points in memory. Defaults to 10000. Env: OTEL_METRICS_MAX_COUNT.
logs_enabled boolean Whether structured log storage is enabled.
logs_exporter string Logs exporter: memory, otlp, or both. Use both when logs should remain queryable in iii while also being exported. Defaults to memory. Env: OTEL_LOGS_EXPORTER.
logs_max_count number Max log entries in memory. Defaults to 1000.
logs_retention_seconds number How long to retain logs in memory. Defaults to 3600.
logs_sampling_ratio number Fraction of logs to retain (0.01.0). Defaults to 1.0.
logs_console_output boolean Print ingested logs to the console. Defaults to true.
level string Minimum log level: trace, debug, info, warn, error. Defaults to info.
format string Log output format: default or json. Defaults to default.
alerts AlertRule[] Alert rules evaluated against metrics.

OTLP Transport

Traces and metrics export over OTLP/gRPC by default. https:// endpoints use TLS with system roots, while http:// endpoints use cleartext transport.

To export traces and metrics with OTLP/HTTP protobuf instead of gRPC, set the standard OpenTelemetry protocol environment variable before starting the engine:

export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

Signal-specific protocol variables override the global value:

export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_METRICS_PROTOCOL=http/protobuf

When HTTP/protobuf is selected, iii treats endpoint as the collector base URL and appends the signal path when needed:

  • traces: /v1/traces
  • metrics: /v1/metrics

The logs exporter sends OTLP logs over HTTP and posts to /v1/logs.

Collectors that require authentication or routing headers can use the standard OTLP headers environment variables:

export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer $OTLP_TOKEN"

Use signal-specific headers when one signal needs different values:

  • OTEL_EXPORTER_OTLP_TRACES_HEADERS
  • OTEL_EXPORTER_OTLP_METRICS_HEADERS
  • OTEL_EXPORTER_OTLP_LOGS_HEADERS

The logs exporter reads OTEL_EXPORTER_OTLP_LOGS_HEADERS first and falls back to OTEL_EXPORTER_OTLP_HEADERS. Keep tokens in environment variables or a secret manager; do not commit them to config files.

To keep the iii console useful while exporting to an external collector, use exporter: both for traces and logs_exporter: both for logs.

Alert Rule Fields

Field Type Description
name string Required. Unique alert rule name.
metric string Required. Metric name to monitor (e.g., iii.invocations.error).
threshold number Required. Threshold value.
operator string Comparison operator: >, >=, <, <=, ==, !=. Defaults to >.
window_seconds number Time window in seconds for metric evaluation. Defaults to 60.
cooldown_seconds number Minimum interval between alert fires. Defaults to 60.
enabled boolean Whether the alert rule is active. Defaults to true.
action AlertAction { "type": "log" }, { "type": "webhook", "url": "..." }, or { "type": "function", "path": "..." }.

Advanced Sampling

sampling:
  default: 1.0
  parent_based: true
  rules:
    - operation: "api.*"
      rate: 0.1
  rate_limit:
    max_traces_per_second: 100

Functions

Logging

Function Description
engine::log::info Log an informational message.
engine::log::warn Log a warning message.
engine::log::error Log an error message.
engine::log::debug Log a debug message.
engine::log::trace Log a trace-level message.

All logging functions accept: message (string, required), data (object), trace_id (string), span_id (string), service_name (string).

Logs API

Function Description
engine::logs::list Query stored log entries. Filters: start_time, end_time, trace_id, span_id, severity_min, severity_text, offset, limit.
engine::logs::clear Clear all stored log entries from memory.

Traces API

Function Description
engine::traces::list List stored spans. Filters: trace_id, service_name, name, status, min_duration_ms, max_duration_ms, start_time, end_time, sort_by, sort_order, attributes, include_internal, offset, limit.
engine::traces::tree Retrieve a trace as a hierarchical span tree. Parameters: trace_id (required).
engine::traces::clear Clear all stored trace spans from memory.

Metrics API

Function Description
engine::metrics::list List metrics with aggregated statistics. Returns engine counters (invocations, workers, performance), SDK metrics, and optional time-bucketed aggregations.
engine::rollups::list List metric rollup aggregations (1-minute, 5-minute, 1-hour windows).

Other APIs

Function Description
engine::baggage::get Get a baggage value from the current trace context.
engine::baggage::set Set a baggage value in the current trace context.
engine::baggage::get_all Get all baggage key-value pairs.
engine::sampling::rules List all active sampling rules.
engine::health::check Check engine health status. Returns status, components, timestamp, version.
engine::alerts::list List all configured alert rules and their current state.
engine::alerts::evaluate Manually trigger evaluation of all alert rules.

Trigger Type: log

Register a function to react to log entries as they are produced.

Config Field Type Description
level string Log level to subscribe to: info, warn, error, debug, or trace. When omitted, fires for all levels.

Sample Code

const fn = iii.registerFunction("monitoring::onError", async (logEntry) => {
  await sendAlert({
    message: logEntry.body,
    severity: logEntry.severity_text,
    traceId: logEntry.trace_id,
  });
  return {};
});

iii.registerTrigger({
  type: "log",
  function_id: fn.id,
  config: { level: "error" },
});

Log entry payload fields: timestamp_unix_nano, observed_timestamp_unix_nano, severity_number, severity_text, body, attributes, trace_id, span_id, resource, service_name, instrumentation_scope_name, instrumentation_scope_version.

Trigger Type: trace

Register a function to react to span activity in the in-memory trace store, so any client — a worker or the web console — can refresh reactively instead of polling. The trigger is a coalesced "traces changed" tick, not a per-span feed: span activity is debounced (~300ms) and the handler receives the distinct affected trace ids for the window. Re-read details via engine::traces::list / engine::traces::tree. Requires the memory exporter (exporter: memory or both); with the OTLP-only exporter there is no in-memory store and trace triggers stay dormant.

Engine-internal spans and the trigger's own delivery spans are excluded from firing it — delivering a trigger via engine.call is itself instrumented as a span, so without this exclusion the trigger would re-fire on its own output (an unbounded feedback loop). This is why a span trigger differs from the log trigger, whose delivery produces spans, not logs.

Config Field Type Description
service_name string Only fire for activity from this service. When omitted, fires for any service. Compared case-insensitively.
status string Only fire when a span with this status (ok, error, or unset) landed in the window. When omitted, fires for any status. Compared case-insensitively.

Both filters are ANDed; omit both to fire on any span activity.

Sample Code

const fn = iii.registerFunction("devtools::onTracesChanged", async ({ trace_ids }) => {
  // A "refetch soon" beat — re-read the traces you care about.
  await refreshTraceViews(trace_ids);
  return {};
});

iii.registerTrigger({
  type: "trace",
  function_id: fn.id,
  config: {}, // or { status: 'error' }
});

Handler payload: { "trace_ids": string[] } — the distinct trace ids with span activity in the coalesced window.

api reference (json)

agent-api-reference.json
{
  "functions": [
    {
      "description": "Force an immediate alert-rule evaluation against current metrics and return any triggered_alerts, bypassing the periodic tick. Returns evaluated:false when the alert manager is not initialized; produces nothing without configured rules.",
      "metadata": {},
      "name": "engine::alerts::evaluate",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "AlertsEvaluateInput",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "evaluated": {
            "type": "boolean"
          },
          "message": {
            "type": [
              "string",
              "null"
            ]
          },
          "timestamp": {
            "format": "int64",
            "type": "integer"
          },
          "triggered_alerts": {
            "description": "Alerts triggered by this evaluation pass (present when the manager is initialized).",
            "items": true,
            "type": [
              "array",
              "null"
            ]
          }
        },
        "required": [
          "evaluated",
          "timestamp"
        ],
        "title": "AlertsEvaluateResult",
        "type": "object"
      }
    },
    {
      "description": "List configured alert rules with their current evaluated states and a firing_count. Returns empty when no alert rules are configured or the alert manager is not initialized.",
      "metadata": {},
      "name": "engine::alerts::list",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "AlertsListInput",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "AlertAction": {
            "description": "Alert action type",
            "oneOf": [
              {
                "description": "Log the alert (default)",
                "properties": {
                  "type": {
                    "enum": [
                      "log"
                    ],
                    "type": "string"
                  }
                },
                "required": [
                  "type"
                ],
                "type": "object"
              },
              {
                "description": "Send webhook notification to the specified URL",
                "properties": {
                  "type": {
                    "enum": [
                      "webhook"
                    ],
                    "type": "string"
                  },
                  "url": {
                    "description": "The webhook URL to send the alert to",
                    "type": "string"
                  }
                },
                "required": [
                  "type",
                  "url"
                ],
                "type": "object"
              },
              {
                "description": "Invoke a function at the specified path",
                "properties": {
                  "path": {
                    "description": "The function path to invoke",
                    "type": "string"
                  },
                  "type": {
                    "enum": [
                      "function"
                    ],
                    "type": "string"
                  }
                },
                "required": [
                  "path",
                  "type"
                ],
                "type": "object"
              }
            ]
          },
          "AlertOperator": {
            "description": "Comparison operator for alert thresholds NOTE: the symbol aliases (`>`, `>=`, ...) are serde-only conveniences for config.yaml. The generated JSON Schema advertises the canonical lowercase names, and `configuration::set` validates against the schema — so remote edits must use the canonical names.",
            "enum": [
              "greaterthan",
              "greaterthanorequal",
              "lessthan",
              "lessthanorequal",
              "equal",
              "notequal"
            ],
            "type": "string"
          },
          "AlertRule": {
            "additionalProperties": false,
            "description": "Single alert rule configuration",
            "properties": {
              "action": {
                "allOf": [
                  {
                    "$ref": "#/definitions/AlertAction"
                  }
                ],
                "default": {
                  "type": "log"
                },
                "description": "Action to take when alert triggers (Log, Webhook { url }, Function { path })"
              },
              "cooldown_seconds": {
                "default": 60,
                "description": "Minimum interval between alert triggers in seconds (default: 60)",
                "format": "uint64",
                "minimum": 1,
                "type": "integer"
              },
              "enabled": {
                "default": true,
                "description": "Whether the alert is enabled (default: true)",
                "type": "boolean"
              },
              "metric": {
                "description": "Metric name to monitor (e.g., \"iii.invocations.error\")",
                "type": "string"
              },
              "name": {
                "description": "Name of the alert (for identification)",
                "type": "string"
              },
              "operator": {
                "allOf": [
                  {
                    "$ref": "#/definitions/AlertOperator"
                  }
                ],
                "default": "greaterthan",
                "description": "Comparison operator (>, >=, <, <=, ==, !=)"
              },
              "threshold": {
                "description": "Threshold value for the alert",
                "format": "double",
                "type": "number"
              },
              "window_seconds": {
                "default": 60,
                "description": "Time window in seconds to evaluate the metric (default: 60)",
                "format": "uint64",
                "minimum": 1,
                "type": "integer"
              }
            },
            "required": [
              "metric",
              "name",
              "threshold"
            ],
            "type": "object"
          }
        },
        "properties": {
          "alerts": {
            "description": "Current evaluated alert states (serialized).",
            "items": true,
            "type": "array"
          },
          "firing_count": {
            "format": "uint",
            "minimum": 0,
            "type": "integer"
          },
          "message": {
            "type": [
              "string",
              "null"
            ]
          },
          "rules": {
            "description": "Configured alert rules (present when the alert manager is initialized).",
            "items": {
              "$ref": "#/definitions/AlertRule"
            },
            "type": [
              "array",
              "null"
            ]
          },
          "timestamp": {
            "format": "int64",
            "type": "integer"
          }
        },
        "required": [
          "alerts",
          "firing_count",
          "timestamp"
        ],
        "title": "AlertsListResult",
        "type": "object"
      }
    },
    {
      "description": "Read one baggage entry by key from the current OTEL context, returning { value } (null if unset). Diagnostic only: reads ambient process context, not per-invocation baggage.",
      "metadata": {},
      "name": "engine::baggage::get",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for baggage.get function",
        "properties": {
          "key": {
            "description": "The baggage key to retrieve",
            "type": "string"
          }
        },
        "required": [
          "key"
        ],
        "title": "BaggageGetInput",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "value": {
            "type": [
              "string",
              "null"
            ]
          }
        },
        "title": "BaggageGetResult",
        "type": "object"
      }
    },
    {
      "description": "Read all baggage entries from the current OTEL context as a { baggage } map. Diagnostic only: reflects ambient process context, not per-invocation baggage.",
      "metadata": {},
      "name": "engine::baggage::get_all",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for baggage.getAll function (empty)",
        "title": "BaggageGetAllInput",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "baggage": {
            "additionalProperties": {
              "type": "string"
            },
            "type": "object"
          }
        },
        "required": [
          "baggage"
        ],
        "title": "BaggageGetAllResult",
        "type": "object"
      }
    },
    {
      "description": "Set a baggage key/value on a fresh OTEL context for verification only; the new context is not propagated to the caller or global state. Use SDK-level headers for real propagation.",
      "metadata": {},
      "name": "engine::baggage::set",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for baggage.set function",
        "properties": {
          "key": {
            "description": "The baggage key to set",
            "type": "string"
          },
          "value": {
            "description": "The baggage value to set",
            "type": "string"
          }
        },
        "required": [
          "key",
          "value"
        ],
        "title": "BaggageSetInput",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "note": {
            "type": "string"
          },
          "success": {
            "type": "boolean"
          }
        },
        "required": [
          "note",
          "success"
        ],
        "title": "BaggageSetResult",
        "type": "object"
      }
    },
    {
      "description": "Report observability subsystem health: per-component status (otel, metrics, logs, spans) marked healthy with counts or disabled, plus engine version. Always succeeds regardless of configuration.",
      "metadata": {},
      "name": "engine::health::check",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "HealthCheckInput",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "HealthComponentsView": {
            "properties": {
              "logs": true,
              "metrics": true,
              "otel": {
                "description": "Each component is `{ status: \"healthy\"|\"disabled\", details: <object|null> }`."
              },
              "spans": true
            },
            "required": [
              "logs",
              "metrics",
              "otel",
              "spans"
            ],
            "type": "object"
          }
        },
        "properties": {
          "components": {
            "$ref": "#/definitions/HealthComponentsView"
          },
          "status": {
            "type": "string"
          },
          "timestamp": {
            "format": "int64",
            "type": "integer"
          },
          "version": {
            "type": "string"
          }
        },
        "required": [
          "components",
          "status",
          "timestamp",
          "version"
        ],
        "title": "HealthCheckResult",
        "type": "object"
      }
    },
    {
      "description": "Record a DEBUG-level OTEL log (severity 5) with optional trace/span correlation and structured data. No-op when logs_enabled is false or dropped by logs_sampling_ratio.",
      "metadata": {},
      "name": "engine::log::debug",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for OTEL log functions (log.info, log.warn, log.error)",
        "properties": {
          "data": {
            "description": "Additional structured data/attributes"
          },
          "message": {
            "description": "The log message",
            "type": "string"
          },
          "service_name": {
            "description": "Service name (defaults to function name if not provided)",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Optional span ID for correlation",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Optional trace ID for correlation",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "message"
        ],
        "title": "OtelLogInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Record an ERROR-level OTEL log (severity 17) with optional trace/span correlation and structured data. No-op when logs_enabled is false or dropped by logs_sampling_ratio.",
      "metadata": {},
      "name": "engine::log::error",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for OTEL log functions (log.info, log.warn, log.error)",
        "properties": {
          "data": {
            "description": "Additional structured data/attributes"
          },
          "message": {
            "description": "The log message",
            "type": "string"
          },
          "service_name": {
            "description": "Service name (defaults to function name if not provided)",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Optional span ID for correlation",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Optional trace ID for correlation",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "message"
        ],
        "title": "OtelLogInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Record an INFO-level OTEL log (severity 9) with optional trace/span correlation and structured data. No-op when logs_enabled is false or dropped by logs_sampling_ratio.",
      "metadata": {},
      "name": "engine::log::info",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for OTEL log functions (log.info, log.warn, log.error)",
        "properties": {
          "data": {
            "description": "Additional structured data/attributes"
          },
          "message": {
            "description": "The log message",
            "type": "string"
          },
          "service_name": {
            "description": "Service name (defaults to function name if not provided)",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Optional span ID for correlation",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Optional trace ID for correlation",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "message"
        ],
        "title": "OtelLogInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Record a TRACE-level OTEL log (severity 1) with optional trace/span correlation and structured data. No-op when logs_enabled is false or dropped by logs_sampling_ratio.",
      "metadata": {},
      "name": "engine::log::trace",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for OTEL log functions (log.info, log.warn, log.error)",
        "properties": {
          "data": {
            "description": "Additional structured data/attributes"
          },
          "message": {
            "description": "The log message",
            "type": "string"
          },
          "service_name": {
            "description": "Service name (defaults to function name if not provided)",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Optional span ID for correlation",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Optional trace ID for correlation",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "message"
        ],
        "title": "OtelLogInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Record a WARN-level OTEL log (severity 13) with optional trace/span correlation and structured data. No-op when logs_enabled is false or dropped by logs_sampling_ratio.",
      "metadata": {},
      "name": "engine::log::warn",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for OTEL log functions (log.info, log.warn, log.error)",
        "properties": {
          "data": {
            "description": "Additional structured data/attributes"
          },
          "message": {
            "description": "The log message",
            "type": "string"
          },
          "service_name": {
            "description": "Service name (defaults to function name if not provided)",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Optional span ID for correlation",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Optional trace ID for correlation",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "message"
        ],
        "title": "OtelLogInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Drop every stored OTEL log, returning { success: true }. Succeeds as a no-op when log storage was never initialized (logs_enabled false).",
      "metadata": {},
      "name": "engine::logs::clear",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "LogsClearInput",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "message": {
            "type": [
              "string",
              "null"
            ]
          },
          "success": {
            "type": "boolean"
          }
        },
        "required": [
          "success"
        ],
        "title": "LogsClearResult",
        "type": "object"
      }
    },
    {
      "description": "List stored OTEL logs filtered by trace/span id, severity, and time range, with pagination and a total count. Returns an empty result when logs_enabled is false (no log storage).",
      "metadata": {},
      "name": "engine::logs::list",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "end_time": {
            "description": "End time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "limit": {
            "description": "Maximum number of logs to return",
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "offset": {
            "description": "Pagination offset (default: 0)",
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "severity_min": {
            "description": "Minimum severity number (1-24, higher = more severe)",
            "format": "int32",
            "type": [
              "integer",
              "null"
            ]
          },
          "severity_text": {
            "description": "Filter by severity text (e.g., \"ERROR\", \"WARN\", \"INFO\")",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Filter by span ID",
            "type": [
              "string",
              "null"
            ]
          },
          "start_time": {
            "description": "Start time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "trace_id": {
            "description": "Filter by trace ID",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "title": "LogsListInput",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "LogsListQuery": {
            "properties": {
              "end_time": {
                "format": "uint64",
                "minimum": 0,
                "type": [
                  "integer",
                  "null"
                ]
              },
              "limit": {
                "format": "uint",
                "minimum": 0,
                "type": [
                  "integer",
                  "null"
                ]
              },
              "offset": {
                "format": "uint",
                "minimum": 0,
                "type": [
                  "integer",
                  "null"
                ]
              },
              "severity_min": {
                "format": "int32",
                "type": [
                  "integer",
                  "null"
                ]
              },
              "severity_text": {
                "type": [
                  "string",
                  "null"
                ]
              },
              "span_id": {
                "type": [
                  "string",
                  "null"
                ]
              },
              "start_time": {
                "format": "uint64",
                "minimum": 0,
                "type": [
                  "integer",
                  "null"
                ]
              },
              "trace_id": {
                "type": [
                  "string",
                  "null"
                ]
              }
            },
            "type": "object"
          }
        },
        "properties": {
          "logs": {
            "description": "Stored OTEL log records (serialized).",
            "items": true,
            "type": "array"
          },
          "query": {
            "anyOf": [
              {
                "$ref": "#/definitions/LogsListQuery"
              },
              {
                "type": "null"
              }
            ],
            "description": "Echo of the applied filters (present only when storage exists)."
          },
          "timestamp": {
            "format": "int64",
            "type": "integer"
          },
          "total": {
            "description": "Total matching logs before pagination.",
            "format": "uint",
            "minimum": 0,
            "type": "integer"
          }
        },
        "required": [
          "logs",
          "timestamp",
          "total"
        ],
        "title": "LogsListResult",
        "type": "object"
      }
    },
    {
      "description": "Return engine invocation/worker counters and span-derived latency percentiles, plus stored SDK metrics filtered by name/time and optionally aggregated by interval. engine_metrics is always present; sdk_metrics is empty when no metric storage exists.",
      "metadata": {},
      "name": "engine::metrics::list",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "aggregate_interval": {
            "description": "Aggregate interval in seconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "end_time": {
            "description": "End time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "metric_name": {
            "description": "Filter by metric name",
            "type": [
              "string",
              "null"
            ]
          },
          "start_time": {
            "description": "Start time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          }
        },
        "title": "MetricsListInput",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "EngineMetricsView": {
            "properties": {
              "invocations": {
                "$ref": "#/definitions/InvocationsView"
              },
              "performance": {
                "$ref": "#/definitions/PerformanceView"
              },
              "workers": {
                "$ref": "#/definitions/WorkersView"
              }
            },
            "required": [
              "invocations",
              "performance",
              "workers"
            ],
            "type": "object"
          },
          "InvocationsView": {
            "properties": {
              "by_function": {
                "additionalProperties": {
                  "format": "uint64",
                  "minimum": 0,
                  "type": "integer"
                },
                "description": "Per-function invocation counts.",
                "type": "object"
              },
              "deferred": {
                "format": "uint64",
                "minimum": 0,
                "type": "integer"
              },
              "error": {
                "format": "uint64",
                "minimum": 0,
                "type": "integer"
              },
              "success": {
                "format": "uint64",
                "minimum": 0,
                "type": "integer"
              },
              "total": {
                "format": "uint64",
                "minimum": 0,
                "type": "integer"
              }
            },
            "required": [
              "by_function",
              "deferred",
              "error",
              "success",
              "total"
            ],
            "type": "object"
          },
          "MetricsListQuery": {
            "properties": {
              "aggregate_interval": {
                "format": "uint64",
                "minimum": 0,
                "type": [
                  "integer",
                  "null"
                ]
              },
              "end_time": {
                "format": "uint64",
                "minimum": 0,
                "type": [
                  "integer",
                  "null"
                ]
              },
              "metric_name": {
                "type": [
                  "string",
                  "null"
                ]
              },
              "start_time": {
                "format": "uint64",
                "minimum": 0,
                "type": [
                  "integer",
                  "null"
                ]
              }
            },
            "type": "object"
          },
          "PerformanceView": {
            "properties": {
              "avg_duration_ms": {
                "format": "double",
                "type": "number"
              },
              "max_duration_ms": {
                "format": "double",
                "type": "number"
              },
              "min_duration_ms": {
                "format": "double",
                "type": "number"
              },
              "p50_duration_ms": {
                "format": "double",
                "type": "number"
              },
              "p95_duration_ms": {
                "format": "double",
                "type": "number"
              },
              "p99_duration_ms": {
                "format": "double",
                "type": "number"
              }
            },
            "required": [
              "avg_duration_ms",
              "max_duration_ms",
              "min_duration_ms",
              "p50_duration_ms",
              "p95_duration_ms",
              "p99_duration_ms"
            ],
            "type": "object"
          },
          "WorkersView": {
            "properties": {
              "active": {
                "format": "uint64",
                "minimum": 0,
                "type": "integer"
              },
              "deaths": {
                "format": "uint64",
                "minimum": 0,
                "type": "integer"
              },
              "spawns": {
                "format": "uint64",
                "minimum": 0,
                "type": "integer"
              }
            },
            "required": [
              "active",
              "deaths",
              "spawns"
            ],
            "type": "object"
          }
        },
        "properties": {
          "aggregated_metrics": {
            "description": "Time-bucketed aggregates (present only when an aggregate_interval was requested and produced data).",
            "items": true,
            "type": "array"
          },
          "engine_metrics": {
            "$ref": "#/definitions/EngineMetricsView"
          },
          "query": {
            "anyOf": [
              {
                "$ref": "#/definitions/MetricsListQuery"
              },
              {
                "type": "null"
              }
            ],
            "description": "Echo of the applied query filters (present only when a filter was supplied)."
          },
          "sdk_metrics": {
            "description": "Stored SDK metric points (serialized).",
            "items": true,
            "type": "array"
          },
          "timestamp": {
            "format": "int64",
            "type": "integer"
          }
        },
        "required": [
          "aggregated_metrics",
          "engine_metrics",
          "sdk_metrics",
          "timestamp"
        ],
        "title": "MetricsListResult",
        "type": "object"
      }
    },
    {
      "description": "Return pre-aggregated metric rollups and histograms for a level (0=1m, 1=5m, 2=1h) over a time range (default last hour), optionally by metric name. Falls back to on-the-fly aggregation over metric storage when no rollup storage exists.",
      "metadata": {},
      "name": "engine::rollups::list",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "end_time": {
            "description": "End time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "level": {
            "description": "Rollup level index (0 = 1 min, 1 = 5 min, 2 = 1 hour)",
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "metric_name": {
            "description": "Filter by metric name",
            "type": [
              "string",
              "null"
            ]
          },
          "start_time": {
            "description": "Start time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          }
        },
        "title": "RollupsListInput",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "RollupsListQuery": {
            "properties": {
              "end_time": {
                "format": "uint64",
                "minimum": 0,
                "type": [
                  "integer",
                  "null"
                ]
              },
              "metric_name": {
                "type": [
                  "string",
                  "null"
                ]
              },
              "start_time": {
                "format": "uint64",
                "minimum": 0,
                "type": [
                  "integer",
                  "null"
                ]
              }
            },
            "type": "object"
          }
        },
        "properties": {
          "histogram_rollups": {
            "description": "Pre-aggregated histogram rollups (serialized).",
            "items": true,
            "type": "array"
          },
          "level": {
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "message": {
            "type": [
              "string",
              "null"
            ]
          },
          "query": {
            "anyOf": [
              {
                "$ref": "#/definitions/RollupsListQuery"
              },
              {
                "type": "null"
              }
            ]
          },
          "rollups": {
            "description": "Pre-aggregated metric rollups (serialized).",
            "items": true,
            "type": "array"
          },
          "source": {
            "description": "\"on_the_fly\" when computed live because no rollup storage exists.",
            "type": [
              "string",
              "null"
            ]
          },
          "timestamp": {
            "format": "int64",
            "type": "integer"
          }
        },
        "required": [
          "histogram_rollups",
          "rollups",
          "timestamp"
        ],
        "title": "RollupsListResult",
        "type": "object"
      }
    },
    {
      "description": "Report the active trace sampling config (default ratio, per-operation/service rules, parent_based) and the logs sampling_ratio, read from live config. Defaults to ratio 1.0 with no rules when sampling is unconfigured.",
      "metadata": {},
      "name": "engine::sampling::rules",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "LogsClearInput",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "SamplingLogsView": {
            "properties": {
              "sampling_ratio": {
                "format": "double",
                "type": "number"
              }
            },
            "required": [
              "sampling_ratio"
            ],
            "type": "object"
          },
          "SamplingRuleView": {
            "properties": {
              "operation": {
                "type": [
                  "string",
                  "null"
                ]
              },
              "rate": {
                "format": "double",
                "type": "number"
              },
              "service": {
                "type": [
                  "string",
                  "null"
                ]
              }
            },
            "required": [
              "rate"
            ],
            "type": "object"
          },
          "SamplingTracesView": {
            "properties": {
              "default_ratio": {
                "format": "double",
                "type": "number"
              },
              "parent_based": {
                "type": "boolean"
              },
              "rules": {
                "items": {
                  "$ref": "#/definitions/SamplingRuleView"
                },
                "type": "array"
              }
            },
            "required": [
              "default_ratio",
              "parent_based",
              "rules"
            ],
            "type": "object"
          }
        },
        "properties": {
          "logs": {
            "$ref": "#/definitions/SamplingLogsView"
          },
          "timestamp": {
            "format": "int64",
            "type": "integer"
          },
          "traces": {
            "$ref": "#/definitions/SamplingTracesView"
          }
        },
        "required": [
          "logs",
          "timestamp",
          "traces"
        ],
        "title": "SamplingRulesResult",
        "type": "object"
      }
    },
    {
      "description": "Drop every span from the in-memory trace store, returning { success: true }. Requires exporter memory or both, else fails memory_exporter_not_enabled.",
      "metadata": {},
      "name": "engine::traces::clear",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "TracesClearInput",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "success": {
            "type": "boolean"
          }
        },
        "required": [
          "success"
        ],
        "title": "OkResult",
        "type": "object"
      }
    },
    {
      "description": "Aggregate stored spans by one attribute into groups (trace_ids, span_count, duration, error_count), newest-first, capped at limit (default 100); skips spans lacking the attribute and engine-internal spans unless include_internal. Requires exporter memory or both.",
      "metadata": {},
      "name": "engine::traces::group_by",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "attribute": {
            "description": "Span attribute key to group by. Spans without this attribute are skipped.",
            "type": "string"
          },
          "include_internal": {
            "default": null,
            "description": "Include engine-internal spans. Defaults to false, matching `traces::list`.",
            "type": [
              "boolean",
              "null"
            ]
          },
          "limit": {
            "default": null,
            "description": "Max groups returned after sorting by `first_seen_ms` descending. Default 100.",
            "format": "uint32",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "since_ms": {
            "default": null,
            "description": "Earliest end_time (ms since epoch) to include.",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          }
        },
        "required": [
          "attribute"
        ],
        "title": "TracesGroupByInput",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "TraceGroup": {
            "properties": {
              "duration_ms": {
                "format": "uint32",
                "minimum": 0,
                "type": "integer"
              },
              "error_count": {
                "format": "uint32",
                "minimum": 0,
                "type": "integer"
              },
              "first_seen_ms": {
                "format": "uint64",
                "minimum": 0,
                "type": "integer"
              },
              "last_seen_ms": {
                "format": "uint64",
                "minimum": 0,
                "type": "integer"
              },
              "span_count": {
                "format": "uint32",
                "minimum": 0,
                "type": "integer"
              },
              "trace_ids": {
                "items": {
                  "type": "string"
                },
                "type": "array"
              },
              "value": {
                "type": "string"
              }
            },
            "required": [
              "duration_ms",
              "error_count",
              "first_seen_ms",
              "last_seen_ms",
              "span_count",
              "trace_ids",
              "value"
            ],
            "type": "object"
          }
        },
        "properties": {
          "groups": {
            "items": {
              "$ref": "#/definitions/TraceGroup"
            },
            "type": "array"
          }
        },
        "required": [
          "groups"
        ],
        "title": "TracesGroupByResult",
        "type": "object"
      }
    },
    {
      "description": "List root spans of stored traces with filtering (service, name, status, duration, time, attributes), pagination, and sort; hides engine-internal spans unless include_internal. Requires exporter memory or both, else fails memory_exporter_not_enabled.",
      "metadata": {},
      "name": "engine::traces::list",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "attributes": {
            "description": "Filter by span attributes (array of [key, value] pairs, AND logic, exact match)",
            "items": {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            "type": [
              "array",
              "null"
            ]
          },
          "end_time": {
            "description": "End time in unix timestamp milliseconds (include spans overlapping before this)",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "include_internal": {
            "default": null,
            "description": "Include internal engine traces (engine.* functions). Defaults to false.",
            "type": [
              "boolean",
              "null"
            ]
          },
          "limit": {
            "description": "Pagination limit (default: 100)",
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "max_duration_ms": {
            "description": "Maximum span duration in milliseconds (sub-ms precision)",
            "format": "double",
            "type": [
              "number",
              "null"
            ]
          },
          "min_duration_ms": {
            "description": "Minimum span duration in milliseconds (sub-ms precision)",
            "format": "double",
            "type": [
              "number",
              "null"
            ]
          },
          "name": {
            "description": "Filter by span name (case-insensitive substring match)",
            "type": [
              "string",
              "null"
            ]
          },
          "offset": {
            "description": "Pagination offset (default: 0)",
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "search_all_spans": {
            "default": null,
            "description": "Search across all spans in each trace, not just root spans. When true and a `name` filter is set, traces are matched if ANY span in the trace matches the name filter. Defaults to false.",
            "type": [
              "boolean",
              "null"
            ]
          },
          "service_name": {
            "description": "Filter by service name (case-insensitive substring match)",
            "type": [
              "string",
              "null"
            ]
          },
          "sort_by": {
            "description": "Sort field: \"start_time\" | \"duration\" (alias \"duration_ms\") | \"service_name\" | \"name\" (default: \"start_time\"). Unknown values fall back to \"start_time\".",
            "type": [
              "string",
              "null"
            ]
          },
          "sort_order": {
            "description": "Sort order: \"asc\" | \"desc\" (default: \"asc\")",
            "type": [
              "string",
              "null"
            ]
          },
          "start_time": {
            "description": "Start time in unix timestamp milliseconds (include spans overlapping after this)",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "status": {
            "description": "Filter by status (case-insensitive substring match)",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Filter by specific trace ID",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "title": "TracesListInput",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "limit": {
            "format": "uint",
            "minimum": 0,
            "type": "integer"
          },
          "offset": {
            "format": "uint",
            "minimum": 0,
            "type": "integer"
          },
          "spans": {
            "description": "Stored spans (each a serialized span record).",
            "items": true,
            "type": "array"
          },
          "total": {
            "description": "Total matching spans before pagination.",
            "format": "uint",
            "minimum": 0,
            "type": "integer"
          }
        },
        "required": [
          "limit",
          "offset",
          "spans",
          "total"
        ],
        "title": "TracesListResult",
        "type": "object"
      }
    },
    {
      "description": "Build the nested span tree for one trace_id as { roots }, pruning no-op trigger wrappers and collapsing configured pass-through spans. Requires exporter memory or both, else fails memory_exporter_not_enabled.",
      "metadata": {},
      "name": "engine::traces::tree",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "trace_id": {
            "description": "Trace ID to build the tree for",
            "type": "string"
          }
        },
        "required": [
          "trace_id"
        ],
        "title": "TracesTreeInput",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "roots": {
            "description": "Root span-tree nodes (each a serialized, flattened span with nested children).",
            "items": true,
            "type": "array"
          }
        },
        "required": [
          "roots"
        ],
        "title": "TracesTreeResult",
        "type": "object"
      }
    },
    {
      "description": "Internal: re-apply the iii-observability configuration when the authoritative configuration entry changes.",
      "metadata": {
        "internal": true
      },
      "name": "iii-observability::on-config-change",
      "request_schema": {},
      "response_schema": {}
    }
  ],
  "triggers": []
}