iii / worker
$worker

iii-observability

v0.11.6-next.3

OpenTelemetry-based traces, metrics, logs, alerts, and sampling.

engine module
baked into the iii engine; no separate install required.
agent-ready brief for v0.11.6-next.3
install + config + dependencies + readme + api reference, all in one place. fetch as agent-context.md for an llm to consume.
the same content rendered as discrete blocks below is exposed as a single markdown document at /workers/iii-observability.md?version=0.11.6-next.3. paste it into an llm prompt or pipe it through curl from a worker.

install

install
$iii worker add iii-observability@0.11.6-next.3

configuration

iii-config.yaml
- enabled: true
  exporter: memory
  logs_console_output: true
  logs_enabled: true
  logs_exporter: memory
  logs_max_count: 1000
  logs_retention_seconds: 3600
  memory_max_spans: 10000
  metrics_enabled: true
  metrics_exporter: memory
  metrics_max_count: 10000
  metrics_retention_seconds: 3600
  sampling_ratio: 1
  service_name: iii
  service_version: ${SERVICE_VERSION:__III_ENGINE_VERSION__}

dependencies

no dependencies for v0.11.6-next.3

readme

README.md

iii-observability

Full OpenTelemetry observability for III Engine: distributed tracing, structured logs, performance metrics, alert rules, and trace sampling — all queryable via built-in functions.

Sample Configuration

- name: iii-observability
  config:
    enabled: true
    service_name: my-service
    service_version: 1.0.0
    exporter: memory
    metrics_enabled: true
    logs_enabled: true
    memory_max_spans: 1000
    sampling_ratio: 1.0
    alerts:
      - name: high-error-rate
        metric: iii.invocations.error
        threshold: 10
        operator: ">"
        window_seconds: 60
        action:
          type: log

Configuration

Field Type Description
enabled boolean Whether OpenTelemetry tracing export is enabled. Defaults to false. Env: OTEL_ENABLED.
service_name string Service name in traces and metrics. Defaults to "iii". Env: OTEL_SERVICE_NAME.
service_version string Service version (service.version OTEL attribute). Env: SERVICE_VERSION.
service_namespace string Service namespace (service.namespace OTEL attribute). Env: SERVICE_NAMESPACE.
exporter string Trace exporter: memory, otlp, or both. Defaults to otlp. Env: OTEL_EXPORTER_TYPE.
endpoint string OTLP collector endpoint. Defaults to "http://localhost:4317". Env: OTEL_EXPORTER_OTLP_ENDPOINT.
sampling_ratio number Global trace sampling ratio (0.01.0). Defaults to 1.0. Env: OTEL_TRACES_SAMPLER_ARG.
memory_max_spans number Max spans to keep in memory. Defaults to 1000. Env: OTEL_MEMORY_MAX_SPANS.
metrics_enabled boolean Whether metrics collection is enabled. Defaults to false. Env: OTEL_METRICS_ENABLED.
metrics_exporter string Metrics exporter: memory or otlp. Defaults to memory. Env: OTEL_METRICS_EXPORTER.
metrics_retention_seconds number How long to retain metrics in memory. Defaults to 3600. Env: OTEL_METRICS_RETENTION_SECONDS.
metrics_max_count number Max metric data points in memory. Defaults to 10000. Env: OTEL_METRICS_MAX_COUNT.
logs_enabled boolean Whether structured log storage is enabled.
logs_exporter string Logs exporter: memory, otlp, or both. Defaults to memory. Env: OTEL_LOGS_EXPORTER.
logs_max_count number Max log entries in memory. Defaults to 1000.
logs_retention_seconds number How long to retain logs in memory. Defaults to 3600.
logs_sampling_ratio number Fraction of logs to retain (0.01.0). Defaults to 1.0.
logs_console_output boolean Print ingested logs to the console. Defaults to true.
level string Minimum log level: trace, debug, info, warn, error. Defaults to info.
format string Log output format: default or json. Defaults to default.
alerts AlertRule[] Alert rules evaluated against metrics.

Alert Rule Fields

Field Type Description
name string Required. Unique alert rule name.
metric string Required. Metric name to monitor (e.g., iii.invocations.error).
threshold number Required. Threshold value.
operator string Comparison operator: >, >=, <, <=, ==, !=. Defaults to >.
window_seconds number Time window in seconds for metric evaluation. Defaults to 60.
cooldown_seconds number Minimum interval between alert fires. Defaults to 60.
enabled boolean Whether the alert rule is active. Defaults to true.
action AlertAction { "type": "log" }, { "type": "webhook", "url": "..." }, or { "type": "function", "path": "..." }.

Advanced Sampling

sampling:
  default: 1.0
  parent_based: true
  rules:
    - operation: "api.*"
      rate: 0.1
  rate_limit:
    max_traces_per_second: 100

Functions

Logging

Function Description
engine::log::info Log an informational message.
engine::log::warn Log a warning message.
engine::log::error Log an error message.
engine::log::debug Log a debug message.
engine::log::trace Log a trace-level message.

All logging functions accept: message (string, required), data (object), trace_id (string), span_id (string), service_name (string).

Logs API

Function Description
engine::logs::list Query stored log entries. Filters: start_time, end_time, trace_id, span_id, severity_min, severity_text, offset, limit.
engine::logs::clear Clear all stored log entries from memory.

Traces API

Function Description
engine::traces::list List stored spans. Filters: trace_id, service_name, name, status, min_duration_ms, max_duration_ms, start_time, end_time, sort_by, sort_order, attributes, include_internal, offset, limit.
engine::traces::tree Retrieve a trace as a hierarchical span tree. Parameters: trace_id (required).
engine::traces::clear Clear all stored trace spans from memory.

Metrics API

Function Description
engine::metrics::list List metrics with aggregated statistics. Returns engine counters (invocations, workers, performance), SDK metrics, and optional time-bucketed aggregations.
engine::rollups::list List metric rollup aggregations (1-minute, 5-minute, 1-hour windows).

Other APIs

Function Description
engine::baggage::get Get a baggage value from the current trace context.
engine::baggage::set Set a baggage value in the current trace context.
engine::baggage::get_all Get all baggage key-value pairs.
engine::sampling::rules List all active sampling rules.
engine::health::check Check engine health status. Returns status, components, timestamp, version.
engine::alerts::list List all configured alert rules and their current state.
engine::alerts::evaluate Manually trigger evaluation of all alert rules.

Trigger Type: log

Register a function to react to log entries as they are produced.

Config Field Type Description
level string Log level to subscribe to: info, warn, error, debug, or trace. When omitted, fires for all levels.

Sample Code

const fn = iii.registerFunction(
  'monitoring::onError',
  async (logEntry) => {
    await sendAlert({
      message: logEntry.body,
      severity: logEntry.severity_text,
      traceId: logEntry.trace_id,
    })
    return {}
  },
)

iii.registerTrigger({
  type: 'log',
  function_id: fn.id,
  config: { level: 'error' },
})

Log entry payload fields: timestamp_unix_nano, observed_timestamp_unix_nano, severity_number, severity_text, body, attributes, trace_id, span_id, resource, service_name, instrumentation_scope_name, instrumentation_scope_version.

api reference (json)

agent-api-reference.json
{
  "functions": [
    {
      "description": "Manually trigger alert evaluation",
      "metadata": {},
      "name": "engine::alerts::evaluate",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "AlertsEvaluateInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "List current alert states",
      "metadata": {},
      "name": "engine::alerts::list",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "AlertsListInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Get a baggage item value from the current context",
      "metadata": {},
      "name": "engine::baggage::get",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for baggage.get function",
        "properties": {
          "key": {
            "description": "The baggage key to retrieve",
            "type": "string"
          }
        },
        "required": [
          "key"
        ],
        "title": "BaggageGetInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Get all baggage items from the current context",
      "metadata": {},
      "name": "engine::baggage::get_all",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for baggage.getAll function (empty)",
        "title": "BaggageGetAllInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Set a baggage item value (returns new context, does not modify global)",
      "metadata": {},
      "name": "engine::baggage::set",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for baggage.set function",
        "properties": {
          "key": {
            "description": "The baggage key to set",
            "type": "string"
          },
          "value": {
            "description": "The baggage value to set",
            "type": "string"
          }
        },
        "required": [
          "key",
          "value"
        ],
        "title": "BaggageSetInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Check system health status",
      "metadata": {},
      "name": "engine::health::check",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "HealthCheckInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Log a debug message using OTEL",
      "metadata": {},
      "name": "engine::log::debug",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for OTEL log functions (log.info, log.warn, log.error)",
        "properties": {
          "data": {
            "description": "Additional structured data/attributes"
          },
          "message": {
            "description": "The log message",
            "type": "string"
          },
          "service_name": {
            "description": "Service name (defaults to function name if not provided)",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Optional span ID for correlation",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Optional trace ID for correlation",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "message"
        ],
        "title": "OtelLogInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Log an error message using OTEL",
      "metadata": {},
      "name": "engine::log::error",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for OTEL log functions (log.info, log.warn, log.error)",
        "properties": {
          "data": {
            "description": "Additional structured data/attributes"
          },
          "message": {
            "description": "The log message",
            "type": "string"
          },
          "service_name": {
            "description": "Service name (defaults to function name if not provided)",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Optional span ID for correlation",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Optional trace ID for correlation",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "message"
        ],
        "title": "OtelLogInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Log an info message using OTEL",
      "metadata": {},
      "name": "engine::log::info",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for OTEL log functions (log.info, log.warn, log.error)",
        "properties": {
          "data": {
            "description": "Additional structured data/attributes"
          },
          "message": {
            "description": "The log message",
            "type": "string"
          },
          "service_name": {
            "description": "Service name (defaults to function name if not provided)",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Optional span ID for correlation",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Optional trace ID for correlation",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "message"
        ],
        "title": "OtelLogInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Log a trace-level message using OTEL",
      "metadata": {},
      "name": "engine::log::trace",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for OTEL log functions (log.info, log.warn, log.error)",
        "properties": {
          "data": {
            "description": "Additional structured data/attributes"
          },
          "message": {
            "description": "The log message",
            "type": "string"
          },
          "service_name": {
            "description": "Service name (defaults to function name if not provided)",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Optional span ID for correlation",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Optional trace ID for correlation",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "message"
        ],
        "title": "OtelLogInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Log a warning message using OTEL",
      "metadata": {},
      "name": "engine::log::warn",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for OTEL log functions (log.info, log.warn, log.error)",
        "properties": {
          "data": {
            "description": "Additional structured data/attributes"
          },
          "message": {
            "description": "The log message",
            "type": "string"
          },
          "service_name": {
            "description": "Service name (defaults to function name if not provided)",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Optional span ID for correlation",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Optional trace ID for correlation",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "message"
        ],
        "title": "OtelLogInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Clear all stored OTEL logs",
      "metadata": {},
      "name": "engine::logs::clear",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "LogsClearInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "List stored OTEL logs",
      "metadata": {},
      "name": "engine::logs::list",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "end_time": {
            "description": "End time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "limit": {
            "description": "Maximum number of logs to return",
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "offset": {
            "description": "Pagination offset (default: 0)",
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "severity_min": {
            "description": "Minimum severity number (1-24, higher = more severe)",
            "format": "int32",
            "type": [
              "integer",
              "null"
            ]
          },
          "severity_text": {
            "description": "Filter by severity text (e.g., \"ERROR\", \"WARN\", \"INFO\")",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Filter by span ID",
            "type": [
              "string",
              "null"
            ]
          },
          "start_time": {
            "description": "Start time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "trace_id": {
            "description": "Filter by trace ID",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "title": "LogsListInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "List current metrics values",
      "metadata": {},
      "name": "engine::metrics::list",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "aggregate_interval": {
            "description": "Aggregate interval in seconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "end_time": {
            "description": "End time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "metric_name": {
            "description": "Filter by metric name",
            "type": [
              "string",
              "null"
            ]
          },
          "start_time": {
            "description": "Start time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          }
        },
        "title": "MetricsListInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Get pre-aggregated metrics rollups",
      "metadata": {},
      "name": "engine::rollups::list",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "end_time": {
            "description": "End time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "level": {
            "description": "Rollup level index (0 = 1 min, 1 = 5 min, 2 = 1 hour)",
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "metric_name": {
            "description": "Filter by metric name",
            "type": [
              "string",
              "null"
            ]
          },
          "start_time": {
            "description": "Start time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          }
        },
        "title": "RollupsListInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Get active sampling rules configuration",
      "metadata": {},
      "name": "engine::sampling::rules",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "LogsClearInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Clear all stored traces (only available when exporter is 'memory' or 'both')",
      "metadata": {},
      "name": "engine::traces::clear",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "TracesClearInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "List stored traces (only available when exporter is 'memory' or 'both')",
      "metadata": {},
      "name": "engine::traces::list",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "attributes": {
            "description": "Filter by span attributes (array of [key, value] pairs, AND logic, exact match)",
            "items": {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            "type": [
              "array",
              "null"
            ]
          },
          "end_time": {
            "description": "End time in unix timestamp milliseconds (include spans overlapping before this)",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "include_internal": {
            "default": null,
            "description": "Include internal engine traces (engine.* functions). Defaults to false.",
            "type": [
              "boolean",
              "null"
            ]
          },
          "limit": {
            "description": "Pagination limit (default: 100)",
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "max_duration_ms": {
            "description": "Maximum span duration in milliseconds (sub-ms precision)",
            "format": "double",
            "type": [
              "number",
              "null"
            ]
          },
          "min_duration_ms": {
            "description": "Minimum span duration in milliseconds (sub-ms precision)",
            "format": "double",
            "type": [
              "number",
              "null"
            ]
          },
          "name": {
            "description": "Filter by span name (case-insensitive substring match)",
            "type": [
              "string",
              "null"
            ]
          },
          "offset": {
            "description": "Pagination offset (default: 0)",
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "search_all_spans": {
            "default": null,
            "description": "Search across all spans in each trace, not just root spans. When true and a `name` filter is set, traces are matched if ANY span in the trace matches the name filter. Defaults to false.",
            "type": [
              "boolean",
              "null"
            ]
          },
          "service_name": {
            "description": "Filter by service name (case-insensitive substring match)",
            "type": [
              "string",
              "null"
            ]
          },
          "sort_by": {
            "description": "Sort field: \"duration\" | \"start_time\" | \"name\" (default: \"start_time\")",
            "type": [
              "string",
              "null"
            ]
          },
          "sort_order": {
            "description": "Sort order: \"asc\" | \"desc\" (default: \"asc\")",
            "type": [
              "string",
              "null"
            ]
          },
          "start_time": {
            "description": "Start time in unix timestamp milliseconds (include spans overlapping after this)",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "status": {
            "description": "Filter by status (case-insensitive substring match)",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Filter by specific trace ID",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "title": "TracesListInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Get trace tree with nested children (only available when exporter is 'memory' or 'both')",
      "metadata": {},
      "name": "engine::traces::tree",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "trace_id": {
            "description": "Trace ID to build the tree for",
            "type": "string"
          }
        },
        "required": [
          "trace_id"
        ],
        "title": "TracesTreeInput",
        "type": "object"
      },
      "response_schema": {}
    }
  ],
  "triggers": [
    {
      "description": "Log event trigger",
      "invocation_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "LogLevel": {
            "enum": [
              "all",
              "debug",
              "info",
              "warn",
              "error"
            ],
            "type": "string"
          }
        },
        "properties": {
          "level": {
            "anyOf": [
              {
                "$ref": "#/definitions/LogLevel"
              },
              {
                "type": "null"
              }
            ],
            "default": "all",
            "description": "Minimum log level to trigger on"
          }
        },
        "title": "LogTriggerConfig",
        "type": "object"
      },
      "metadata": {},
      "name": "log",
      "return_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "attributes": {
            "description": "Log attributes"
          },
          "body": {
            "description": "Log message body",
            "type": "string"
          },
          "instrumentation_scope_name": {
            "description": "Instrumentation scope name",
            "type": "string"
          },
          "instrumentation_scope_version": {
            "description": "Instrumentation scope version",
            "type": "string"
          },
          "observed_timestamp_unix_nano": {
            "description": "Observed timestamp in nanoseconds",
            "format": "uint64",
            "minimum": 0,
            "type": "integer"
          },
          "resource": {
            "description": "OpenTelemetry resource"
          },
          "service_name": {
            "description": "Service name",
            "type": "string"
          },
          "severity_number": {
            "description": "Severity number (OpenTelemetry)",
            "format": "uint32",
            "minimum": 0,
            "type": "integer"
          },
          "severity_text": {
            "description": "Severity text (e.g. INFO, ERROR)",
            "type": "string"
          },
          "span_id": {
            "description": "Span ID",
            "type": "string"
          },
          "timestamp_unix_nano": {
            "description": "Log timestamp in nanoseconds",
            "format": "uint64",
            "minimum": 0,
            "type": "integer"
          },
          "trace_id": {
            "description": "Trace ID",
            "type": "string"
          }
        },
        "required": [
          "attributes",
          "body",
          "instrumentation_scope_name",
          "instrumentation_scope_version",
          "observed_timestamp_unix_nano",
          "resource",
          "service_name",
          "severity_number",
          "severity_text",
          "span_id",
          "timestamp_unix_nano",
          "trace_id"
        ],
        "title": "LogCallRequest",
        "type": "object"
      }
    }
  ]
}