$worker

iii-observability

v0.11.6-next.6

OpenTelemetry-based traces, metrics, logs, alerts, and sampling.

engine module

baked into the iii engine; no separate install required.

agent-ready brief for v0.11.6-next.6

install + config + dependencies + readme + api reference, all in one place. fetch as agent-context.md for an llm to consume.

full markdown

open as agent-context.md

the same content rendered as discrete blocks below is exposed as a single markdown document at /workers/iii-observability.md?version=0.11.6-next.6. paste it into an llm prompt or pipe it through curl from a worker.

install

$iii worker add iii-observability@0.11.6-next.6

configuration

iii-config.yaml

- enabled: true
  exporter: memory
  logs_console_output: true
  logs_enabled: true
  logs_exporter: memory
  logs_max_count: 1000
  logs_retention_seconds: 3600
  memory_max_spans: 10000
  metrics_enabled: true
  metrics_exporter: memory
  metrics_max_count: 10000
  metrics_retention_seconds: 3600
  sampling_ratio: 1
  service_name: iii
  service_version: ${SERVICE_VERSION:__III_ENGINE_VERSION__}

dependencies

no dependencies for v0.11.6-next.6

readme

README.md

iii-observability

Full OpenTelemetry observability for III Engine: distributed tracing, structured logs, performance metrics, alert rules, and trace sampling — all queryable via built-in functions.

Sample Configuration

- name: iii-observability
  config:
    enabled: true
    service_name: my-service
    service_version: 1.0.0
    exporter: memory
    metrics_enabled: true
    logs_enabled: true
    memory_max_spans: 1000
    sampling_ratio: 1.0
    alerts:
      - name: high-error-rate
        metric: iii.invocations.error
        threshold: 10
        operator: ">"
        window_seconds: 60
        action:
          type: log

Configuration

Field	Type	Description
`enabled`	boolean	Whether OpenTelemetry tracing export is enabled. Defaults to `false`. Env: `OTEL_ENABLED`.
`service_name`	string	Service name in traces and metrics. Defaults to `"iii"`. Env: `OTEL_SERVICE_NAME`.
`service_version`	string	Service version (`service.version` OTEL attribute). Env: `SERVICE_VERSION`.
`service_namespace`	string	Service namespace (`service.namespace` OTEL attribute). Env: `SERVICE_NAMESPACE`.
`exporter`	string	Trace exporter: `memory`, `otlp`, or `both`. Defaults to `otlp`. Env: `OTEL_EXPORTER_TYPE`.
`endpoint`	string	OTLP collector endpoint. Defaults to `"http://localhost:4317"`. Env: `OTEL_EXPORTER_OTLP_ENDPOINT`.
`sampling_ratio`	number	Global trace sampling ratio (`0.0`–`1.0`). Defaults to `1.0`. Env: `OTEL_TRACES_SAMPLER_ARG`.
`memory_max_spans`	number	Max spans to keep in memory. Defaults to `1000`. Env: `OTEL_MEMORY_MAX_SPANS`.
`metrics_enabled`	boolean	Whether metrics collection is enabled. Defaults to `false`. Env: `OTEL_METRICS_ENABLED`.
`metrics_exporter`	string	Metrics exporter: `memory` or `otlp`. Defaults to `memory`. Env: `OTEL_METRICS_EXPORTER`.
`metrics_retention_seconds`	number	How long to retain metrics in memory. Defaults to `3600`. Env: `OTEL_METRICS_RETENTION_SECONDS`.
`metrics_max_count`	number	Max metric data points in memory. Defaults to `10000`. Env: `OTEL_METRICS_MAX_COUNT`.
`logs_enabled`	boolean	Whether structured log storage is enabled.
`logs_exporter`	string	Logs exporter: `memory`, `otlp`, or `both`. Defaults to `memory`. Env: `OTEL_LOGS_EXPORTER`.
`logs_max_count`	number	Max log entries in memory. Defaults to `1000`.
`logs_retention_seconds`	number	How long to retain logs in memory. Defaults to `3600`.
`logs_sampling_ratio`	number	Fraction of logs to retain (`0.0`–`1.0`). Defaults to `1.0`.
`logs_console_output`	boolean	Print ingested logs to the console. Defaults to `true`.
`level`	string	Minimum log level: `trace`, `debug`, `info`, `warn`, `error`. Defaults to `info`.
`format`	string	Log output format: `default` or `json`. Defaults to `default`.
`alerts`	AlertRule[]	Alert rules evaluated against metrics.

Alert Rule Fields

Field	Type	Description
`name`	string	Required. Unique alert rule name.
`metric`	string	Required. Metric name to monitor (e.g., `iii.invocations.error`).
`threshold`	number	Required. Threshold value.
`operator`	string	Comparison operator: `>`, `>=`, `<`, `<=`, `==`, `!=`. Defaults to `>`.
`window_seconds`	number	Time window in seconds for metric evaluation. Defaults to `60`.
`cooldown_seconds`	number	Minimum interval between alert fires. Defaults to `60`.
`enabled`	boolean	Whether the alert rule is active. Defaults to `true`.
`action`	AlertAction	`{ "type": "log" }`, `{ "type": "webhook", "url": "..." }`, or `{ "type": "function", "path": "..." }`.

Advanced Sampling

sampling:
  default: 1.0
  parent_based: true
  rules:
    - operation: "api.*"
      rate: 0.1
  rate_limit:
    max_traces_per_second: 100

Functions

Logging

Function	Description
`engine::log::info`	Log an informational message.
`engine::log::warn`	Log a warning message.
`engine::log::error`	Log an error message.
`engine::log::debug`	Log a debug message.
`engine::log::trace`	Log a trace-level message.

All logging functions accept: message (string, required), data (object), trace_id (string), span_id (string), service_name (string).

Logs API

Function	Description
`engine::logs::list`	Query stored log entries. Filters: `start_time`, `end_time`, `trace_id`, `span_id`, `severity_min`, `severity_text`, `offset`, `limit`.
`engine::logs::clear`	Clear all stored log entries from memory.

Traces API

Function	Description
`engine::traces::list`	List stored spans. Filters: `trace_id`, `service_name`, `name`, `status`, `min_duration_ms`, `max_duration_ms`, `start_time`, `end_time`, `sort_by`, `sort_order`, `attributes`, `include_internal`, `offset`, `limit`.
`engine::traces::tree`	Retrieve a trace as a hierarchical span tree. Parameters: `trace_id` (required).
`engine::traces::clear`	Clear all stored trace spans from memory.

Metrics API

Function	Description
`engine::metrics::list`	List metrics with aggregated statistics. Returns engine counters (invocations, workers, performance), SDK metrics, and optional time-bucketed aggregations.
`engine::rollups::list`	List metric rollup aggregations (1-minute, 5-minute, 1-hour windows).

Other APIs

Function	Description
`engine::baggage::get`	Get a baggage value from the current trace context.
`engine::baggage::set`	Set a baggage value in the current trace context.
`engine::baggage::get_all`	Get all baggage key-value pairs.
`engine::sampling::rules`	List all active sampling rules.
`engine::health::check`	Check engine health status. Returns `status`, `components`, `timestamp`, `version`.
`engine::alerts::list`	List all configured alert rules and their current state.
`engine::alerts::evaluate`	Manually trigger evaluation of all alert rules.

Trigger Type: `log`

Config Field	Type	Description
`level`	string	Log level to subscribe to: `info`, `warn`, `error`, `debug`, or `trace`. When omitted, fires for all levels.

Sample Code

const fn = iii.registerFunction(
  'monitoring::onError',
  async (logEntry) => {
    await sendAlert({
      message: logEntry.body,
      severity: logEntry.severity_text,
      traceId: logEntry.trace_id,
    })
    return {}
  },
)

iii.registerTrigger({
  type: 'log',
  function_id: fn.id,
  config: { level: 'error' },
})

Log entry payload fields: timestamp_unix_nano, observed_timestamp_unix_nano, severity_number, severity_text, body, attributes, trace_id, span_id, resource, service_name, instrumentation_scope_name, instrumentation_scope_version.

api reference (json)

agent-api-reference.json

{
  "functions": [
    {
      "description": "Manually trigger alert evaluation",
      "metadata": {},
      "name": "engine::alerts::evaluate",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "AlertsEvaluateInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "List current alert states",
      "metadata": {},
      "name": "engine::alerts::list",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "AlertsListInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Get a baggage item value from the current context",
      "metadata": {},
      "name": "engine::baggage::get",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for baggage.get function",
        "properties": {
          "key": {
            "description": "The baggage key to retrieve",
            "type": "string"
          }
        },
        "required": [
          "key"
        ],
        "title": "BaggageGetInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Get all baggage items from the current context",
      "metadata": {},
      "name": "engine::baggage::get_all",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for baggage.getAll function (empty)",
        "title": "BaggageGetAllInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Set a baggage item value (returns new context, does not modify global)",
      "metadata": {},
      "name": "engine::baggage::set",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for baggage.set function",
        "properties": {
          "key": {
            "description": "The baggage key to set",
            "type": "string"
          },
          "value": {
            "description": "The baggage value to set",
            "type": "string"
          }
        },
        "required": [
          "key",
          "value"
        ],
        "title": "BaggageSetInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Check system health status",
      "metadata": {},
      "name": "engine::health::check",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "HealthCheckInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Log a debug message using OTEL",
      "metadata": {},
      "name": "engine::log::debug",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for OTEL log functions (log.info, log.warn, log.error)",
        "properties": {
          "data": {
            "description": "Additional structured data/attributes"
          },
          "message": {
            "description": "The log message",
            "type": "string"
          },
          "service_name": {
            "description": "Service name (defaults to function name if not provided)",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Optional span ID for correlation",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Optional trace ID for correlation",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "message"
        ],
        "title": "OtelLogInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Log an error message using OTEL",
      "metadata": {},
      "name": "engine::log::error",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for OTEL log functions (log.info, log.warn, log.error)",
        "properties": {
          "data": {
            "description": "Additional structured data/attributes"
          },
          "message": {
            "description": "The log message",
            "type": "string"
          },
          "service_name": {
            "description": "Service name (defaults to function name if not provided)",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Optional span ID for correlation",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Optional trace ID for correlation",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "message"
        ],
        "title": "OtelLogInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Log an info message using OTEL",
      "metadata": {},
      "name": "engine::log::info",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for OTEL log functions (log.info, log.warn, log.error)",
        "properties": {
          "data": {
            "description": "Additional structured data/attributes"
          },
          "message": {
            "description": "The log message",
            "type": "string"
          },
          "service_name": {
            "description": "Service name (defaults to function name if not provided)",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Optional span ID for correlation",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Optional trace ID for correlation",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "message"
        ],
        "title": "OtelLogInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Log a trace-level message using OTEL",
      "metadata": {},
      "name": "engine::log::trace",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for OTEL log functions (log.info, log.warn, log.error)",
        "properties": {
          "data": {
            "description": "Additional structured data/attributes"
          },
          "message": {
            "description": "The log message",
            "type": "string"
          },
          "service_name": {
            "description": "Service name (defaults to function name if not provided)",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Optional span ID for correlation",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Optional trace ID for correlation",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "message"
        ],
        "title": "OtelLogInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Log a warning message using OTEL",
      "metadata": {},
      "name": "engine::log::warn",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for OTEL log functions (log.info, log.warn, log.error)",
        "properties": {
          "data": {
            "description": "Additional structured data/attributes"
          },
          "message": {
            "description": "The log message",
            "type": "string"
          },
          "service_name": {
            "description": "Service name (defaults to function name if not provided)",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Optional span ID for correlation",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Optional trace ID for correlation",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "message"
        ],
        "title": "OtelLogInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Clear all stored OTEL logs",
      "metadata": {},
      "name": "engine::logs::clear",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "LogsClearInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "List stored OTEL logs",
      "metadata": {},
      "name": "engine::logs::list",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "end_time": {
            "description": "End time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "limit": {
            "description": "Maximum number of logs to return",
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "offset": {
            "description": "Pagination offset (default: 0)",
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "severity_min": {
            "description": "Minimum severity number (1-24, higher = more severe)",
            "format": "int32",
            "type": [
              "integer",
              "null"
            ]
          },
          "severity_text": {
            "description": "Filter by severity text (e.g., \"ERROR\", \"WARN\", \"INFO\")",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Filter by span ID",
            "type": [
              "string",
              "null"
            ]
          },
          "start_time": {
            "description": "Start time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "trace_id": {
            "description": "Filter by trace ID",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "title": "LogsListInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "List current metrics values",
      "metadata": {},
      "name": "engine::metrics::list",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "aggregate_interval": {
            "description": "Aggregate interval in seconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "end_time": {
            "description": "End time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "metric_name": {
            "description": "Filter by metric name",
            "type": [
              "string",
              "null"
            ]
          },
          "start_time": {
            "description": "Start time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          }
        },
        "title": "MetricsListInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Get pre-aggregated metrics rollups",
      "metadata": {},
      "name": "engine::rollups::list",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "end_time": {
            "description": "End time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "level": {
            "description": "Rollup level index (0 = 1 min, 1 = 5 min, 2 = 1 hour)",
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "metric_name": {
            "description": "Filter by metric name",
            "type": [
              "string",
              "null"
            ]
          },
          "start_time": {
            "description": "Start time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          }
        },
        "title": "RollupsListInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Get active sampling rules configuration",
      "metadata": {},
      "name": "engine::sampling::rules",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "LogsClearInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Clear all stored traces (only available when exporter is 'memory' or 'both')",
      "metadata": {},
      "name": "engine::traces::clear",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "TracesClearInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "List stored traces (only available when exporter is 'memory' or 'both')",
      "metadata": {},
      "name": "engine::traces::list",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "attributes": {
            "description": "Filter by span attributes (array of [key, value] pairs, AND logic, exact match)",
            "items": {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            "type": [
              "array",
              "null"
            ]
          },
          "end_time": {
            "description": "End time in unix timestamp milliseconds (include spans overlapping before this)",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "include_internal": {
            "default": null,
            "description": "Include internal engine traces (engine.* functions). Defaults to false.",
            "type": [
              "boolean",
              "null"
            ]
          },
          "limit": {
            "description": "Pagination limit (default: 100)",
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "max_duration_ms": {
            "description": "Maximum span duration in milliseconds (sub-ms precision)",
            "format": "double",
            "type": [
              "number",
              "null"
            ]
          },
          "min_duration_ms": {
            "description": "Minimum span duration in milliseconds (sub-ms precision)",
            "format": "double",
            "type": [
              "number",
              "null"
            ]
          },
          "name": {
            "description": "Filter by span name (case-insensitive substring match)",
            "type": [
              "string",
              "null"
            ]
          },
          "offset": {
            "description": "Pagination offset (default: 0)",
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "search_all_spans": {
            "default": null,
            "description": "Search across all spans in each trace, not just root spans. When true and a `name` filter is set, traces are matched if ANY span in the trace matches the name filter. Defaults to false.",
            "type": [
              "boolean",
              "null"
            ]
          },
          "service_name": {
            "description": "Filter by service name (case-insensitive substring match)",
            "type": [
              "string",
              "null"
            ]
          },
          "sort_by": {
            "description": "Sort field: \"duration\" | \"start_time\" | \"name\" (default: \"start_time\")",
            "type": [
              "string",
              "null"
            ]
          },
          "sort_order": {
            "description": "Sort order: \"asc\" | \"desc\" (default: \"asc\")",
            "type": [
              "string",
              "null"
            ]
          },
          "start_time": {
            "description": "Start time in unix timestamp milliseconds (include spans overlapping after this)",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "status": {
            "description": "Filter by status (case-insensitive substring match)",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Filter by specific trace ID",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "title": "TracesListInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Get trace tree with nested children (only available when exporter is 'memory' or 'both')",
      "metadata": {},
      "name": "engine::traces::tree",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "trace_id": {
            "description": "Trace ID to build the tree for",
            "type": "string"
          }
        },
        "required": [
          "trace_id"
        ],
        "title": "TracesTreeInput",
        "type": "object"
      },
      "response_schema": {}
    }
  ],
  "triggers": []
}

iii-observability

full markdown

install

configuration

dependencies

readme

iii-observability

Sample Configuration

Configuration

Alert Rule Fields

Advanced Sampling

Functions

Logging

Logs API

Traces API

Metrics API

Other APIs

Trigger Type: log

Sample Code

api reference (json)

Trigger Type: `log`