# iii-observability

> OpenTelemetry-based traces, metrics, logs, alerts, and sampling.

| field | value |
|-------|-------|
| version | 0.19.4 |
| type | engine |
| repo | https://github.com/iii-hq/iii |
| author | iii |

## installation

```sh
iii worker add iii-observability@0.19.4
```

## configuration

```yaml
- enabled: true
  exporter: memory
  logs_console_output: true
  logs_enabled: true
  logs_exporter: memory
  logs_max_count: 1000
  logs_retention_seconds: 3600
  memory_max_spans: 1000000
  metrics_enabled: true
  metrics_exporter: memory
  metrics_max_count: 10000
  metrics_retention_seconds: 3600
  sampling_ratio: 1
  service_name: iii
  service_version: ${SERVICE_VERSION:__III_ENGINE_VERSION__}
```

## readme

# iii-observability

Full OpenTelemetry observability for III Engine: distributed tracing, structured logs, performance
metrics, alert rules, and trace sampling — all queryable via built-in functions.

## Install

```bash
iii worker add iii-observability
```

Resolves from the worker registry at [workers.iii.dev](https://workers.iii.dev/).

## Skills

Install the `iii-observability` agent skill for Claude Code, Cursor, and 30+ other agents:

```bash
npx skills add iii-hq/iii --full-depth --skill iii-observability
```

## Sample Configuration

```yaml
- name: iii-observability
  config:
    enabled: true
    service_name: my-service
    service_version: 1.0.0
    exporter: memory
    metrics_enabled: true
    logs_enabled: true
    memory_max_spans: 1000
    sampling_ratio: 1.0
    alerts:
      - name: high-error-rate
        metric: iii.invocations.error
        threshold: 10
        operator: ">"
        window_seconds: 60
        action:
          type: log
```

## Configuration

| Field                       | Type        | Description                                                                                                                                                                  |
| --------------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enabled`                   | boolean     | Whether OpenTelemetry tracing export is enabled. Defaults to `false`. Env: `OTEL_ENABLED`.                                                                                   |
| `service_name`              | string      | Service name in traces and metrics. Defaults to `"iii"`. Env: `OTEL_SERVICE_NAME`.                                                                                           |
| `service_version`           | string      | Service version (`service.version` OTEL attribute). Env: `SERVICE_VERSION`.                                                                                                  |
| `service_namespace`         | string      | Service namespace (`service.namespace` OTEL attribute). Env: `SERVICE_NAMESPACE`.                                                                                            |
| `exporter`                  | string      | Trace exporter: `memory`, `otlp`, or `both`. Use `both` when traces should remain queryable in iii while also being exported. Defaults to `otlp`. Env: `OTEL_EXPORTER_TYPE`. |
| `endpoint`                  | string      | OTLP collector base endpoint. Defaults to `"http://localhost:4317"`. Env: `OTEL_EXPORTER_OTLP_ENDPOINT`.                                                                     |
| `sampling_ratio`            | number      | Global trace sampling ratio (`0.0`–`1.0`). Defaults to `1.0`. Env: `OTEL_TRACES_SAMPLER_ARG`.                                                                                |
| `memory_max_spans`          | number      | Max spans to keep in memory. Defaults to `1000`. Env: `OTEL_MEMORY_MAX_SPANS`.                                                                                               |
| `metrics_enabled`           | boolean     | Whether metrics collection is enabled. Defaults to `false`. Env: `OTEL_METRICS_ENABLED`.                                                                                     |
| `metrics_exporter`          | string      | Metrics exporter: `memory` or `otlp`. Defaults to `memory`. Env: `OTEL_METRICS_EXPORTER`.                                                                                    |
| `metrics_retention_seconds` | number      | How long to retain metrics in memory. Defaults to `3600`. Env: `OTEL_METRICS_RETENTION_SECONDS`.                                                                             |
| `metrics_max_count`         | number      | Max metric data points in memory. Defaults to `10000`. Env: `OTEL_METRICS_MAX_COUNT`.                                                                                        |
| `logs_enabled`              | boolean     | Whether structured log storage is enabled.                                                                                                                                   |
| `logs_exporter`             | string      | Logs exporter: `memory`, `otlp`, or `both`. Use `both` when logs should remain queryable in iii while also being exported. Defaults to `memory`. Env: `OTEL_LOGS_EXPORTER`.  |
| `logs_max_count`            | number      | Max log entries in memory. Defaults to `1000`.                                                                                                                               |
| `logs_retention_seconds`    | number      | How long to retain logs in memory. Defaults to `3600`.                                                                                                                       |
| `logs_sampling_ratio`       | number      | Fraction of logs to retain (`0.0`–`1.0`). Defaults to `1.0`.                                                                                                                 |
| `logs_console_output`       | boolean     | Print ingested logs to the console. Defaults to `true`.                                                                                                                      |
| `level`                     | string      | Minimum log level: `trace`, `debug`, `info`, `warn`, `error`. Defaults to `info`.                                                                                            |
| `format`                    | string      | Log output format: `default` or `json`. Defaults to `default`.                                                                                                               |
| `alerts`                    | AlertRule[] | Alert rules evaluated against metrics.                                                                                                                                       |

### OTLP Transport

Traces and metrics export over OTLP/gRPC by default. `https://` endpoints use TLS with system roots,
while `http://` endpoints use cleartext transport.

To export traces and metrics with OTLP/HTTP protobuf instead of gRPC, set the standard OpenTelemetry
protocol environment variable before starting the engine:

```bash
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
```

Signal-specific protocol variables override the global value:

```bash
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_METRICS_PROTOCOL=http/protobuf
```

When HTTP/protobuf is selected, iii treats `endpoint` as the collector base URL and appends the
signal path when needed:

- traces: `/v1/traces`
- metrics: `/v1/metrics`

The logs exporter sends OTLP logs over HTTP and posts to `/v1/logs`.

Collectors that require authentication or routing headers can use the standard OTLP headers
environment variables:

```bash
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer $OTLP_TOKEN"
```

Use signal-specific headers when one signal needs different values:

- `OTEL_EXPORTER_OTLP_TRACES_HEADERS`
- `OTEL_EXPORTER_OTLP_METRICS_HEADERS`
- `OTEL_EXPORTER_OTLP_LOGS_HEADERS`

The logs exporter reads `OTEL_EXPORTER_OTLP_LOGS_HEADERS` first and falls back to
`OTEL_EXPORTER_OTLP_HEADERS`. Keep tokens in environment variables or a secret manager; do not
commit them to config files.

To keep the iii console useful while exporting to an external collector, use `exporter: both` for
traces and `logs_exporter: both` for logs.

### Alert Rule Fields

| Field              | Type        | Description                                                                                             |
| ------------------ | ----------- | ------------------------------------------------------------------------------------------------------- |
| `name`             | string      | Required. Unique alert rule name.                                                                       |
| `metric`           | string      | Required. Metric name to monitor (e.g., `iii.invocations.error`).                                       |
| `threshold`        | number      | Required. Threshold value.                                                                              |
| `operator`         | string      | Comparison operator: `>`, `>=`, `<`, `<=`, `==`, `!=`. Defaults to `>`.                                 |
| `window_seconds`   | number      | Time window in seconds for metric evaluation. Defaults to `60`.                                         |
| `cooldown_seconds` | number      | Minimum interval between alert fires. Defaults to `60`.                                                 |
| `enabled`          | boolean     | Whether the alert rule is active. Defaults to `true`.                                                   |
| `action`           | AlertAction | `{ "type": "log" }`, `{ "type": "webhook", "url": "..." }`, or `{ "type": "function", "path": "..." }`. |

### Advanced Sampling

```yaml
sampling:
  default: 1.0
  parent_based: true
  rules:
    - operation: "api.*"
      rate: 0.1
  rate_limit:
    max_traces_per_second: 100
```

## Functions

### Logging

| Function             | Description                   |
| -------------------- | ----------------------------- |
| `engine::log::info`  | Log an informational message. |
| `engine::log::warn`  | Log a warning message.        |
| `engine::log::error` | Log an error message.         |
| `engine::log::debug` | Log a debug message.          |
| `engine::log::trace` | Log a trace-level message.    |

All logging functions accept: `message` (string, required), `data` (object), `trace_id` (string),
`span_id` (string), `service_name` (string).

### Logs API

| Function              | Description                                                                                                                             |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `engine::logs::list`  | Query stored log entries. Filters: `start_time`, `end_time`, `trace_id`, `span_id`, `severity_min`, `severity_text`, `offset`, `limit`. |
| `engine::logs::clear` | Clear all stored log entries from memory.                                                                                               |

### Traces API

| Function                | Description                                                                                                                                                                                                             |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `engine::traces::list`  | List stored spans. Filters: `trace_id`, `service_name`, `name`, `status`, `min_duration_ms`, `max_duration_ms`, `start_time`, `end_time`, `sort_by`, `sort_order`, `attributes`, `include_internal`, `offset`, `limit`. |
| `engine::traces::tree`  | Retrieve a trace as a hierarchical span tree. Parameters: `trace_id` (required).                                                                                                                                        |
| `engine::traces::clear` | Clear all stored trace spans from memory.                                                                                                                                                                               |

### Metrics API

| Function                | Description                                                                                                                                                 |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `engine::metrics::list` | List metrics with aggregated statistics. Returns engine counters (invocations, workers, performance), SDK metrics, and optional time-bucketed aggregations. |
| `engine::rollups::list` | List metric rollup aggregations (1-minute, 5-minute, 1-hour windows).                                                                                       |

### Other APIs

| Function                   | Description                                                                         |
| -------------------------- | ----------------------------------------------------------------------------------- |
| `engine::baggage::get`     | Get a baggage value from the current trace context.                                 |
| `engine::baggage::set`     | Set a baggage value in the current trace context.                                   |
| `engine::baggage::get_all` | Get all baggage key-value pairs.                                                    |
| `engine::sampling::rules`  | List all active sampling rules.                                                     |
| `engine::health::check`    | Check engine health status. Returns `status`, `components`, `timestamp`, `version`. |
| `engine::alerts::list`     | List all configured alert rules and their current state.                            |
| `engine::alerts::evaluate` | Manually trigger evaluation of all alert rules.                                     |

## Trigger Type: `log`

Register a function to react to log entries as they are produced.

| Config Field | Type   | Description                                                                                                  |
| ------------ | ------ | ------------------------------------------------------------------------------------------------------------ |
| `level`      | string | Log level to subscribe to: `info`, `warn`, `error`, `debug`, or `trace`. When omitted, fires for all levels. |

### Sample Code

```typescript
const fn = iii.registerFunction("monitoring::onError", async (logEntry) => {
  await sendAlert({
    message: logEntry.body,
    severity: logEntry.severity_text,
    traceId: logEntry.trace_id,
  });
  return {};
});

iii.registerTrigger({
  type: "log",
  function_id: fn.id,
  config: { level: "error" },
});
```

Log entry payload fields: `timestamp_unix_nano`, `observed_timestamp_unix_nano`, `severity_number`,
`severity_text`, `body`, `attributes`, `trace_id`, `span_id`, `resource`, `service_name`,
`instrumentation_scope_name`, `instrumentation_scope_version`.

## Trigger Type: `trace`

Register a function to react to span activity in the in-memory trace store, so any client — a worker
or the web console — can refresh reactively instead of polling. The trigger is a **coalesced "traces
changed" tick**, not a per-span feed: span activity is debounced (~300ms) and the handler receives
the distinct affected trace ids for the window. Re-read details via `engine::traces::list` /
`engine::traces::tree`. Requires the memory exporter (`exporter: memory` or `both`); with the
OTLP-only exporter there is no in-memory store and trace triggers stay dormant.

Engine-internal spans and the trigger's **own delivery spans** are excluded from firing it —
delivering a trigger via `engine.call` is itself instrumented as a span, so without this exclusion
the trigger would re-fire on its own output (an unbounded feedback loop). This is why a span trigger
differs from the `log` trigger, whose delivery produces spans, not logs.

| Config Field   | Type   | Description                                                                                                                                               |
| -------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `service_name` | string | Only fire for activity from this service. When omitted, fires for any service. Compared case-insensitively.                                               |
| `status`       | string | Only fire when a span with this status (`ok`, `error`, or `unset`) landed in the window. When omitted, fires for any status. Compared case-insensitively. |

Both filters are ANDed; omit both to fire on any span activity.

### Sample Code

```typescript
const fn = iii.registerFunction("devtools::onTracesChanged", async ({ trace_ids }) => {
  // A "refetch soon" beat — re-read the traces you care about.
  await refreshTraceViews(trace_ids);
  return {};
});

iii.registerTrigger({
  type: "trace",
  function_id: fn.id,
  config: {}, // or { status: 'error' }
});
```

Handler payload: `{ "trace_ids": string[] }` — the distinct trace ids with span activity in the
coalesced window.

## api reference

```json
{
  "functions": [
    {
      "description": "Manually trigger alert evaluation",
      "metadata": {},
      "name": "engine::alerts::evaluate",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "AlertsEvaluateInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "List current alert states",
      "metadata": {},
      "name": "engine::alerts::list",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "AlertsListInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Get a baggage item value from the current context",
      "metadata": {},
      "name": "engine::baggage::get",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for baggage.get function",
        "properties": {
          "key": {
            "description": "The baggage key to retrieve",
            "type": "string"
          }
        },
        "required": [
          "key"
        ],
        "title": "BaggageGetInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Get all baggage items from the current context",
      "metadata": {},
      "name": "engine::baggage::get_all",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for baggage.getAll function (empty)",
        "title": "BaggageGetAllInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Set a baggage item value (returns new context, does not modify global)",
      "metadata": {},
      "name": "engine::baggage::set",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for baggage.set function",
        "properties": {
          "key": {
            "description": "The baggage key to set",
            "type": "string"
          },
          "value": {
            "description": "The baggage value to set",
            "type": "string"
          }
        },
        "required": [
          "key",
          "value"
        ],
        "title": "BaggageSetInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Check system health status",
      "metadata": {},
      "name": "engine::health::check",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "HealthCheckInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Log a debug message using OTEL",
      "metadata": {},
      "name": "engine::log::debug",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for OTEL log functions (log.info, log.warn, log.error)",
        "properties": {
          "data": {
            "description": "Additional structured data/attributes"
          },
          "message": {
            "description": "The log message",
            "type": "string"
          },
          "service_name": {
            "description": "Service name (defaults to function name if not provided)",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Optional span ID for correlation",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Optional trace ID for correlation",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "message"
        ],
        "title": "OtelLogInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Log an error message using OTEL",
      "metadata": {},
      "name": "engine::log::error",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for OTEL log functions (log.info, log.warn, log.error)",
        "properties": {
          "data": {
            "description": "Additional structured data/attributes"
          },
          "message": {
            "description": "The log message",
            "type": "string"
          },
          "service_name": {
            "description": "Service name (defaults to function name if not provided)",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Optional span ID for correlation",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Optional trace ID for correlation",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "message"
        ],
        "title": "OtelLogInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Log an info message using OTEL",
      "metadata": {},
      "name": "engine::log::info",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for OTEL log functions (log.info, log.warn, log.error)",
        "properties": {
          "data": {
            "description": "Additional structured data/attributes"
          },
          "message": {
            "description": "The log message",
            "type": "string"
          },
          "service_name": {
            "description": "Service name (defaults to function name if not provided)",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Optional span ID for correlation",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Optional trace ID for correlation",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "message"
        ],
        "title": "OtelLogInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Log a trace-level message using OTEL",
      "metadata": {},
      "name": "engine::log::trace",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for OTEL log functions (log.info, log.warn, log.error)",
        "properties": {
          "data": {
            "description": "Additional structured data/attributes"
          },
          "message": {
            "description": "The log message",
            "type": "string"
          },
          "service_name": {
            "description": "Service name (defaults to function name if not provided)",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Optional span ID for correlation",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Optional trace ID for correlation",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "message"
        ],
        "title": "OtelLogInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Log a warning message using OTEL",
      "metadata": {},
      "name": "engine::log::warn",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Input for OTEL log functions (log.info, log.warn, log.error)",
        "properties": {
          "data": {
            "description": "Additional structured data/attributes"
          },
          "message": {
            "description": "The log message",
            "type": "string"
          },
          "service_name": {
            "description": "Service name (defaults to function name if not provided)",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Optional span ID for correlation",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Optional trace ID for correlation",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "message"
        ],
        "title": "OtelLogInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Clear all stored OTEL logs",
      "metadata": {},
      "name": "engine::logs::clear",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "LogsClearInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "List stored OTEL logs",
      "metadata": {},
      "name": "engine::logs::list",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "end_time": {
            "description": "End time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "limit": {
            "description": "Maximum number of logs to return",
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "offset": {
            "description": "Pagination offset (default: 0)",
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "severity_min": {
            "description": "Minimum severity number (1-24, higher = more severe)",
            "format": "int32",
            "type": [
              "integer",
              "null"
            ]
          },
          "severity_text": {
            "description": "Filter by severity text (e.g., \"ERROR\", \"WARN\", \"INFO\")",
            "type": [
              "string",
              "null"
            ]
          },
          "span_id": {
            "description": "Filter by span ID",
            "type": [
              "string",
              "null"
            ]
          },
          "start_time": {
            "description": "Start time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "trace_id": {
            "description": "Filter by trace ID",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "title": "LogsListInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "List current metrics values",
      "metadata": {},
      "name": "engine::metrics::list",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "aggregate_interval": {
            "description": "Aggregate interval in seconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "end_time": {
            "description": "End time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "metric_name": {
            "description": "Filter by metric name",
            "type": [
              "string",
              "null"
            ]
          },
          "start_time": {
            "description": "Start time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          }
        },
        "title": "MetricsListInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Get pre-aggregated metrics rollups",
      "metadata": {},
      "name": "engine::rollups::list",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "end_time": {
            "description": "End time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "level": {
            "description": "Rollup level index (0 = 1 min, 1 = 5 min, 2 = 1 hour)",
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "metric_name": {
            "description": "Filter by metric name",
            "type": [
              "string",
              "null"
            ]
          },
          "start_time": {
            "description": "Start time in Unix timestamp milliseconds",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          }
        },
        "title": "RollupsListInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Get active sampling rules configuration",
      "metadata": {},
      "name": "engine::sampling::rules",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "LogsClearInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Clear all stored traces (only available when exporter is 'memory' or 'both')",
      "metadata": {},
      "name": "engine::traces::clear",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "TracesClearInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Group stored spans by an attribute value (only available when exporter is 'memory' or 'both')",
      "metadata": {},
      "name": "engine::traces::group_by",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "attribute": {
            "description": "Span attribute key to group by. Spans without this attribute are skipped.",
            "type": "string"
          },
          "include_internal": {
            "default": null,
            "description": "Include engine-internal spans. Defaults to false, matching `traces::list`.",
            "type": [
              "boolean",
              "null"
            ]
          },
          "limit": {
            "default": null,
            "description": "Max groups returned after sorting by `first_seen_ms` descending. Default 100.",
            "format": "uint32",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "since_ms": {
            "default": null,
            "description": "Earliest end_time (ms since epoch) to include.",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          }
        },
        "required": [
          "attribute"
        ],
        "title": "TracesGroupByInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "List stored traces (only available when exporter is 'memory' or 'both')",
      "metadata": {},
      "name": "engine::traces::list",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "attributes": {
            "description": "Filter by span attributes (array of [key, value] pairs, AND logic, exact match)",
            "items": {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            "type": [
              "array",
              "null"
            ]
          },
          "end_time": {
            "description": "End time in unix timestamp milliseconds (include spans overlapping before this)",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "include_internal": {
            "default": null,
            "description": "Include internal engine traces (engine.* functions). Defaults to false.",
            "type": [
              "boolean",
              "null"
            ]
          },
          "limit": {
            "description": "Pagination limit (default: 100)",
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "max_duration_ms": {
            "description": "Maximum span duration in milliseconds (sub-ms precision)",
            "format": "double",
            "type": [
              "number",
              "null"
            ]
          },
          "min_duration_ms": {
            "description": "Minimum span duration in milliseconds (sub-ms precision)",
            "format": "double",
            "type": [
              "number",
              "null"
            ]
          },
          "name": {
            "description": "Filter by span name (case-insensitive substring match)",
            "type": [
              "string",
              "null"
            ]
          },
          "offset": {
            "description": "Pagination offset (default: 0)",
            "format": "uint",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "search_all_spans": {
            "default": null,
            "description": "Search across all spans in each trace, not just root spans. When true and a `name` filter is set, traces are matched if ANY span in the trace matches the name filter. Defaults to false.",
            "type": [
              "boolean",
              "null"
            ]
          },
          "service_name": {
            "description": "Filter by service name (case-insensitive substring match)",
            "type": [
              "string",
              "null"
            ]
          },
          "sort_by": {
            "description": "Sort field: \"start_time\" | \"duration\" (alias \"duration_ms\") | \"service_name\" | \"name\" (default: \"start_time\"). Unknown values fall back to \"start_time\".",
            "type": [
              "string",
              "null"
            ]
          },
          "sort_order": {
            "description": "Sort order: \"asc\" | \"desc\" (default: \"asc\")",
            "type": [
              "string",
              "null"
            ]
          },
          "start_time": {
            "description": "Start time in unix timestamp milliseconds (include spans overlapping after this)",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "status": {
            "description": "Filter by status (case-insensitive substring match)",
            "type": [
              "string",
              "null"
            ]
          },
          "trace_id": {
            "description": "Filter by specific trace ID",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "title": "TracesListInput",
        "type": "object"
      },
      "response_schema": {}
    },
    {
      "description": "Get trace tree with nested children (only available when exporter is 'memory' or 'both')",
      "metadata": {},
      "name": "engine::traces::tree",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "trace_id": {
            "description": "Trace ID to build the tree for",
            "type": "string"
          }
        },
        "required": [
          "trace_id"
        ],
        "title": "TracesTreeInput",
        "type": "object"
      },
      "response_schema": {}
    }
  ],
  "triggers": []
}
```
