# memory-consolidate

> Scheduled hygiene for the memory worker — deterministic dedup of near-duplicate memories, applied supersede-only through the public memory functions, pinned untouchable, with catch-up-on-boot scheduling. Install, stop, or remove it without touching stored memory.

| field | value |
|-------|-------|
| version | 0.1.4 |
| type | binary |
| repo | https://github.com/iii-hq/workers |
| supported_targets | x86_64-apple-darwin, aarch64-apple-darwin, i686-pc-windows-msvc, x86_64-pc-windows-msvc, aarch64-pc-windows-msvc, x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu, x86_64-unknown-linux-musl, armv7-unknown-linux-gnueabihf |
| author | iii |

## installation

```sh
iii worker add memory-consolidate@0.1.4
```

## dependencies

- `memory` @ `^0.1.0`
- `configuration` @ `^0.21.3`
- `state` @ `^0.21.0`

## readme

# memory-consolidate

Scheduled hygiene sibling of the [`memory`](https://github.com/iii-hq/workers/tree/main/memory) worker. Finds near-duplicate memories in every bank and merges them: the duplicate retires with a `superseded_by` pointer (never a delete), the survivor absorbs the observation as corroboration. Strictly supersede-only through memory's public functions, pinned memories untouchable, every change visible as a `memory::item-changed` event. Install, stop, or remove it without touching stored memory.

## Install

```
iii worker add memory-consolidate
```

Requires the `memory` worker. Scheduling reuses the engine's cron trigger infrastructure: an hourly heartbeat binds `memory-consolidate::on-tick`, and the tick runs a pass only when `interval_hours` (default 24) have elapsed since the last one — the last completed pass persists in the state worker. **Catch-up-on-boot**: a pass missed while this worker was down runs shortly after boot instead of waiting for the next heartbeat; a slim backstop loop also keeps the schedule alive on rigs with no cron trigger owner.

## Quickstart

Plan without writing, then look at what a pass would do:

```bash
iii trigger memory-consolidate::run dry_run=true
iii trigger memory-consolidate::status
```

Apply for one bank only:

```bash
iii trigger memory-consolidate::run bank=blog
```

The report names every group: the surviving memory, the retired duplicates, and anything skipped because it is pinned. The retired records stay on disk and remain queryable with `include_superseded: true`.

## Configuration

All fields hot-reload through the `configuration` worker: `enabled`, `interval_hours`, `dry_run` (scheduled passes plan-only), `banks` (allowlist, empty = all), `max_supersedes_per_run` (safety cap; remainder waits for the next pass).

## The LLM tier (optional)

`llm_assist_enabled` (off by default) adds one `router::complete` judge call per bank after the deterministic pass:

- **Reorder groups** — word-order matches the deterministic pass surfaces report-only ("The deploy runs on Fridays" vs "On Fridays the deploy runs") are merged only when the judge confirms the meaning is identical; role swaps ("Alice manages Bob") stay untouched.
- **Rule promotion** — memories re-observed `promote_corroboration_threshold`+ times are offered as candidates; ones the judge classifies as standing instructions land as one-line entries in the bank's auto-managed `learned` rule, fingerprint-deduped, append-only. Hand-authored rules are never touched, and the judge can only act on candidates it was offered — it cannot invent targets.

Fail-soft: no router or a malformed reply just skips the tier (named in the report's errors); the deterministic pass has already completed.

## What counts as a duplicate

v1 is deterministic and deliberately conservative: two live memories merge only when their normalized text matches (case, punctuation, and whitespace insensitive) or their token sets are equal (word-order shuffles). One differing word is NOT a duplicate — "always publishes" and "never publishes" must never merge. Survivor choice is stable: pinned first, then highest corroboration, then the oldest record (it carries the original provenance). Semantic near-duplicate merging is a later, LLM-assisted tier.

## Boundaries

- Never touches memory's files; every mutation goes through `memory::supersede` and `memory::save` — the same append-only, last-wins contract as everything else.
- Pinned memories are never superseded, even inside a duplicate group.
- Not extraction (the memory worker captures), not recall, not rule management. One job: keep banks clean.

## api reference

```json
{
  "functions": [
    {
      "description": "Internal: hot-reload memory-consolidate from the authoritative configuration when it changes.",
      "metadata": {
        "internal": true,
        "trace_hidden": true
      },
      "name": "memory-consolidate::on-config-change",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "id": {
            "default": null,
            "type": [
              "string",
              "null"
            ]
          }
        },
        "title": "OnConfigChangeEvent",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "ok": {
            "type": "boolean"
          }
        },
        "required": [
          "ok"
        ],
        "title": "OnConfigChangeResponse",
        "type": "object"
      }
    },
    {
      "description": "Internal: schedule heartbeat (cron trigger + boot catch-up backstop). Runs a pass only when interval_hours have elapsed since the last one.",
      "metadata": {
        "internal": true,
        "trace_hidden": true
      },
      "name": "memory-consolidate::on-tick",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "The cron worker's tick payload (all advisory; the handler re-derives due-ness from persisted state). Typed so the published interface carries a real schema.",
        "properties": {
          "actual_time": {
            "default": null,
            "type": [
              "string",
              "null"
            ]
          },
          "job_id": {
            "default": null,
            "type": [
              "string",
              "null"
            ]
          },
          "scheduled_time": {
            "default": null,
            "type": [
              "string",
              "null"
            ]
          },
          "trigger": {
            "default": null,
            "type": [
              "string",
              "null"
            ]
          }
        },
        "title": "TickInput",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "ran": {
            "description": "A pass actually ran (false = not due yet or disabled).",
            "type": "boolean"
          }
        },
        "required": [
          "ran"
        ],
        "title": "TickResponse",
        "type": "object"
      }
    },
    {
      "description": "Run one consolidation pass now (all configured banks, or one bank). Deterministic near-duplicate dedup applied strictly through memory::supersede + memory::save; pinned memories are untouchable. dry_run plans without writing.",
      "metadata": {},
      "name": "memory-consolidate::run",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "bank": {
            "default": null,
            "description": "Consolidate only this bank (must be in the configured allowlist, when one is set).",
            "type": [
              "string",
              "null"
            ]
          },
          "dry_run": {
            "default": null,
            "description": "Override the configured dry_run for this call.",
            "type": [
              "boolean",
              "null"
            ]
          }
        },
        "title": "RunRequest",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "BankReport": {
            "properties": {
              "bank": {
                "type": "string"
              },
              "capped": {
                "description": "Work left behind by the per-run cap; the next pass picks it up.",
                "format": "uint",
                "minimum": 0,
                "type": "integer"
              },
              "errors": {
                "items": {
                  "type": "string"
                },
                "type": "array"
              },
              "groups": {
                "items": {
                  "$ref": "#/definitions/PlannedGroup"
                },
                "type": "array"
              },
              "llm_merged": {
                "description": "Reorder groups merged after the LLM judge confirmed equivalence.",
                "format": "uint",
                "minimum": 0,
                "type": "integer"
              },
              "promoted": {
                "description": "Standing-instruction lines the judge promoted into the bank's `learned` rule.",
                "format": "uint",
                "minimum": 0,
                "type": "integer"
              },
              "reinforced": {
                "description": "Winner reinforcements applied (one per absorbed duplicate).",
                "format": "uint",
                "minimum": 0,
                "type": "integer"
              },
              "scanned": {
                "description": "Live memories scanned.",
                "format": "uint",
                "minimum": 0,
                "type": "integer"
              },
              "superseded": {
                "description": "Supersedes actually applied (0 on dry runs).",
                "format": "uint",
                "minimum": 0,
                "type": "integer"
              }
            },
            "required": [
              "bank",
              "groups",
              "reinforced",
              "scanned",
              "superseded"
            ],
            "type": "object"
          },
          "PlannedGroup": {
            "description": "One planned merge: `losers` retire in favor of `winner`.",
            "properties": {
              "loser_ids": {
                "items": {
                  "type": "string"
                },
                "type": "array"
              },
              "loser_texts": {
                "description": "Texts of the losers, aligned with `loser_ids` (the judge tier and human review need the words, not just ids).",
                "items": {
                  "type": "string"
                },
                "type": "array"
              },
              "report_only": {
                "description": "True = surfaced for review only (token-set match: same words, different order — potentially different meaning). Never written automatically.",
                "type": "boolean"
              },
              "skipped_pinned": {
                "description": "Losers left alone because they are pinned (pinned is untouchable).",
                "items": {
                  "type": "string"
                },
                "type": "array"
              },
              "winner_id": {
                "type": "string"
              },
              "winner_text": {
                "type": "string"
              }
            },
            "required": [
              "loser_ids",
              "loser_texts",
              "winner_id",
              "winner_text"
            ],
            "type": "object"
          }
        },
        "properties": {
          "banks": {
            "items": {
              "$ref": "#/definitions/BankReport"
            },
            "type": "array"
          },
          "checkpointed": {
            "description": "The pass completed cleanly AND its last-run checkpoint persisted. False = the schedule will retry the pass next check.",
            "type": "boolean"
          },
          "dry_run": {
            "type": "boolean"
          },
          "superseded": {
            "description": "Total supersedes applied across banks this pass.",
            "format": "uint",
            "minimum": 0,
            "type": "integer"
          }
        },
        "required": [
          "banks",
          "checkpointed",
          "dry_run",
          "superseded"
        ],
        "title": "RunResponse",
        "type": "object"
      }
    },
    {
      "description": "Schedule and last-pass report: enabled, interval, last run, whether a pass is due, and the most recent per-bank results.",
      "metadata": {},
      "name": "memory-consolidate::status",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "StatusRequest",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "dry_run": {
            "type": "boolean"
          },
          "due": {
            "description": "A scheduled pass is overdue right now.",
            "type": "boolean"
          },
          "enabled": {
            "type": "boolean"
          },
          "interval_hours": {
            "format": "uint64",
            "minimum": 0,
            "type": "integer"
          },
          "last_report": true,
          "last_run": {
            "description": "Milliseconds since epoch of the last completed pass; 0 = never.",
            "format": "uint64",
            "minimum": 0,
            "type": "integer"
          }
        },
        "required": [
          "dry_run",
          "due",
          "enabled",
          "interval_hours",
          "last_run"
        ],
        "title": "StatusResponse",
        "type": "object"
      }
    }
  ],
  "triggers": []
}
```