memory-consolidate
v0.1.9Scheduled hygiene for the memory worker — deterministic dedup of near-duplicate memories, applied supersede-only through the public memory functions, pinned untouchable, with catch-up-on-boot scheduling. Install, stop, or remove it without touching stored memory.
iiiverified- macOS: arm64 · x64
- Linux: arm64 · armv7 · x64
- Windows: arm64 · x64 · x86
exact versions are immutable; binary and bundle artifacts are digest-pinned.
full markdown
/workers/memory-consolidate.md. paste it into an llm prompt or pipe it through curl from a worker.install
dependencies
readme
memory-consolidate
Scheduled hygiene sibling of the memory worker. Finds near-duplicate memories in every bank and merges them: the duplicate retires with a superseded_by pointer (never a delete), the survivor absorbs the observation as corroboration. Strictly supersede-only through memory's public functions, pinned memories untouchable, every change visible as a memory::item-changed event. Install, stop, or remove it without touching stored memory.
Install
iii trigger compose::add worker=memory-consolidateRequires the memory worker. Scheduling reuses the engine's cron trigger infrastructure: an hourly heartbeat binds memory-consolidate::on-tick, and the tick runs a pass only when interval_hours (default 24) have elapsed since the last one — the last completed pass persists in the state worker. Catch-up-on-boot: a pass missed while this worker was down runs shortly after boot instead of waiting for the next heartbeat; a slim backstop loop also keeps the schedule alive on rigs with no cron trigger owner.
Quickstart
Plan without writing, then look at what a pass would do:
iii trigger memory-consolidate::run dry_run=true
iii trigger memory-consolidate::statusApply for one bank only:
iii trigger memory-consolidate::run bank=blogThe report names every group: the surviving memory, the retired duplicates, and anything skipped because it is pinned. The retired records stay on disk and remain queryable with include_superseded: true.
Configuration
All fields hot-reload through the configuration worker: enabled, interval_hours, dry_run (scheduled passes plan-only), banks (allowlist, empty = all), max_supersedes_per_run (safety cap; remainder waits for the next pass).
The LLM tier (optional)
llm_assist_enabled (off by default) adds one router::complete judge call per bank after the deterministic pass:
- Reorder groups — word-order matches the deterministic pass surfaces report-only ("The deploy runs on Fridays" vs "On Fridays the deploy runs") are merged only when the judge confirms the meaning is identical; role swaps ("Alice manages Bob") stay untouched.
- Rule promotion — memories re-observed
promote_corroboration_threshold+ times are offered as candidates; ones the judge classifies as standing instructions land as one-line entries in the bank's auto-managedlearnedrule, fingerprint-deduped, append-only. Hand-authored rules are never touched, and the judge can only act on candidates it was offered — it cannot invent targets.
Fail-soft: no router or a malformed reply just skips the tier (named in the report's errors); the deterministic pass has already completed.
What counts as a duplicate
v1 is deterministic and deliberately conservative: two live memories merge only when their normalized text matches (case, punctuation, and whitespace insensitive) or their token sets are equal (word-order shuffles). One differing word is NOT a duplicate — "always publishes" and "never publishes" must never merge. Survivor choice is stable: pinned first, then highest corroboration, then the oldest record (it carries the original provenance). Semantic near-duplicate merging is a later, LLM-assisted tier.
Boundaries
- Never touches memory's files; every mutation goes through
memory::supersedeandmemory::save— the same append-only, last-wins contract as everything else. - Pinned memories are never superseded, even inside a duplicate group.
- Not extraction (the memory worker captures), not recall, not rule management. One job: keep banks clean.
api reference (json)
{
"functions": [
{
"description": "Internal: hot-reload memory-consolidate from the authoritative configuration when it changes.",
"metadata": {
"internal": true,
"trace_hidden": true
},
"name": "memory-consolidate::on-config-change",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"id": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "OnConfigChangeEvent",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"ok": {
"type": "boolean"
}
},
"required": [
"ok"
],
"title": "OnConfigChangeResponse",
"type": "object"
}
},
{
"description": "Internal: schedule heartbeat (cron trigger + boot catch-up backstop). Runs a pass only when interval_hours have elapsed since the last one.",
"metadata": {
"internal": true,
"trace_hidden": true
},
"name": "memory-consolidate::on-tick",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "The cron worker's tick payload (all advisory; the handler re-derives due-ness from persisted state). Typed so the published interface carries a real schema.",
"properties": {
"actual_time": {
"default": null,
"type": [
"string",
"null"
]
},
"job_id": {
"default": null,
"type": [
"string",
"null"
]
},
"scheduled_time": {
"default": null,
"type": [
"string",
"null"
]
},
"trigger": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "TickInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"ran": {
"description": "A pass actually ran (false = not due yet or disabled).",
"type": "boolean"
}
},
"required": [
"ran"
],
"title": "TickResponse",
"type": "object"
}
},
{
"description": "Run one consolidation pass now (all configured banks, or one bank). Deterministic near-duplicate dedup applied strictly through memory::supersede + memory::save; pinned memories are untouchable. dry_run plans without writing.",
"metadata": {},
"name": "memory-consolidate::run",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"bank": {
"default": null,
"description": "Consolidate only this bank (must be in the configured allowlist, when one is set).",
"type": [
"string",
"null"
]
},
"dry_run": {
"default": null,
"description": "Override the configured dry_run for this call.",
"type": [
"boolean",
"null"
]
}
},
"title": "RunRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"BankReport": {
"properties": {
"bank": {
"type": "string"
},
"capped": {
"description": "Work left behind by the per-run cap; the next pass picks it up.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"errors": {
"items": {
"type": "string"
},
"type": "array"
},
"groups": {
"items": {
"$ref": "#/definitions/PlannedGroup"
},
"type": "array"
},
"llm_merged": {
"description": "Reorder groups merged after the LLM judge confirmed equivalence.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"promoted": {
"description": "Standing-instruction lines the judge promoted into the bank's `learned` rule.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"reinforced": {
"description": "Winner reinforcements applied (one per absorbed duplicate).",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"scanned": {
"description": "Live memories scanned.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"superseded": {
"description": "Supersedes actually applied (0 on dry runs).",
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"bank",
"groups",
"reinforced",
"scanned",
"superseded"
],
"type": "object"
},
"PlannedGroup": {
"description": "One planned merge: `losers` retire in favor of `winner`.",
"properties": {
"loser_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"loser_texts": {
"description": "Texts of the losers, aligned with `loser_ids` (the judge tier and human review need the words, not just ids).",
"items": {
"type": "string"
},
"type": "array"
},
"report_only": {
"description": "True = surfaced for review only (token-set match: same words, different order — potentially different meaning). Never written automatically.",
"type": "boolean"
},
"skipped_pinned": {
"description": "Losers left alone because they are pinned (pinned is untouchable).",
"items": {
"type": "string"
},
"type": "array"
},
"winner_id": {
"type": "string"
},
"winner_text": {
"type": "string"
}
},
"required": [
"loser_ids",
"loser_texts",
"winner_id",
"winner_text"
],
"type": "object"
}
},
"properties": {
"banks": {
"items": {
"$ref": "#/definitions/BankReport"
},
"type": "array"
},
"checkpointed": {
"description": "The pass completed cleanly AND its last-run checkpoint persisted. False = the schedule will retry the pass next check.",
"type": "boolean"
},
"dry_run": {
"type": "boolean"
},
"superseded": {
"description": "Total supersedes applied across banks this pass.",
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"banks",
"checkpointed",
"dry_run",
"superseded"
],
"title": "RunResponse",
"type": "object"
}
},
{
"description": "Schedule and last-pass report: enabled, interval, last run, whether a pass is due, and the most recent per-bank results.",
"metadata": {},
"name": "memory-consolidate::status",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "StatusRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"dry_run": {
"type": "boolean"
},
"due": {
"description": "A scheduled pass is overdue right now.",
"type": "boolean"
},
"enabled": {
"type": "boolean"
},
"interval_hours": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"last_report": true,
"last_run": {
"description": "Milliseconds since epoch of the last completed pass; 0 = never.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"dry_run",
"due",
"enabled",
"interval_hours",
"last_run"
],
"title": "StatusResponse",
"type": "object"
}
}
],
"triggers": []
}