memory-consolidate
v0.1.2Scheduled hygiene for the memory worker — deterministic dedup of near-duplicate memories, applied supersede-only through the public memory functions, pinned untouchable, with catch-up-on-boot scheduling. Install, stop, or remove it without touching stored memory.
Published as experimental by its release pipeline. It installs and resolves like any other worker, but its interface may change without notice.
iiiverified- macOS: arm64 · x64
- Linux: arm64 · armv7 · x64
- Windows: arm64 · x64 · x86
exact versions are immutable; binary and bundle artifacts are digest-pinned.
full markdown
/workers/memory-consolidate.md?version=0.1.2. paste it into an llm prompt or pipe it through curl from a worker.install
dependencies
readme
memory-consolidate
Scheduled hygiene sibling of the memory worker. Finds near-duplicate memories in every bank and merges them: the duplicate retires with a superseded_by pointer (never a delete), the survivor absorbs the observation as corroboration. Strictly supersede-only through memory's public functions, pinned memories untouchable, every change visible as a memory::item-changed event. Install, stop, or remove it without touching stored memory.
Install
iii worker add memory-consolidateRequires the memory worker. Scheduling reuses the engine's cron trigger infrastructure: an hourly heartbeat binds memory-consolidate::on-tick, and the tick runs a pass only when interval_hours (default 24) have elapsed since the last one — the last completed pass persists in the state worker. Catch-up-on-boot: a pass missed while this worker was down runs shortly after boot instead of waiting for the next heartbeat; a slim backstop loop also keeps the schedule alive on rigs with no cron trigger owner.
Quickstart
Plan without writing, then look at what a pass would do:
iii trigger memory-consolidate::run dry_run=true
iii trigger memory-consolidate::statusApply for one bank only:
iii trigger memory-consolidate::run bank=blogThe report names every group: the surviving memory, the retired duplicates, and anything skipped because it is pinned. The retired records stay on disk and remain queryable with include_superseded: true.
Configuration
All fields hot-reload through the configuration worker: enabled, interval_hours, dry_run (scheduled passes plan-only), banks (allowlist, empty = all), max_supersedes_per_run (safety cap; remainder waits for the next pass).
The LLM tier (optional)
llm_assist_enabled (off by default) adds one router::complete judge call per bank after the deterministic pass:
- Reorder groups — word-order matches the deterministic pass surfaces report-only ("The deploy runs on Fridays" vs "On Fridays the deploy runs") are merged only when the judge confirms the meaning is identical; role swaps ("Alice manages Bob") stay untouched.
- Rule promotion — memories re-observed
promote_corroboration_threshold+ times are offered as candidates; ones the judge classifies as standing instructions land as one-line entries in the bank's auto-managedlearnedrule, fingerprint-deduped, append-only. Hand-authored rules are never touched, and the judge can only act on candidates it was offered — it cannot invent targets.
Fail-soft: no router or a malformed reply just skips the tier (named in the report's errors); the deterministic pass has already completed.
What counts as a duplicate
v1 is deterministic and deliberately conservative: two live memories merge only when their normalized text matches (case, punctuation, and whitespace insensitive) or their token sets are equal (word-order shuffles). One differing word is NOT a duplicate — "always publishes" and "never publishes" must never merge. Survivor choice is stable: pinned first, then highest corroboration, then the oldest record (it carries the original provenance). Semantic near-duplicate merging is a later, LLM-assisted tier.
Boundaries
- Never touches memory's files; every mutation goes through
memory::supersedeandmemory::save— the same append-only, last-wins contract as everything else. - Pinned memories are never superseded, even inside a duplicate group.
- Not extraction (the memory worker captures), not recall, not rule management. One job: keep banks clean.
api reference (json)
{
"functions": [
{
"description": "Internal: hot-reload memory-consolidate from the authoritative configuration when it changes.",
"metadata": {
"internal": true,
"trace_hidden": true
},
"name": "memory-consolidate::on-config-change",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"id": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "OnConfigChangeEvent",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"ok": {
"type": "boolean"
}
},
"required": [
"ok"
],
"title": "OnConfigChangeResponse",
"type": "object"
}
},
{
"description": "Internal: schedule heartbeat (cron trigger + boot catch-up backstop). Runs a pass only when interval_hours have elapsed since the last one.",
"metadata": {
"internal": true,
"trace_hidden": true
},
"name": "memory-consolidate::on-tick",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "The cron worker's tick payload (all advisory; the handler re-derives due-ness from persisted state). Typed so the published interface carries a real schema.",
"properties": {
"actual_time": {
"default": null,
"type": [
"string",
"null"
]
},
"job_id": {
"default": null,
"type": [
"string",
"null"
]
},
"scheduled_time": {
"default": null,
"type": [
"string",
"null"
]
},
"trigger": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "TickInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"ran": {
"description": "A pass actually ran (false = not due yet or disabled).",
"type": "boolean"
}
},
"required": [
"ran"
],
"title": "TickResponse",
"type": "object"
}
},
{
"description": "Run one consolidation pass now (all configured banks, or one bank). Deterministic near-duplicate dedup applied strictly through memory::supersede + memory::save; pinned memories are untouchable. dry_run plans without writing.",
"metadata": {},
"name": "memory-consolidate::run",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"bank": {
"default": null,
"description": "Consolidate only this bank (must be in the configured allowlist, when one is set).",
"type": [
"string",
"null"
]
},
"dry_run": {
"default": null,
"description": "Override the configured dry_run for this call.",
"type": [
"boolean",
"null"
]
}
},
"title": "RunRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"BankReport": {
"properties": {
"bank": {
"type": "string"
},
"capped": {
"description": "Work left behind by the per-run cap; the next pass picks it up.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"errors": {
"items": {
"type": "string"
},
"type": "array"
},
"groups": {
"items": {
"$ref": "#/definitions/PlannedGroup"
},
"type": "array"
},
"llm_merged": {
"description": "Reorder groups merged after the LLM judge confirmed equivalence.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"promoted": {
"description": "Standing-instruction lines the judge promoted into the bank's `learned` rule.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"reinforced": {
"description": "Winner reinforcements applied (one per absorbed duplicate).",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"scanned": {
"description": "Live memories scanned.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"superseded": {
"description": "Supersedes actually applied (0 on dry runs).",
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"bank",
"groups",
"reinforced",
"scanned",
"superseded"
],
"type": "object"
},
"PlannedGroup": {
"description": "One planned merge: `losers` retire in favor of `winner`.",
"properties": {
"loser_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"loser_texts": {
"description": "Texts of the losers, aligned with `loser_ids` (the judge tier and human review need the words, not just ids).",
"items": {
"type": "string"
},
"type": "array"
},
"report_only": {
"description": "True = surfaced for review only (token-set match: same words, different order — potentially different meaning). Never written automatically.",
"type": "boolean"
},
"skipped_pinned": {
"description": "Losers left alone because they are pinned (pinned is untouchable).",
"items": {
"type": "string"
},
"type": "array"
},
"winner_id": {
"type": "string"
},
"winner_text": {
"type": "string"
}
},
"required": [
"loser_ids",
"loser_texts",
"winner_id",
"winner_text"
],
"type": "object"
}
},
"properties": {
"banks": {
"items": {
"$ref": "#/definitions/BankReport"
},
"type": "array"
},
"checkpointed": {
"description": "The pass completed cleanly AND its last-run checkpoint persisted. False = the schedule will retry the pass next check.",
"type": "boolean"
},
"dry_run": {
"type": "boolean"
},
"superseded": {
"description": "Total supersedes applied across banks this pass.",
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"banks",
"checkpointed",
"dry_run",
"superseded"
],
"title": "RunResponse",
"type": "object"
}
},
{
"description": "Schedule and last-pass report: enabled, interval, last run, whether a pass is due, and the most recent per-bank results.",
"metadata": {},
"name": "memory-consolidate::status",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "StatusRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"dry_run": {
"type": "boolean"
},
"due": {
"description": "A scheduled pass is overdue right now.",
"type": "boolean"
},
"enabled": {
"type": "boolean"
},
"interval_hours": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"last_report": true,
"last_run": {
"description": "Milliseconds since epoch of the last completed pass; 0 = never.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"dry_run",
"due",
"enabled",
"interval_hours",
"last_run"
],
"title": "StatusResponse",
"type": "object"
}
}
],
"triggers": []
}