memory-consolidate
v0.1.15Scheduled hygiene for the memory worker — deterministic dedup of near-duplicate memories, applied supersede-only through the public memory functions, pinned untouchable, with catch-up-on-boot scheduling. Install, stop, or remove it without touching stored memory.
iiiverified- macOS: arm64 · x64
- Linux: arm64 · armv7 · x64
- Windows: arm64 · x64
exact versions are immutable; binary and bundle artifacts are digest-pinned.
full markdown
/workers/memory-consolidate.md. paste it into an llm prompt or pipe it through curl from a worker.install
dependencies
readme
memory-consolidate
Scheduled hygiene sibling of the memory worker. Finds near-duplicate memories in every bank and merges them: the duplicate retires with a superseded_by pointer (never a delete), the survivor absorbs the observation as corroboration. Strictly supersede-only through memory's public functions, pinned memories untouchable, every change visible as a memory::item-changed event. Install, stop, or remove it without touching stored memory.
Install
iii trigger compose::add worker=memory-consolidateRequires the memory worker. Scheduling reuses the engine's cron trigger infrastructure: an hourly heartbeat binds memory-consolidate::on-tick, and the tick runs a pass only when interval_hours (default 24) have elapsed since the last one — the last completed pass persists in the state worker. Catch-up-on-boot: a pass missed while this worker was down runs shortly after boot instead of waiting for the next heartbeat; a slim backstop loop also keeps the schedule alive on rigs with no cron trigger owner.
Quickstart
Plan without writing, then look at what a pass would do:
iii trigger memory-consolidate::run dry_run=true
iii trigger memory-consolidate::statusApply for one bank only:
iii trigger memory-consolidate::run bank=blogThe report names every group: the surviving memory, the retired duplicates, and anything skipped because it is pinned. The retired records stay on disk and remain queryable with include_superseded: true.
Configuration
All fields hot-reload through the configuration worker: enabled, interval_hours, dry_run (scheduled passes plan-only), banks (allowlist, empty = all), max_supersedes_per_run (safety cap; remainder waits for the next pass).
The LLM tier (optional)
llm_assist_enabled (off by default) adds one router::complete judge call per bank after the deterministic pass:
- Reorder groups — word-order matches the deterministic pass surfaces report-only ("The deploy runs on Fridays" vs "On Fridays the deploy runs") are merged only when the judge confirms the meaning is identical; role swaps ("Alice manages Bob") stay untouched.
- Rule promotion — memories re-observed
promote_corroboration_threshold+ times are offered as candidates; ones the judge classifies as standing instructions land as one-line entries in the bank's auto-managedlearnedrule, fingerprint-deduped, append-only. Hand-authored rules are never touched, and the judge can only act on candidates it was offered — it cannot invent targets.
Fail-soft: an unavailable router just skips the tier (named in the report's notes, so it never blocks the checkpoint), and a malformed reply is treated as "no merges, no promotions". Either way, the deterministic pass has already completed.
The semantic tier (on by default)
semantic_enabled (on by default; set it to false to run without model calls) runs one more tier, last, on a fresh fetch. It costs one router::complete call per 8 new memories, capped at semantic_max_per_run per bank per pass. Without a router or an embed provider, it stops with a note and never blocks the pass. A router::complete judge looks at each memory written since the last pass, next to its nearest neighbours, and decides:
- merge: both state the same fact (paraphrase, another language). The duplicate retires and the survivor absorbs its observations. The survivor must keep every detail; when both are equally complete, the older one survives.
- supersede: a state change ("moved to", "switched to", a rescheduled date). The outdated record retires in favour of the current one, which is the one stated last.
- nothing: distinct facts on one topic (two allergies, two projects). This is the normal answer.
Every change goes through memory::supersede and carries the judge's one-sentence reason on memory's item-changed events. The design follows Hindsight's consolidator (hindsight_api/engine/consolidation/):
- Candidates. Neighbours come from the raw lexical and raw semantic
memory::recallarms, interleaved, because a fused rank buries the near-duplicate. - Targets. The judge may only pair a memory with one of its own candidates, through short per-batch aliases.
- Validation. Before anything is written, every action is checked: pinned records are never retired; direction follows
last stated; a merge may not drop a number the survivor lacks; the reason is required; the per-run cap applies. - Freshness. Each record is re-read right before the write, so a record edited since the judge read it drops its action.
- Failure handling.
- Transport, auth or quota errors stop the tier for that bank with a note, and the cursor stays put.
- An unparseable or truncated reply bisects the batch down to one memory. A memory that still fails is listed in
semantic_failedand skipped, so it cannot wedge the tier.
- Cursor. Progress is a per-bank cursor in the state worker, keyed by
(updated_at, id), so edited or resurrected records are judged again. It advances only over batches that were fully judged and fully applied. - Dry runs judge and report the plan without writing or moving the cursor.
Config: semantic_batch (8 memories per LLM call), semantic_candidates (4 per arm), semantic_max_per_run (48 per bank per pass). The model comes from llm_model. router::complete has no temperature control, so the provider default applies.
Judge eval. tests/judge_golden.rs runs the real prompt and validator over 18 cases from memory's recall golden set: 8 supersedes, 4 merges and 6 keeps. It reports accuracy and harmful actions, which must be 0. With claude-sonnet-5-5 the result was 18/18 with 0 harmful. UPDATE_CONSOLIDATION=1 records the decisions into memory's tests/recall_quality/consolidation.json, which memory's recall eval replays in its hybrid-consolidated mode: traps go from 2 to 0 and duplicate pollution from 23 to 0.
III_NAMESPACE=<ns> cargo test --test judge_golden -- --ignored --nocaptureWhat counts as a duplicate
v1 is deterministic and deliberately conservative: two live memories merge only when their normalized text matches (case, punctuation, and whitespace insensitive) or their token sets are equal (word-order shuffles). One differing word is NOT a duplicate — "always publishes" and "never publishes" must never merge. Survivor choice is stable: pinned first, then highest corroboration, then the oldest record (it carries the original provenance). Semantic near-duplicate merging is a later, LLM-assisted tier.
Boundaries
- Never touches memory's files; every mutation goes through
memory::supersede(withreinforceon merges, so the survivor absorbs the duplicate's observations in the same call) — the same append-only, last-wins contract as everything else. Every change carries areasonon itsmemory::item-changedevents. - Pinned memories are never superseded, even inside a duplicate group.
- Not extraction (the memory worker captures), not recall, not rule management. One job: keep banks clean.
api reference (json)
{
"functions": [
{
"description": "Internal: hot-reload memory-consolidate from the authoritative configuration when it changes.",
"metadata": {
"internal": true,
"trace_hidden": true
},
"name": "memory-consolidate::on-config-change",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"id": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "OnConfigChangeEvent",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"ok": {
"type": "boolean"
}
},
"required": [
"ok"
],
"title": "OnConfigChangeResponse",
"type": "object"
}
},
{
"description": "Internal: schedule heartbeat (cron trigger + boot catch-up backstop). Runs a pass only when interval_hours have elapsed since the last one.",
"metadata": {
"internal": true,
"trace_hidden": true
},
"name": "memory-consolidate::on-tick",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "The cron worker's tick payload (all advisory; the handler re-derives due-ness from persisted state). Typed so the published interface carries a real schema.",
"properties": {
"actual_time": {
"default": null,
"type": [
"string",
"null"
]
},
"job_id": {
"default": null,
"type": [
"string",
"null"
]
},
"scheduled_time": {
"default": null,
"type": [
"string",
"null"
]
},
"trigger": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "TickInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"ran": {
"description": "A pass actually ran (false = not due yet or disabled).",
"type": "boolean"
}
},
"required": [
"ran"
],
"title": "TickResponse",
"type": "object"
}
},
{
"description": "Run one consolidation pass now (all configured banks, or one bank). Deterministic near-duplicate dedup, plus the optional LLM tiers when enabled (reorder judge; semantic merge/supersede of new memories, each with a reason), applied strictly through memory::supersede; pinned memories are untouchable. dry_run plans without writing.",
"metadata": {},
"name": "memory-consolidate::run",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"bank": {
"default": null,
"description": "Consolidate only this bank (must be in the configured allowlist, when one is set).",
"type": [
"string",
"null"
]
},
"dry_run": {
"default": null,
"description": "Override the configured dry_run for this call.",
"type": [
"boolean",
"null"
]
}
},
"title": "RunRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ActionKind": {
"enum": [
"merge",
"supersede"
],
"type": "string"
},
"BankReport": {
"properties": {
"bank": {
"type": "string"
},
"capped": {
"description": "Work left behind by the per-run cap; the next pass picks it up.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"errors": {
"description": "Real failures (reads, writes). Any error keeps the pass due.",
"items": {
"type": "string"
},
"type": "array"
},
"groups": {
"items": {
"$ref": "#/definitions/PlannedGroup"
},
"type": "array"
},
"llm_merged": {
"description": "Reorder groups merged after the LLM judge confirmed equivalence.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"notes": {
"description": "Soft skips (an optional tier unavailable, a sibling too old) — telemetry only: they never block the checkpoint, so an absent router cannot re-run the whole pass on every heartbeat.",
"items": {
"type": "string"
},
"type": "array"
},
"promoted": {
"description": "Standing-instruction lines the judge promoted into the bank's `learned` rule.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"reinforced": {
"description": "Winner reinforcements applied (one per absorbed duplicate).",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"scanned": {
"description": "Live memories scanned.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"semantic": {
"description": "Semantic-tier decisions (applied, or planned on dry runs), each with the judge's reason.",
"items": {
"$ref": "#/definitions/SemanticAction"
},
"type": "array"
},
"semantic_failed": {
"description": "New memories the judge could not decide even alone (a poison input); skipped past the cursor so they cannot wedge the tier.",
"items": {
"type": "string"
},
"type": "array"
},
"semantic_rejected": {
"description": "Judge actions dropped by validation (unknown alias, pinned target, wrong direction, a lost number, ...).",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"superseded": {
"description": "Supersedes actually applied (0 on dry runs).",
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"bank",
"groups",
"reinforced",
"scanned",
"superseded"
],
"type": "object"
},
"PlannedGroup": {
"description": "One planned merge: `losers` retire in favor of `winner`.",
"properties": {
"loser_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"loser_texts": {
"description": "Texts of the losers, aligned with `loser_ids` (the judge tier and human review need the words, not just ids).",
"items": {
"type": "string"
},
"type": "array"
},
"report_only": {
"description": "True = surfaced for review only (token-set match: same words, different order — potentially different meaning). Never written automatically.",
"type": "boolean"
},
"skipped_pinned": {
"description": "Losers left alone because they are pinned (pinned is untouchable).",
"items": {
"type": "string"
},
"type": "array"
},
"winner_id": {
"type": "string"
},
"winner_text": {
"type": "string"
}
},
"required": [
"loser_ids",
"loser_texts",
"winner_id",
"winner_text"
],
"type": "object"
},
"SemanticAction": {
"description": "One semantic-tier decision as reported (and kept in `last_report`).",
"properties": {
"applied": {
"description": "False on dry runs. Actions dropped by the pre-write freshness check are not listed here; they count in `semantic_rejected` with a note.",
"type": "boolean"
},
"keep_id": {
"description": "Merge survivor / supersede replacement.",
"type": "string"
},
"keep_text": {
"type": "string"
},
"kind": {
"$ref": "#/definitions/ActionKind"
},
"reason": {
"type": "string"
},
"retire_id": {
"description": "Merge duplicate / supersede outdated record.",
"type": "string"
},
"retire_text": {
"type": "string"
}
},
"required": [
"applied",
"keep_id",
"keep_text",
"kind",
"reason",
"retire_id",
"retire_text"
],
"type": "object"
}
},
"properties": {
"banks": {
"items": {
"$ref": "#/definitions/BankReport"
},
"type": "array"
},
"checkpointed": {
"description": "The pass completed cleanly AND its last-run checkpoint persisted. False = the schedule will retry the pass next check.",
"type": "boolean"
},
"dry_run": {
"type": "boolean"
},
"superseded": {
"description": "Total supersedes applied across banks this pass.",
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"banks",
"checkpointed",
"dry_run",
"superseded"
],
"title": "RunResponse",
"type": "object"
}
},
{
"description": "Check the consolidation schedule and last pass: enabled, interval, last run, due, and the latest per-bank results.",
"metadata": {},
"name": "memory-consolidate::status",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "StatusRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"dry_run": {
"type": "boolean"
},
"due": {
"description": "A scheduled pass is overdue right now.",
"type": "boolean"
},
"enabled": {
"type": "boolean"
},
"interval_hours": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"last_report": true,
"last_run": {
"description": "Milliseconds since epoch of the last completed pass; 0 = never.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"dry_run",
"due",
"enabled",
"interval_hours",
"last_run"
],
"title": "StatusResponse",
"type": "object"
}
}
],
"triggers": []
}