How the study
was actually run.

Build first, measure after

The project ran roughly in this order: build the apparatus through the first half of 2026, run a structured audit against it in June, discover the retrieval failure, repair it, and then conduct the local-model and scale work using the instrumentation that the repair had forced me to build.

While that ordering was not a plan, I do not think the questions are answerable any other way. All 3 of them ask about emergent behavior, which means what a memory system does at thousands of records, across months, against a real model, and getting a system into that state requires building one first.

The audit procedure

The June audit was a 12-phase sweep across the codebase that produced a written report with a claim list attached. The important design decision was what happened to that list next, because an audit producing a list of worries is not evidence of anything.

1. Every claim had to be verified or falsified against the source

   Not "this looks risky," but a specific file and line that either demonstrates the problem or shows it is not there. Anything that could not be grounded that way got dropped rather than carried forward as a soft concern.

2. Adversarial passes against each confirmed finding

   Confirmed findings went to a separate reviewing pass whose only job was to refute them, going back to the code and arguing the finding was wrong. 17 verification passes ran in total and no skeptic overturned a confirmed finding, which is the result that let me treat the list as reliable rather than as a hunch with footnotes attached.

3. Falsification actually happened, which is the point

   One headline claim was refuted. The audit asserted that conversation summaries were never embedded, when in fact the writer stamps a vector onto every summarized conversation at a specific line I could point at. The real bug sat one level up, in a loader reading from a sidecar index that deliberately excluded vectors.

   The conclusion survived and the stated mechanism did not. A procedure that never overturns its own inputs is not verifying anything, so this was the most reassuring result of the whole audit.

How repairs were constrained

There were 2 rules, both chosen because the original failure had been invisible and I did not trust my ability to notice the next one.

Heal existing data, and never require a migration. The fix for the schema mismatch could have been to rename the field in the writers and reindex everybody. Instead the loaders were taught to accept both field names, so every index already sitting on disk started retrieving correctly the moment the code updated, with no rebuild, no migration step, and no window during which somebody loses their memory because a data migration went wrong. The writers were then extended to emit the richer schema for new records, and nothing was renamed.

A wrong vector is worse than no vector. Where an embedding could not be regenerated correctly, which happens after a model change, the stored vector gets nulled rather than kept. A stale vector creates confidently wrong rankings while a missing one creates an honest keyword fallback, and given that the entire failure under investigation was silent wrongness, that trade was not close.

Making the tests falsifiable

The pre-existing recall baseline had failed as a safeguard by recording what the system did rather than what it should do, so the new guards had to meet a specific standard: a test for this class of bug must fail on the broken code.

16 contract tests were written against the repaired system and all 16 pass. The tree was then stashed back to the pre-fix state and the suite re-run, where 7 of the 16 fail. Those 7 are the ones that would actually have caught the original bug, while the other 9 guard properties that were never broken, which is worth knowing as well.

The old recall baseline was then re-captured against the repaired pipeline. The expected scores moved substantially, which under normal circumstances is the signature of a regression; here it was the signature of a regression ending. Documenting why a golden file moved is the only thing that separates those 2 cases.

What this method does not cover

A methods section listing only strengths is marketing. The real limits:

• One system, one author. There is no control implementation, no second architecture to compare against, and no independent replication. Every finding here is what happened in this system, and the generalizations are mine rather than demonstrated.
• Code audit is not runtime measurement. Large parts of this evidence come from reading the source carefully rather than observing production behavior, which is strong for "this code path cannot execute" and weak for "this is how often users hit it." I have no usage data at all, because the system collects none, deliberately, and that is a real methodological cost of question 1's answer.
• Accuracy is under-measured. As question 3 says, I can report retrieval reliability precisely and retrieval accuracy only anecdotally. A proper relevance-judged evaluation set is the obvious missing piece.
• Known issues were left unfixed. Several lower-severity findings are documented and deliberately not repaired, and they are listed in the repository rather than quietly dropped, because an audit whose leftovers vanish is not an audit.

If you want to check any of this

The primary documents sit in the repository, written during the work rather than written for this site:

They are working documents and they read like it. They also contain the blunt self-assessments, which I have left in.