How much it can hold,
and how accurate it gets.

The ceiling nobody had written down

A memory system's job is to select relevant material and put it in front of the model, and the model can only read a fixed amount of text at once, which is its context window. Somewhere in the system there has to be a budget answering how much memory can be injected before it exceeds what the model can actually read.

I went looking for that budget, grepping the entire codebase for every term I could think of: n_ctx, context_window, ctx_size, num_ctx, context_length, max_context, and context_size.

Against a large cloud model that is comfortable. Against a local model with a 4,096-token window, which is the size most people can actually run, it is roughly 6 times over, and against an 8k window it is 3 times over. In addition, the configuration format had no field in which to declare the model's window even if the budgeting code had existed to use it.

The part that creates a memory problem

Exceeding a context window is ordinarily a loud failure. The server returns an error, you see it, and you send less, which is the good case and the one that happens with most hosted providers.

The common local inference server does something else by default. Rather than rejecting an over-length prompt it drops text from the front until the prompt fits, then answers normally and returns HTTP 200.

This is the same shape as the retrieval failure in question 2, arrived at through a completely different route, which is why I think the pattern rather than either individual bug is the real finding of this project. A memory system has an unusually large number of ways to fail that create confident output instead of errors, and confident output is not merely unhelpful; it is worse than an error, because it gets believed.

Two compounding factors

Token counting was optimistic in the wrong direction. Lacking a tokenizer for local models, the system estimated tokens as characters divided by 4, while the Llama, Mistral, and Qwen tokenizers actually run around 3.3 to 3.8 characters per token. The estimate undercounts by 10 to 20%, which means it reports that you are inside a budget at the moment you have already exceeded it. Local models also reached that estimator through accidental fallthrough in a conditional rather than an explicit branch, so reordering the condition would have changed the behavior silently.

Output reservation ate the remainder. Inference servers require that input tokens and reserved output tokens fit inside the window together. The system requested 2,000 output tokens for chat and up to 8,000 for synthesis as fixed values, never scaled down, and reserving 8,000 output tokens out of a 4,096-token window is not a tight fit; it is arithmetically impossible. Document synthesis could not succeed on a small local model at all, at any input size.

Where the ceiling actually binds

The useful reframing is that the limit on how much a system can usefully remember is neither storage nor retrieval quality, because both of those scale fine. The limit is the width of the channel between the memory and the model, and the model sets that width.

This means the engineering problem is not storing more; it is selecting better. Every improvement in retrieval precision converts directly into effective memory capacity, because the budget is fixed and spending it on an irrelevant record costs you a relevant one.

One place this was measurable: synthesizing a large uploaded document into memories was capped at roughly 100 memories per upload, because the synthesis step had to fit inside a single model call. Sharding that work across parallel calls and merging the results took a real upload from 140 memories to 337, using the same source material, the same model, and the same retrieval. The ceiling had been the shape of the request rather than anything about the data.

How accurate does it get?

Honestly, I can report reliability precisely and accuracy only partially, and I would rather be clear about which is which than blur them together.

Reliability is measurable and good. The system can run 2 completely different embedding backends, one on PyTorch and one on ONNX Runtime, against the same model weights, and their outputs agree at cosine = 1.000000 on every text tested including the prefixed query form. A query embedded by one backend correctly ranks documents embedded by the other, scoring 0.603 for the target against 0.398 or lower for the distractors. Retrieval quality therefore does not quietly depend on which machine built the index, which is a real correctness property and one I could verify cheaply.

Accuracy is harder and my evidence is weaker. Retrieval accuracy means whether the right memory surfaced for a given question, and the ground truth there is a judgement about relevance. I have 25 baseline queries with known-correct answers along with the paraphrase tests on the previous page, and while both show the mechanism working, neither is large enough to publish a number against. I am not going to invent one.

The negative result is what I can say with confidence, and it is the one worth carrying: a retrieval system's accuracy cannot be assessed from its output. For 4 months this one created genuinely good answers and scored well against its own baseline while the retrieval mechanism I believed was running was not running at all. Output quality told me nothing, because a fluent model holding a handful of keyword matches already clears the very low bar that an empty context window sets. Any accuracy claim not grounded in instrumentation of the retrieval step itself is measuring the model's fluency rather than the memory's quality.