This post starts a new series on the blog: Paper → Live demo → Honest post. The idea is simple: take a recent paper, build a demo anyone can try, measure it carefully, and report what came out, including what didn’t go the way the paper suggests. The first one is about memory for agents.
I’d been turning an idea over for a while: a conversational agent shouldn’t remember everything like meeting minutes, but do something closer to what we do when we sleep. During the day we pile up loose details; at night, the brain decides what stays, what merges with what we already knew, and what gets thrown away. An agent with short-, mid- and long-term memory, and an offline phase that reorganizes it. It was one of those ideas you jot down for “someday”.
So it was a joy to come across LightMem, presented in Lightweight LLM Agent Memory with Small Language Models by Jiaquan Zhang, Chaoning Zhang, Shuxu Chen, Zhenzhen Huang, Pengcheng Zheng, Zhicheng Wang, Ping Guo, Fan Mo, Sung-Ho Bae, Jie Zou, Jiwei Wei and Yang Yang, accepted at ACL 2026 (arXiv:2604.07798). It’s great to see an idea you had floating around turned into formal work, with a clear architecture, serious experiments and a name. All the credit for the design is theirs; mine is a demo inspired by their architecture and a measurement under far more modest conditions than theirs.
A clarification I make out of respect for the authors’ work: this is not a reproduction of the paper, nor does it evaluate it. I changed important pieces (I detail which ones and why below), so my numbers say something about my version with small local models, not about LightMem.
And the same as in previous posts: I built it together with a coding assistant, over long sessions of decisions and measurements, with a log that records every problem and how it was solved.
What the paper proposes
LightMem organizes an agent’s memory into three levels:
- STM (short-term memory): the last turns of the conversation, as they are.
- MTM (mid-term memory): semantic summaries of what was said, with their embedding, temporal information and access statistics.
- LTM (long-term memory): consolidated, de-identified knowledge, which in the paper is a graph of entities and relations.
The interesting part is who manages that memory. Instead of one large model doing everything, the paper uses specialized small language models (SLMs): a Controller that turns the question into queries and decides where to look, a Selector that filters the retrieved candidates, and a Writer that summarizes each turn into MTM entries. Every so many turns, an offline phase consolidates the MTM into the LTM: it merges, updates, reinforces, or lets weakly supported entries decay.
The paper’s name says it: lightweight memory. They report improvements over A-MEM on LoCoMo and retrieval with a p50 latency of 83 ms. Two details matter for what follows: the Selector is fine-tuned with LoRA, and the final answer is generated by GPT-4o-mini. The small models manage the memory; a large one answers.
The constraints shaped the design (again)
Like the previous demos, this one lives on the site’s ephemeral infrastructure, so it inherited its rules:
- Fully local. Ollama and open models. Zero external APIs, zero API keys.
- CPU only, on a small pod.
- Stateless. The memory lives in an ephemeral SQLite: when the demo is destroyed, everything is forgotten. Very fitting for an agent that dreams.
That forced me to depart from the paper in several places:
| Paper | My version | Why |
|---|---|---|
| 1–1.5B SLMs, LoRA-tuned Selector | Qwen 2.5 3B and 7B, no fine-tuning; the same model with different prompts | Fine-tuning is out of scope for a demo |
| Responder: GPT-4o-mini | Local responder, the same small model | Zero-API rule. It’s the biggest difference |
| LTM as a graph | LTM as a list of entries | Simplicity |
| Automatic consolidation every 10–15 turns | A 🌙 Dream button | So the consolidation can be seen |
| English embeddings (MiniLM) | bge-m3, multilingual | My conversations mix Spanish and English |
How the demo works
On each turn, the Controller decides whether the message carries new information and writes the search queries. The Retriever searches MTM and LTM by cosine similarity, the Selector keeps what’s relevant, and the Responder answers with that plus the last few turns. After answering, the Writer extracts atomic facts from the user’s message (“The user lives in Quito”) and stores them in MTM.
Every module’s output is JSON validated against a schema, with one retry and a deterministic fallback if the model fails. Each turn’s trace shows what every module decided and how long it took.
The dream is the part I enjoyed building the most. When you press Dream:
- Entries are grouped by similarity, topics or shared entities.
- The model proposes operations on each group:
MERGE,UPDATE,PROMOTE(from MTM to LTM) andFORGET. - A deterministic validator checks each proposal before applying it.
- What’s approved is applied in a single transaction, without truly deleting anything: what was replaced stays marked and points to what replaced it.
The UI shows the result as a diff: new entries in green, replaced or forgotten ones struck through in red, and separately the proposals the validator rejected, with the reason.
That validator became the heart of the project, for a reason that keeps coming up on this blog: the model proposes, the code decides. It rejects an UPDATE that introduces a name or a number that wasn’t in any of the original entries, one that loses the current value (“works as a UX designer” → “used to work as an architect”), a FORGET on long-term knowledge, or a PROMOTE without enough evidence.
Models copy the examples
The first lessons showed up before the benchmark, and almost all of them came from the same place: the examples in the prompts.
The Writer had an example with a dog named Toby who lives in Madrid. In the first test conversation, the memory ended up saying the user had a dog named Toby and lived in Madrid, even though the user had never said so. I swapped the examples for neutral entities and “Bilbao” showed up. I swapped them again and the 3B started recording that the user was a teacher, owned a bakery or had a cat, all taken from the examples.
The fix that worked best was grounding: every proper noun and every number in a fact has to appear in the user’s message (with tolerance for translations: “Lisbon” counts if the message says “Lisboa”). It catches made-up names and numbers. It doesn’t catch common nouns: “teacher” can’t be checked against a Spanish message without translating. That limit comes back later.
The benchmark
To measure it, I built a fictional user, Mateo, with 8 conversation sessions: 45 turns in Spanish and English, with paraphrases (his parrot Kiwi shows up in five sessions), changes (he moves from Valparaíso to Montevideo, leaves architecture for UX design, stops being vegetarian) and noise (the weather, traffic, what he had for breakfast). Then, 28 questions in new sessions: simple facts, updated facts, questions that span sessions, temporal questions, and questions about things Mateo never said, where the right answer is “I don’t know”.
Four conditions, same model and same data:
| Condition | Memory |
|---|---|
| A | No memory: only the last turns |
| B | Naive RAG: every user sentence stored verbatim; the 8 most similar go to the Responder |
| C | The full pipeline, no dream |
| D | The pipeline plus a Dream at the end of each session |
Condition B is deliberately dumb: it doesn’t extract, summarize or consolidate. It’s the control.
Three runs per condition, because even with temperature 0 and a fixed seed the results vary. (I learned that on the first test.)
v1: the control wins by 20 points
| Condition | Accuracy (7B) |
|---|---|
| A — no memory | 18 % |
| B — naive RAG | 89 % |
| C — pipeline | 69 % |
| D — pipeline + Dream | 66 % |
The naive RAG beat the pipeline by 20 points, and the dream took off 3 more. I rebuilt the memory from one run to understand why, and found a bug in every module:
- The Controller decided not to search. Asked “what language am I learning?”, it flagged that no memory was needed, and the Responder made up the answer.
- The Selector pruned too much. Temporal questions need two facts joined (“was an architect at Estudio Faro” and “took a UX course”); the Selector kept one and the answer was “I have no information”.
- The Dream made the Writer’s mistakes permanent. From “my parrot Kiwi learned to whistle”, the Writer had extracted “the user’s name is Kiwi”. The Dream read it as a name change and consolidated it. In 2 of 3 runs, the agent ended up telling Mateo “Your name is Kiwi”.
- The Dream forgot the peanut allergy. A fact said only once looks like noise, and
FORGETcan’t tell “had croissants for breakfast” from “is allergic to peanuts”.
The validator did its job: it blocked 18 updates that lost the current value. But the validator checks form, not truth. “Your name is Kiwi” doesn’t invent any name: Kiwi was in the message.
v2: fixing one piece exposes the next
With that diagnosis I made four changes, validated first with development conversations and not with the benchmark, so as not to tune the system to the questions:
- Always search, even if the Controller says it isn’t needed.
- Pass the Responder the original texts the Selector chose (at least 3, dated), instead of its summary.
- The Writer’s grounding.
- Only allow forgetting one-off events, never facts about health, work, family, pets or plans.
| Condition | v1 | v2 |
|---|---|---|
| B — naive RAG | 89 % | 88 % |
| C — pipeline | 69 % | 79 % |
| D — pipeline + Dream | 66 % | 79 % ± 9 (71 / 75 / 89) |
Ten points better, and the naive RAG is still on top, with half the latency (3.4 s vs 7.9 s per turn) and a third of the model calls.
The most interesting part of v2 was what the fix uncovered. Once it got the original texts instead of the summary, the Responder started seeing “his name is Mateo” and “his name is Kiwi”. Since the prompt tells it to trust the most recent, C answered “Your name is Kiwi” in all three runs. In v1 the Selector’s summary hid that Writer mistake, and it hid it by accident. The dream sometimes repaired it: it merged “has a parrot named Kiwi” with “is named Kiwi”. Sometimes it didn’t. Same configuration, from 71 to 89 % depending on the run.
And corruptions appeared that pass every rule: the Dream wrote that Mateo lives in Pagora (Pagora is the company he works for) and that he “lives in Valparaíso (used to be allergic to peanuts)”. All of it is grounded: every word comes from a real fact. The rules stop the model from inventing; they don’t stop it from mixing.
With 3B, the gap explodes
The paper uses 1–1.5B models, so I ran the same thing with Qwen 2.5 3B:
| Condition | 7B | 3B |
|---|---|---|
| B — naive RAG | 88 % | 93 % |
| C — pipeline | 79 % | 58 % |
| D — pipeline + Dream | 79 % | 49 % |
The naive RAG with 3B was the best configuration in the whole study, and the fastest: 1.4 s per turn. The pipeline with the same model lost 35 points.
That’s the line I take away from this project: the small model reads well and writes badly. As a reader of verbatim sentences, the 3B matches or beats the 7B. As a writer of memory, it falls apart:
- The Controller flagged 58 % of turns as “no new information”, among them Mateo’s name, his birthday and his parrot. Those facts never reached memory.
- The prompt examples leaked in as common nouns, exactly where grounding can’t reach: “teacher” 3 times, “bakery” 6, “cat” 9 (for a parrot).
- The dream consolidated the junk along with the real: “is allergic to peanuts and goes to the pool”, “took a UX course and learned to whistle a new song” (the parrot, again), “has friends named Kiwi and Ramón” (Ramón is the dog).
There’s an irony that made me laugh: my earlier post about a RAG chatbot was exactly condition B, and here it beat a far more elaborate architecture.
Why the naive RAG wins
The naive RAG stores what the user said, verbatim. There’s no step where a small model can rewrite, lose or invent a fact. The pipeline, instead, rewrites the memory twice: the Writer turns each message into facts and the Dream turns those facts into others. Every rewrite is a chance to get it wrong, and consolidation makes those mistakes permanent: the error stops being one odd sentence among many and becomes the memory.
This doesn’t contradict the paper; it bounds it. Their Selector is LoRA-tuned for that task, and the answer is written by GPT-4o-mini. My result says that without those big pieces, the architecture doesn’t hold up on its own with a small model. It’s a different question, and under my conditions the answer is clear.
Shipping it: the demo that took 90 seconds per call
It still had to go online, and on CPU the problem that taught me the most about the infrastructure side showed up.
The first measurement with the 3B, capped at 2 vCPU like the demo pod, was a disaster: the Controller took between 87 and 180 seconds, and the Writer ran out of time on 4 of 10 turns. Even the 1.5B took ~50 seconds per call. It didn’t match anything I’d seen before.
The cause: inside the container, Ollama sees the host’s 10 cores, not the 2 the limit gives it, and spawns one thread per core. The kernel’s CPU limit splits 2 cores’ worth of time across 10 threads that need to sync layer by layer, and they all end up waiting on each other. Pinning the threads to the real limit (num_thread=2) took the same call from 91 to 4 seconds.
With that fixed, what’s left is the physics of a model on CPU: it reads the prompt fast and generates a few tokens per second, so each module costs what it writes. The pipeline makes 2 or 3 calls before answering:
| qwen2.5:3b | Answer (p50) | Writer (in the background) |
|---|---|---|
| 2 vCPU | ~75 s | ~100 s |
| 4 vCPU | ~30 s | ~50 s |
I went with the 3B and 4 vCPU. To make the wait bearable, the Writer runs after answering: you see the answer first and, a few seconds later, the facts it learned show up in memory tagged “learning…”.
And since the main result is that the naive RAG wins, the demo doesn’t hide it. It has a selector with three modes: LightMem, RAG and Compare, which answers the same question both ways side by side, with their latencies and traces. It’s the most honest way I could think of to show the architecture: let you compare it yourself.
What I deliberately didn’t do
- No fine-tuning. A LoRA-tuned Selector, like the paper’s, is probably the piece that would change the numbers the most. It was out of scope.
- No large model to answer. That’s the zero-API rule, and it’s the biggest difference from the paper.
- No graph in the LTM. A flat list is enough to show consolidation.
- No multi-user. A single user, ephemeral memory.
And the honest limitation of the measurement: one fictional user and 28 questions, scored with keywords and reviewed by hand (I fixed two scoring errors in that review). It’s a case study, not a general benchmark.
What I’d do differently
If I started over, I’d separate storing from interpreting. I’d always store the user’s verbatim sentence, like the naive RAG, and let the Writer and the dream produce only indexes over those sentences: tags, links, a summary to search better. Never replacements. If the dream gets it wrong, you lose an index, not a memory. It’s the same lesson as the trip planner, in a different domain: take away from the model the decisions it can’t make well.
What I take away
Three ideas stay with me.
The first: with small local models, the best memory is the one that rewrites the least. Every stage that rephrases a fact is a chance to lose it, and consolidation makes those mistakes permanent.
The second: a validator checks form, not truth. My rules stop invented names and lost values; they don’t stop “Pagora” from going from company to city. That takes something that understands the sentence, and in this system that’s precisely the small model that got it wrong.
The third: fixing one piece can expose another’s mistake. The “Your name is Kiwi” was there from the first run; the Selector’s summary covered it up by accident. Without three runs per condition and without rebuilding the memory by hand, I’d have celebrated v2’s 10 points without seeing what they uncovered.
None of this takes anything away from LightMem. It leaves me wanting to try the piece I couldn’t: a fine-tuned Selector, which is exactly where the paper places its bet. Thanks to its authors for turning into serious work an idea many of us had been mulling over.
You can try the agent at alexisalulema.com/projects: it’s provisioned when you request it, runs for 40 minutes and destroys itself, memory included. Tell it something, ask it things, press 🌙 Dream and compare with the RAG. The code, with the full log of decisions, measurements and mistakes, is at github.com/alulema/dreaming-agent.
If it tells you your name is Kiwi, now you know why.