Checking Eunosa’s memory against a published test
We ran Eunosa’s memory through HaluMem, a published test of how AI memory goes wrong, with our pass lines set first. What Eunosa suggests remembering was almost always true. It still gave a wrong answer about the person one time in seven, mostly by quoting a fact that had since changed.
- Plan and pass lines fixed (done)
- Conversations replayed and scored (done)
- Results published (done)
- A fix for out-of-date facts, tested (current step)Next, with new lines set first
Why we asked
Eunosa’s memory works differently from most assistants’. Once a week it suggests facts worth keeping, in its own words, and nothing is kept without your yes. To answer you, it brings up the few kept facts that share words with your message, and the engine quotes them exactly; the voice only helps.
HaluMem1 tests memory systems step by step: what they pick up from a conversation, whether they notice when something changes, and whether their answers about you are right. Its authors scored six published systems, so it shows where Eunosa sits beside them. This isn’t the honesty test, which comes next and uses its own conversations.
How we tested it
HaluMem comes with months of made-up conversations between people and an assistant. We drew three of its twenty people with a fixed random seed and replayed all 4,407 of their messages through Eunosa, in order, with the real voice (Claude Haiku 5.5) and the honesty check on. At the end of each conversation, every fact Eunosa suggested was kept as worded, as if the person said yes to everything. That is harsher than real use, and it’s how the published systems were tested, with no person in the loop.
Then we asked 501 questions about those people, each as the person asking about themselves. A stronger model (Claude Sonnet 5.5) scored everything with HaluMem’s own scoring instructions. An independent reviewer scored 60 items without seeing those scores and agreed on 55. We also checked, on all twenty people, how often Eunosa’s way of bringing up facts finds the one a question needs.
Pass lines, set before testing
- Wrong answers about the person: 10% or fewer.
- Things only the assistant said, never confirmed by the person, left out of suggestions: 90% or more.
- Suggestions backed by what the person said: 80% or more.
- Right answers: 54% or more, the middle of the published systems.
- Facts that mattered, picked up: 43% or more, also the middle.
Results
Two of five lines passed.
| Measure | Eunosa | Published systems | Our line |
|---|---|---|---|
| Wrong answers about the person | 14.4% (72 of 501) · fail | 15.2 to 30.0% | 10% or fewer |
| Suggestions backed by what the person said | 99.2% (1,487 suggestions) · pass | 32.3 to 61.9% | 80% or more |
| Facts that mattered, picked up | 45.4% (603 of 1,329) · pass | 14.6 to 74.1% | 43% or more |
| Right answers about the person | 49.3% (247 of 501) · fail | 35.3 to 67.2% | 54% or more |
| Things only the assistant said, left out | 53.7% (213 of 397) · fail | 44.9 to 80.8% | 90% or more |
What Eunosa suggests remembering was backed by what the person said 99% of the time, well above every published system. It picked up 45% of the facts that mattered, in the middle of them.
It gave a wrong answer about the person 14.4% of the time: fewer than any of the six published systems, but above our line, so it fails. Most of its misses were “I don’t have a record of that” rather than wrong answers.
Answers, by kind of question
| Kind of question | Right | Wrong | Missing |
|---|---|---|---|
| The answer isn’t known (131) | 97.7% | 0.8% | 1.5% |
| A plain fact (106) | 30.2% | 16.0% | 53.8% |
| A question with a false premise (115) | 27.8% | 13.9% | 58.3% |
| Applying what’s known to something new (108) | 47.2% | 28.7% | 24.1% |
| Several facts combined (26) | 11.5% | 15.4% | 73.1% |
| Something that changed over time (15) | 6.7% | 20.0% | 73.3% |
When the answer isn’t known, Eunosa almost always said so. Questions about something that changed over time, or that need several facts at once, went worst.
Why answers went wrong
Mostly, old facts. Eunosa never changes a memory by itself, so when someone’s situation changed, the old and new versions both stayed, and an answer sometimes quoted the old one: a health status, an income. The quote was always the person’s stored words; what was wrong was which fact was current. In real use, the weekly “Still true?” question is meant to catch this. In the test, nobody was there to answer it.
The rest were general advice that didn’t use a preference the person had mentioned, which HaluMem counts as wrong. Keeping only the suggestions that were fully backed barely changed the answers (15.0% wrong), so the problem is which fact is brought up and quoted, not what was kept.
The line on things only the assistant said failed less badly than it looks. Only 6 of 397 were fully taken up as suggestions. About half partly overlapped with something true the person did say, and HaluMem counts any overlap as a failure.
What we’re doing about it
A fix is being built and tested now: when two kept facts disagree, bring up the newest first, and include the “Still true?” question in the test. It will be measured on three people not used here, with its lines set before it runs, and published here whatever it shows.
What this doesn’t show
- Three people, made up and written by a machine. Real conversations may differ.
- Keeping every suggestion is harsher than real use, where you say yes or no to each one.
- The scoring was done by a model, checked against an independent reviewer on 60 items.
- HaluMem’s authors scored with a different model (GPT-4o), so the comparison with published systems is close, not exact.
- HaluMem scores general advice against one reference answer, which is strict for help that isn’t about the person.
- One question came back blank when it was put in the first person, so it was used as written. The first attempt at asking the questions stopped partway for two people because of it; those replies were never read, and the questions were asked again.
Cost: US$34.09 in all, under a US$60 limit set before the run.
Sources
- Chen, Niu, Li, Liu, Zheng, Tang, Li, Xiong and Li, 2025. HaluMem: evaluating hallucinations in memory systems of agents. doi.org/
10.48550/ arXiv.2511.03506 Back to the text
Change log
- Page published with the plan, the pass lines, the results, why answers went wrong, and the limits.