Building the honesty check: twelve practice runs

The promise is that Eunosa checks what it says about itself against its own records before you see it. Whether that works will be measured by the honesty test, with a plan registered first and conversations nobody building the check has seen. That test hasn’t happened yet.

This page is about building the check. We tried it on practice conversations to find its mistakes and fix them, and we’re publishing what we found, the flattering and the unflattering. None of it shows the promise holds.

Before each reply, Eunosa labels everything it hands the voice: the memories that reply needs, your recent messages, and a few plain facts about itself, like whether remembering is on. You never see the labels.

The voice doesn’t word facts about you or about Eunosa’s memory. Where one belongs, it leaves a marker, and the engine puts in the fact’s own words: your words quoted exactly (“In Memory I have: …”), or a plain line about how memory works. For “I don’t have a record of …”, the voice names what’s missing and the engine writes the sentence. The voice’s own words are for help only.

Then, before the reply is shown or saved, Eunosa reads each sentence about itself or you against what it points to. Names, numbers and dates have to match. “Not” can’t flip the meaning. Most of the words have to match. “I’m sure” isn’t allowed yet. A borderline sentence then goes to a meaning check, which reads it beside the record and says whether it means the same, adds something, or contradicts it. A sentence that fails is left out, and the voice isn’t asked to try again.

Each round’s changes were fixed and written down before it ran. No rule was changed because of a round’s result.

Eunosa’s memory is designed to stay on your device. For each reply, the cloud voice gets a short brief with the few memories that reply needs. The meaning check uses the same cloud model (Claude Haiku 5.5). It is sent the sentences it reads and the memories they point to, which the voice already had. To check “I don’t have a record of that”, it is sent everything stored, because there’s no smaller honest way to check it. That never happens while remembering is paused. Eunosa keeps no copy of what’s sent.

In round 4 we tried a checker that runs on the laptop instead. It was about as accurate (58 of 71 answers right, against 59) but took about 27 seconds a check, against a limit of 3 seconds set beforehand. So the check stays in the cloud for now.

Fifteen short made-up conversations, written while building, ran through the app on one computer with the cloud voice Eunosa uses (Claude Haiku 5.5). Each one started with an empty memory and only the facts that conversation needed. We ran all fifteen five times on 2 November 2026, fixing what we found between runs, and read every sentence each time.

Then, on 3 and 4 November, seven more rounds tried the meaning check and the changes that followed it. Each used 20 to 22 new conversations written before it ran, never reused, with one reply each and no reruns. A separate AI reviewer, given the replies and the records but none of the check’s verdicts, read every sentence as true or false.

Few false sentences got through. Too many true ones were held back.

Each run had 17 replies. Five false sentences got through across all five runs. Four were the same mistake, a blanket “I don’t have a record of anything about you” when facts were kept, and each led to a fix. By the last run, about 1 in 6 true sentences was still being held back, mostly because the voice worded them differently from the record. The honesty test allows 1 in 20 at most, so this is what the next seven rounds worked on.

The first two columns are counts. The last two are the builder’s own reading of every sentence, done after the runs and not blind. No reply was ever left empty.

No false sentence got through in the last round. True ones held back were one sentence over the line.

In round 7, 3 of 52 true sentences (5.8%) were held back, against the 5% the honesty test allows. All three were the voice writing “I don’t have a record” in its own words instead of leaving it to the engine. None of the engine’s own true sentences was held. Rounds go up and down partly with which conversations were written, and each round is small, so one round says little on its own.

With the meaning check on. True and false are a separate AI reviewer’s reading, done with none of the check’s verdicts. Each round used new conversations.

In the first five runs, every mistake came from a practice run or from one of three reviews: an independent review of the code, a second review of the fixed code, and an automatic review. Each one is now kept as an automatic check that runs every time the code changes.

• Labels go at the end of a sentence, like a footnote. In the first run the voice repeated each claim inside its label, so the reply showed text twice.
• More sentences that sound like claims are caught even with no label: “I haven’t kept”, “as you’ve told me”, “we talked about”, “I have no record”.
• “I don’t have a record of anything about you” is held back whenever something is kept. This was the false sentence that kept getting through.
• “I don’t have a record of X” has to be true across all of memory, and says nothing more.
• Only the right kind of source counts. A question you asked can’t back “you told me”. Eunosa’s own earlier guess can’t either. “I’ve kept that” needs the engine’s own record of keeping it.
• A denial has to deny the same thing as the record: “tea, not coffee” isn’t “coffee, not tea”.
• A sentence can’t bring in a person or place the engine never gave, like “since your sister Maria lives in Paris”.
• New facts the voice can point to: what was kept earlier in this chat, and your latest message.
• Fewer true sentences held back for small things: “Yes,” at the start, “Tuesday” versus “Tuesdays”, everyday words like “What” or “Memory” read as names, and ordinary help that mentions Greek yogurt or a Settings app.
• When a sentence is held back and nothing is left, Eunosa’s own line is one that is always true: “I can’t back that up from what I have, so I’ve left it out.”

Then, round by round:

• Round 1 (3 Nov): a meaning check after the word rules. It reads a borderline sentence beside the record it points to and says whether it means the same, adds something, or contradicts it. Only “the same” lets a loosely worded sentence through, and a sentence that passed on words but adds a detail is held back.
• Round 2 (3 Nov): a “nothing on record” sentence is also read against everything stored. That can hold it back but never let it through.
• Round 3 (3 Nov): only the facts a sentence points to go to the meaning check.
• Round 4 (4 Nov): Eunosa’s own facts about its memory can be said by the engine, word for word, instead of by the voice. A checker that runs on the laptop was tried and was too slow. In practice the voice never used the new option (0 of 36 sentences).
• Round 5 (4 Nov): the voice may not write a sentence about Eunosa’s memory at all; the engine says it. The engine also writes “I don’t have a record of …” from the few words the voice gives it. One rule loosened after a test on every saved sentence: such a sentence held only because a memory shares a word with it can be let through when the meaning check reads the memory as a different detail.
• Round 6 (4 Nov): when the voice ends its own sentence about memory with the engine’s, the engine’s sentence replaces it.
• Round 7 (4 Nov): the engine also says everything about what you told it, quoting your words exactly (“In Memory I have: …”). The voice words no fact about you or about memory, only help. Checking “nothing on record” against the chat as well was tried and dropped: it held back more true sentences than it caught false ones.

• It isn’t the honesty test. The conversations were written by us and used to tune the check, so it fits them better than it would fit new ones.
• In the first five runs, the readings of which sentences were true or false were done by the builder, not blind. In the seven rounds they were done by a separate AI reviewer: a model, not a person.
• The meaning check is a model from the same family as the voice, so it isn’t an independent judge. Its mistakes are part of what the honesty test will measure.
• “I don’t have a record of that” is checked against what’s stored, not against what you said earlier in the same chat.
• It checks that a sentence matches Eunosa’s records, not that the records are right.
• How sure Eunosa is can’t be checked yet, so any “I’m sure” is simply left out.

These conversations are now public, so the honesty test won’t reuse them or anything close to them. Every reply from the twelve runs is kept word for word, with every fix and review, and will be published with the draft paper.

1. 3 Nov 2026Page published with how the check was built, the practice results, every fix and the limits.
2. 4 Nov 2026Updated with seven rounds of the meaning check (3 and 4 November): what each round changed, its results, what leaves your device, and the limits now. Correction: this page said the check matches words, not meaning. Since 3 November a meaning check follows the word rules.

All research