Behind the scenes

Teaching An Assistant To Admit It Forgot

Sam Bakker · Founder, GoCushy · August 11, 2026

TL;DR

We are building an assistant into GoCushy. It is not switched on — no merchant account can reach it yet, and this is a look at the work rather than something to go and use. Two fixes I think are worth writing down: it now cites the exact section of a doc and the server throws away any citation pointing at a section that does not exist, and it carries far more of a conversation and says it cannot see that far back instead of denying something it did.

An assistant that is confidently wrong is worse than one that says it does not know.

That looks obvious written down. It is still very easy to build the other thing by accident, because an assistant that always has an answer demos beautifully and only fails later, in front of somebody who had started to trust it.

So this is not a launch post. The assistant is off. The only account it runs on is mine, which is deliberate — I would rather find these failures on my own offers than on yours. What follows is two failures I found, and what we changed.

What the thing is

It answers questions about GoCushy, and it can propose a small number of changes to an account.

The answers come from the same documentation the public docs are built from. Not an export, not a summary written specially for it — the same source. Edit a docs page and its answers change with the page. That matters more than it sounds. The ordinary way to build one of these is to copy your documentation into a second store, and from that day the copy starts drifting from the original, quietly, in the direction of being wrong. Ours cannot drift, because there is only ever one of it.

The changes are where I have been careful. It can propose exactly four things: change a price, pause an offer, set an offer live, and create a coupon. That is the entire list. Each one arrives as a confirmation card showing the real before and the real after — computed on the server, not typed into a sentence by a model — and nothing happens to the account until a person presses the button. Each card works once.

A proposed change waits for a human press before anything is written You ask for a change. The assistant returns a confirmation card showing a price going from forty-nine dollars to thirty-nine dollars, computed on the server, usable once. A dashed line runs across the figure: everything above it has touched nothing. Pressing the button crosses the line and the price becomes thirty-nine dollars. Not pressing it stops at the line and nothing is written. You ask Change a price Confirmation card $49.00 → $39.00 Computed on the server Works once You press the button You don't press Or close the chat Nothing above this line has touched your account The price is $39.00 Nothing is written
All four proposals — change a price, pause an offer, set an offer live, create a coupon — take this shape. The before and after are worked out on the server, the card is good for one press, and the account is untouched until a person presses it. The prices here are an example.

That gate is the shape everything else has to fit into. If a change cannot be shown as a before and an after that a person can check in two seconds, it does not get to be one of the four.

The first lie: read more

Early on, it liked to finish an answer with a link to the documentation. Which is fine, except the link was to the documentation — the whole of it. Sending somebody to the top of a very long page when you know exactly which paragraph answers their question is a way of looking helpful without being helpful.

Then I asked it to be specific, and it was specific. Sometimes specifically wrong.

It would cite a section that sounded exactly right and did not exist. Section anchors are easy to invent because they read like ordinary text: a heading called "Refunds and disputes" is a completely plausible thing for a payments doc to contain. The link renders. It is underlined. You click it, and you land at the top of a page with no explanation of why. That is the worst kind of broken link, because it never announces that it is broken — it just quietly makes you feel stupid for not finding the thing.

So the server checks. After an answer is written and before it reaches a person, every citation is tested against the actual documentation: does this section genuinely exist? If it does, the link points straight at it. If it does not, the link is downgraded to the page it claimed to be part of. The invented anchor never reaches a human at all.

Citations are verified on the server before the answer is shown An answer that cites a document goes to a server check asking whether that section exists. If it exists, the link goes to that section. If it does not, the link is downgraded to the page and the invented anchor is dropped. An answer That cites a doc Does that section exist? Checked on the server Before you see it It exists It doesn't The link goes to that section As the answer said The link goes to the page The invented anchor is dropped
The check happens on the server, after the answer is composed and before anyone reads it. A real section is cited exactly. A section that was never there is quietly demoted to the page it claimed to live in, so a citation never lands you nowhere.

I want to be precise about what that does and does not fix. It does not stop the assistant being wrong about content. It stops it handing you a link that goes nowhere while sounding certain. Those are two different problems and only one of them can be solved by checking.

The second lie: no, I didn't

This is the one that bothered me.

Conversation history was cut at ten turns. Not summarised, not flagged — cut, with no signal that anything had been removed. Which means that deep into a chat, the assistant was reading a conversation that appeared, from where it stood, to have begun in the middle.

Here is what that looks like from the other side of the screen. Early in a chat you pause an offer. It proposes the change, you press the button, the offer pauses. Some time later, in the same chat, you ask whether that got done. And it says no. Flatly, with no hedge, while the paused offer sits right there in the dashboard.

It is not lying in the sense of knowing better. That part of the conversation had simply stopped existing for it. But nobody experiences it that way. What you experience is a tool contradicting something you watched it do, and after that you do not trust it about anything else either, and you are right not to.

Two changes. It carries far more of the conversation now, so an ordinary working chat does not go near the edge. And when a chat genuinely does outrun what it can see, it says so:

I can't see that far back in this conversation.

Which is a worse demo and a better tool. On top of that, changes that were actually applied are carried by name, so "did you pause the Summer offer" has an answer that does not depend on how much chat has piled up between the asking and the doing.

The old ten-turn cut produced a false denial; the new window admits the gap Before: a conversation bar where everything older than the last ten turns was dropped silently, so a question about something done at turn three got the answer no, I haven't, which was false. Now: far more of the bar is carried, and anything genuinely beyond the window produces the answer I can't see that far back. Before Turn 3 — you pause an offer Dropped Last 10 turns "No, I haven't." Turn 14 · false Nothing told you the thread was cut Now Turn 3 — same moment Older Still carried "I can't see that far back." Applied changes keep their names
The old failure was not that the window was small. It was that nothing told the assistant the window had a far edge, so it read a truncated conversation as the whole conversation and answered accordingly.

Why I think this is the whole job

Anything with an assistant attached to money has the same two failure modes. It does something you did not ask for, and it says something false with confidence.

The first gets all the attention, and it is the easier of the two. You solve it by making the list of possible actions short enough to read, showing the exact effect before it happens, and putting a human finger on the button. That is the four changes and the confirmation card, and once it is built it stays built.

The second one never finishes. Every answer is a fresh opportunity to sound sure about something untrue, and you cannot fix that with a permission model. All you can do is keep finding the specific places where certainty is unearned — an anchor nobody verified, a memory nobody bounded — and close them one at a time.

Where this is the wrong tool

I would rather set this out now than have anybody discover it later.

It answers from documentation. If the documentation does not cover something, the honest answer is that it does not, and that is what I want it to say rather than reasoning its way to a plausible-sounding guess. It will be a bad choice for anybody hoping to interrogate it about undocumented behaviour.

It proposes four changes, and four is not a soft limit that grows if you ask nicely. Anything outside that list, it cannot do — which includes most of what running a business on GoCushy actually involves. Building offers, adding bumps, wiring a post-purchase sequence: that is work for your own AI through the connection you already point at your account, not for this.

And it is not a substitute for looking. A confirmation card tells you what is about to change. It does not tell you whether changing it is a good idea. That judgement stays with you, which is the correct place for it to stay.

When

I am not going to give a date, because the honest answer is that it goes on when a week of using it produces nothing I would be embarrassed to have shown you. Right now it is switched off everywhere and runs on exactly one account, which is mine.

The reason this post exists is that the failures that week keeps producing are not the ones I expected. Almost none of them are "it got a number wrong". Nearly all of them are "it sounded sure". A tool that says I don't know costs you thirty seconds. A tool that invents a section heading, or denies work it did in front of you, costs you the ability to believe the next thing it says — and there is no version of that trade I want to make on a product that touches your prices.

In the meantime, the parts of GoCushy that are live are worth your time: cart offers went out last week, and the documentation is the same documentation the assistant will be reading.