How We Find The Bugs Before You Do
Before a release goes near your checkout, we point a fleet of AI reviewers at the change from a dozen different angles, then make a second fleet try to prove each finding wrong. The last pass confirmed 55 real problems. The most valuable thing it found was not a bug in the product — it was four of our own tests passing for the wrong reasons.
Every platform says it tests before it ships. So does this one: a few thousand automated checks run before anything reaches you. The uncomfortable question is the one underneath that, and it is the reason this post exists.
How do you know the tests are telling the truth?
A test that passes for the wrong reason is worse than no test at all, because it produces confidence you have not earned. You stop looking. And the whole point of a checkout is that the thing you stopped looking at is somebody's money.
What actually runs
Before a release, the change gets read by a fleet of independent AI reviewers. Not one pass with one prompt — a dozen different lenses, each looking for a different kind of failure. One is only thinking about money: can a number be charged that disagrees with a number displayed. One is only thinking about what a page tells a buyer. One is only thinking about what happens when a request arrives twice, or arrives late, or never arrives.
Then the important half. Every finding is handed to reviewers whose job is to kill it — to go back to the code and prove the problem is imaginary. A finding that survives being attacked from several directions is worth fixing. One that does not, is noise, and noise is what makes people ignore reports.
The last full pass produced 55 confirmed findings, one the reviewers could not agree on, and none that survived scrutiny only to turn out imaginary. Twenty-nine further suggestions were judged too minor to act on now, and written down rather than done.
The findings that matter are the quiet ones
Nothing found in that pass was on fire. Nothing was down. That is the point — an outage announces itself, and the failures that actually cost merchants money are the ones that look fine.
Three from the last audit, because specifics are more useful than reassurance.
A declined buyer was told the sale worked
The thank-you page had branches for a payment still finalising, a payment stalled, and a trial — and then, underneath, an unconditional success message for everything else. A card that finalised and then failed fell through to that fallback. Worse, the finalising branch refreshes itself every three seconds, so it actively delivered people into the wrong answer: land while pending, the payment fails, the page reloads, the pending check is now false, and the buyer reads You're in — order confirmed.
Nobody was charged. That is precisely why it could have run for months: no support ticket, no refund request, just a buyer who thinks they own something they do not. It now has honest failed and refunded states, and a way to try again.
Failed checkouts counted as revenue in an export
An order row is created when someone starts a checkout, and it carries the full total from the moment it exists. A new net-revenue column read that total without checking whether the payment ever completed — so an export handed to an accountant included money that never arrived.
A page that got slower the better you did
A delivery check ignored data the page had already loaded and re-ran itself three times per live offer. At fifty offers that was 390 of the page's 477 database queries. Measured after the fix: 40. This is the kind of thing that never appears as a bug report — the merchant with five offers never notices, and the merchant with fifty assumes the dashboard is just slow.
The part I did not expect
The most valuable thing the audit found was not in the product. It was in the tests.
Four of our own automated guards were passing for reasons that had nothing to do with what they claimed to check:
- A test that counted how many tools our API exposes was being satisfied by a number that appeared in the page's stylesheet — calc(100vh - 48px) — rather than by the documentation it was supposed to be checking.
- That same guard, in one environment, quietly marked itself skipped instead of failing. A skipped test is green at a glance.
- A test for a bug in a reopening dialog asserted only the behaviour the original bug already had, so it would have passed against the broken code.
- Several browser-level checks existed and ran on nobody's machine automatically.
All four are fixed. The tool-count guard now fails rather than skipping, and the browser checks run automatically. But the lesson generalises past those four: a safety net you have never seen catch anything is not evidence of safety. So new tests now get run against the broken code first. If a test does not fail when the bug is present, it is not a test — it is decoration.
What this does and does not buy you
Honest limits, because a post like this can easily read as a promise it cannot keep.
It does not mean your checkout will never break. It means the specific class of problem that hides — money that displays one number and charges another, a page that reassures a buyer it should be warning, a report that counts money that never arrived — gets hunted deliberately, by reviewers who have no ego invested in the code being right, before it reaches you.
It also does not replace the boring things. Payments still settle on your own Stripe account. Prices are still computed on the server rather than in a browser anyone can edit. The audit is a layer on top of those, not a substitute for them.
And the audits keep finding things, which is the correct outcome. The day one comes back clean is the day I will assume the reviewers are broken rather than the code is perfect — that has happened once already, when three reviewers hit a limit and returned nothing, and an empty result looked exactly like a pass.
If you want the detail on any of this, or you hit something the audits missed, reply to any email from me. Those reports are worth more than another lens.