The Oracle Problem Moved Into Your Pension: What a 10,000-Answer Test Actually Proves

A test issued 121 financial questions to a fleet of large language models and asked each one five times. Ten thousand answers. And buried beneath the headline — a 57% failure rate — sits a number the researchers generated but never published: the rate at which the same model, asked the same question five times, gave five different answers. I have spent twenty-three years watching systems that people trust with money fail in the gap between what they claim and what they do. In 2018 I held a Parity Wallet release for two weeks because a single mis-sequenced state transition could have drained user funds inside a nested call. Management wanted the deadline. I wanted the proof. The proof won. That instinct is the only useful lens for this story. What a UK fintech has now demonstrated is not that chatbots are unreliable. Everyone suspected that. What it has demonstrated is that we have rebuilt, in natural language, the exact oracle failure we spent a decade engineering out of blockchain systems — and this time there is no hash to verify and no revert to catch the lie. The test itself is straightforward, and its simplicity is the point. A fintech called Saturn — I will return to that name, because it matters — assembled 121 prompts covering the ordinary surface of personal finance: tax obligations, pension contributions, savings allocation, HMRC thresholds. The questions were designed to have verifiable correct answers. Each was submitted five times to a range of models, both free tiers and paid tiers. The result set exceeded ten thousand responses, which is enough volume to make the sample statistically meaningful in a way that most “I asked a chatbot about my taxes” anecdotes are not. The numbers that emerged: a 57% overall failure rate. On the hardest subset, 88%. Free models failed at 63%; paid models failed at 49%. The strongest model available — named in the reporting as “Claude Opus 5,” a designation I will flag as suspect — still failed 39% of the time. On the hardest questions, the free models failed 93%. And at least one wrong pension answer, if acted upon, maps to a penalty of roughly £17,500 from HMRC. Now place those numbers against the adoption curve. An EY survey found 49% of users already lean on AI for savings and investment decisions. A PensionBee study found that nearly six in ten users would act on AI advice without independently verifying it. And an FCA study found 56% of respondents trust AI more than they trust television and radio — while 44% of the same population believe AI financial information is regulated. It is not. There is no compensation scheme, no ombudsman route, no institutional counterparty standing behind the answer. That is the whole story in one sentence: adoption has reached roughly fifty percent while reliability is still failing at roughly fifty percent, and the users cannot tell the difference, because they believe a regulator is watching. No regulator is watching. Let me be precise about the failure modes, because “the AI is wrong” is not an audit finding. It is a headline. An audit enumerates specific, reproducible defects and assigns each one a severity. Saturn’s test surfaces three distinct classes, and they are not equally dangerous. First, arithmetic failure. The models miscalculate. This is the least surprising and, ironically, the most fixable class. A language model is a probability engine pretending to be a calculator. It has no reliable symbolic arithmetic core. Any block of code that performs money-critical math without a dedicated computation module is defective by design, and you do not fix that by adding parameters. You fix it by routing the calculation through deterministic tooling. In blockchain terms: you do not ask the validator to guess the balance. You call balanceOf(). The lesson was learned in Solidity years ago — never compute state transitions in your head; compute them on-chain where the result is checkable. Second, knowledge staleness. The models missed tax rule changes. This is a training-cutoff problem, and it is structural, not incidental. A model frozen at training time cannot know that a threshold moved last April. In our world, this is the stale oracle. A price feed that reports yesterday’s price with confidence is worse than no feed at all, because downstream contracts will liquidate against a fiction. The DeFi industry answers this with fresh feeds, heartbeat intervals, and deviation thresholds. Consumer AI financial advice answers it with nothing. Third, and most dangerous, the rule hallucination. The models invented rules that do not exist. They generated plausible-sounding regulatory clauses, fabricated pension mechanics, and asserted tax treatments with total composure. This is a “reasonable but false” output, and it is the one that should terrify anyone who has ever read a smart contract exploit postmortem. The hallucination is not an error the model recognizes as an error. It is a confident, grammatically perfect, contextually appropriate assertion. It reads exactly like a correct answer. In a £17,500 HMRC penalty scenario, the user has no signal to distinguish the fabricated rule from the real one. Now the fourth category, which the researchers folded into their failure count and which deserves separate treatment: missing risk warnings. The models failed to append adequate disclaimers. I find this the most methodologically interesting inclusion, because it inflates the failure rate in a way that is defensible and simultaneously masks the deeper problem. On one hand, a financial recommendation without a disclaimer is genuinely defective — it is the equivalent of a function that mutates state without a require() guard. On the other hand, the conversational formatting of free-tier products routinely strips boilerplate, which means part of that 57% is measuring presentation, not reasoning. The absence of a severity model is another audit gap. Saturn reports a flat 57% failure rate, which treats a wrong answer about a savings account and a wrong answer about a pension tax deadline as equivalent. They are not. In my own audits I score defects on two axes: probability of occurrence and magnitude of loss. A hallucinated pension rule that triggers an HMRC penalty sits at the intersection of moderate probability and severe impact — the two axes that demand a blocking fix, not a documented caveat. A miscalculated savings projection sits at high probability and low impact — an annoyance, not a catastrophe. Collapsing both into a single percentage destroys the information a decision-maker actually needs. The headline tells you the system is unreliable. It does not tell you where the live grenades are, and those are different questions. Here is where I part company with the framing. The published number — 57% — is neither the most useful nor the most honest metric. The aggregation is undisclosed. Is 57% the simple average of per-model failure rates, or the weighted failure rate across all 10,566 answers? Those are different numbers, and they support different conclusions. A simple average lets one catastrophic model drag the figure up. A weighted rate tells you what an individual user actually experienced. Saturn does not say which it used. An audit report that does not disclose its aggregation method would not survive my review, and it should not survive yours. Then there is the geographic skew. If a large share of the 121 questions concern UK tax and pension law, the models are being graded on jurisdiction-specific knowledge that is thin in most training corpora. This does not excuse the failure — a financial tool deployed in the UK must know UK rules — but it reframes the finding. The models are not universally incompetent at finance. They are incompetent at the specific, local, frequently-updated corner of finance where the stakes are highest and the data is scarcest. That is a more precise and more useful statement than “57% fail.” What the study never tested is whether the failures are mitigable. This is not a rhetorical complaint; it is the single most important omission for anyone who intends to build. Every one of the three failure classes has a known engineering remedy. Arithmetic failure is solved by tool invocation — force the model to call a deterministic calculator or a computational API rather than doing the math in its own head. Knowledge staleness is solved by retrieval augmentation — ground every answer in a live, authoritative, versioned tax and pension database instead of frozen weights. Rule hallucination is partially contained by constrained decoding and citation requirements, though never fully. A study that reports a 57% failure rate for a bare model, and neglects to report the failure rate for the same model wrapped in tool-calling and retrieval layers, has measured the wrong configuration. It has tested the model as a curiosity and drawn a conclusion about the model as a product. The comparison to conventional benchmarks weakens further when you ask what the ceiling even is. The best model failed 39% of the time in what the reporting describes as its reasoning mode. Thirty-nine percent failure is not a tuning gap. It is a paradigm boundary. You cannot tune your way across it, the same way you cannot optimize your way out of a non-deterministic consensus algorithm. When the strongest available system, given unlimited money and compute, still produces two wrong answers for every three right ones on high-stakes financial questions, the correct engineering conclusion is that the architecture is unsuited to unsupervised deployment — not that it needs another iteration. I watched this exact reasoning fail to land in the ZK-rollup boom of 2022. I spent four months benchmarking proof generation times against gas costs and concluded that the compression algorithms of the era were not viable for high-frequency trading. The narrative said otherwise. The math did not care. A major investment was delayed, the project later missed its mainnet target, and nobody wrote a thank-you note. The point is not that I was right. The point is that feasibility math beats narrative every time, and financial AI is now exactly where those rollups were — a compelling pitch running ahead of a capability that does not yet exist. There is a second parallel worth drawing, and it is closer to home. Chainlink solved the oracle decentralization problem by introducing a set of nodes that are, in practice, operated by a small and identifiable set of parties. The marketing called it decentralized. The topology said otherwise. What we are now being sold in financial AI is the same sleight of hand in reverse: a single, opaque, centralized model, marketed as a neutral assistant, feeding answers into decisions that carry the same weight as an oracle call. The decentralization was cosmetic in one case; the neutrality is cosmetic in the other. In both cases, the user is told to trust an intermediary whose failure modes are hidden behind a confident interface. That brings me to the beneficiary of all this scrutiny. Saturn ran the test. Saturn’s CEO is quoted calling on the FCA to act. Saturn did not disclose the failure rate of its own product. You do not need a forensic accounting degree to see the structure: a licensed fintech publishes a study showing that unlicensed, unregulated, general-purpose AI is dangerously unreliable at financial advice, and simultaneously calls for the regulator to draw a line between that category — and Saturn. I am not accusing Saturn of fraud. I am reading the incentive table. This is the same pattern the blockchain industry has lived with since KYC became a marketing feature. Compliance is advertised as consumer protection and functions as a moat. When you raise the cost of entry, you do not eliminate the bad actors — they move offshore or buy a wallet’s worth of holdings to pass the check. You eliminate the honest small operator who cannot afford the compliance overhead, and you hand the market to whoever already built the paperwork. Saturn calling for FCA action is not altruism. It is a bid to make its existing compliance capability the barrier the next entrant has to clear. And this is where the reporting’s own blind spot becomes visible. The article frames the problem as “general AI is unsafe.” The more honest frame is “there is no accountable party, licensed or not.” A licensed fintech selling AI financial advice faces the same arithmetic-only-capability problem as a general chatbot. Licensing does not add a calculator to the model. It adds a legal entity to sue. The distinction matters precisely because the article — and Saturn — would rather you believe licensing solves the capability problem, when all it solves is the liability problem. Those are not the same fix, and conflating them is how an interested party turns a safety concern into a sales argument. Let me also address the model naming. “Claude Opus 5” does not correspond to a publicly documented release in my knowledge set, which runs to early 2025. Either the reporting is working from a newer snapshot than my information covers, or the name is imprecise. I raise this not to score a point but to calibrate confidence. The single most quoted number in the entire study — “even the best model fails 39%” — is attached to a model designation I cannot independently verify. If the name is wrong, the rigor of the surrounding technical detail is in question. If the name is right, my own reference frame is stale. Either way, no one should treat that 39% as a settled figure without the primary methodology. Which is the recurring absence across this entire study: the primary methodology. Saturn is both the tester and a stakeholder, and the study has not been published as a peer-reviewed or third-party-audited artifact in a form I can inspect. In my field, we do not accept an audit from the audited party at face value. We demand the diff, the test suite, and an independent reproduction. Here we have a press release and a headline number. That is not nothing — the underlying concern is real and the adoption data corroborates it — but it is not yet proof. Now the part the study generated and then ignored — the number that would actually change my behavior as a user. Every question was asked five times. That repetition was almost certainly a design choice to detect output variance. It is the kind of decision a competent evaluator makes. And then the study reports only aggregate failure. It never reports the rate at which the same model, asked the same question five times, produced five different answers. That inconsistency rate is more dangerous than the failure rate, and its absence is the most revealing omission in the study. Consider the user experience. A 57% failure rate means that, roughly, one answer in two is wrong. But if the model is consistent, the user at least faces a stable fiction — they will get the same wrong answer every time, form a habit, and the error is at least predictable. If the model is inconsistent, the user gets a different answer each time they ask, has no basis to detect the error, and will simply select the answer they prefer. A financial decision engine that tells you five different things about your pension and lets you keep the one you like is not a tool. It is a slot machine with a tax problem. This is the deepest structural break between what blockchain systems do and what language models do. A smart contract is deterministic. Given the same state and the same input, it returns the same output, for everyone, forever. That determinism is not a luxury. It is the property that makes verification possible — the property that lets me write a proof, check it against the chain, and be certain. Language models have no such property. They are non-deterministic by construction. The same prompt, the same context, the same everything, and the output drifts. And financial advice is a domain where determinism is not nice-to-have. It is the entire basis of trust. I designed a proof-of-personhood protocol in 2025 for exactly this reason. Autonomous agents had to prove their origin and intent without revealing proprietary logic, using a commitment scheme that made each claim verifiable and each agent accountable. The architecture was not optional decoration. The whole point was that a system making consequential decisions must leave a checkable trail. A language model giving financial advice leaves no such trail. There is no commitment, no proof, no anchor. There is only an output that evaporates the moment the session closes. The art is the hash; the value is the proof. A financial recommendation that cannot be reproduced cannot be audited, and a recommendation that cannot be audited should never be the sole basis for a decision about someone’s retirement. The study measured the wrong axis. It counted how often the models were wrong. It should have measured how often they disagreed with themselves. So where does this leave the developer, the builder, anyone shipping a financial product on top of a model that failed 39% of the time at its best? It leaves you exactly where the audit culture of this industry should have trained you to stand: at the edge of the system, asking what guarantees come back. A model is an oracle. It sits outside your contract, feeds you a value, and your logic trusts it. We spent ten years learning that you never trust an oracle unconditionally. You verify it, you bound it, you add a heartbeat so you know when it has gone stale, and you route money-critical math through something deterministic that does not have opinions. None of that discipline exists in consumer AI financial advice. Instead we have an oracle that answers fluently, updates unpredictably, invents its inputs, and is trusted by half the population because they assume someone is watching. We do not build for today. We build for the failure that has not happened yet. The failure that has not happened yet is a generation of users acting on confident, fabricated, non-reproducible financial advice with no counterparty to hold accountable when it goes wrong — and a regulatory intervention that will arrive, as it always does, after the losses are already on the books, and that will protect the incumbents who wrote the warnings rather than the users who believed them. Watch the FCA. Watch whether Saturn publishes its full methodology. Watch whether any of these models ever ships a financial mode with a deterministic calculation core. The verdict on whether you should trust a chatbot with your money is not the study’s to give. It is the market’s — and the market has already answered, at 49% adoption, faster than the technology can justify. Reentrancy does not announce itself. Neither does a fabricated tax rule. Both drain the account the same way.