What did Full Fact's study find?
Full Fact found that AI chatbots give outdated, miscalculated and premature answers in the same confident tone as correct ones, so a reader cannot tell them apart. The charity's first benchmark report, titled Wrong on Sunday, right on Monday, tested Gemini 3.1, ChatGPT, Grok, DeepSeek and Claude on questions of UK public interest.
Full Fact is careful to say that the models answered correctly much of the time. The problem it documents is not frequency but signal: nothing in an answer warns the reader when it is wrong. Its examples are specific.
- Superseded figures. Several chatbots quoted a weekly asylum support rate that had been replaced by a higher rate in June 2025.
- Proposals presented as fact. Some answers described a proposed policy as if it had already been adopted.
- Out-of-date office-holders. DeepSeek was still naming Keir Starmer as Prime Minister as late as September 2026.
- Different answers on different days. Asked for England's most recent major football trophy, some models gave the 2025 Women's Euro on some days and 2022 on others.
- Arithmetic across mismatched periods. Gemini 2.5 Flash produced a small-boat arrivals figure by combining totals that covered different dates.
- Weak citations. Some answers to UK voter ID questions cited American government websites, and one answer about a Northampton constituency cited a 2012 American local newspaper.
Full Fact's conclusion is that citations can give the look of sourcing without the substance, and that confidence is not evidence.
How was the study done?
Full Fact's fact-checkers wrote questions with reference answers, then put the same 22 questions to five chatbots once a day for ten weeks, with web search switched on. Each answer was broken into individual factual claims and scored against the reference answers.
| Measure | Figure |
|---|---|
| Questions analysed in full | 22 |
| Chatbots tested | 5 |
| Testing period | 15 July to 22 September 2026 (70 days) |
| Answers collected | 7,625 |
| Individual factual claims extracted | 74,538 |
| Scoring criteria | Factuality, transparency, timeliness, consistency, civic responsibility |
The limits are stated plainly. Twenty-two questions over ten weeks is a narrow window. Answers were collected through the companies' developer interfaces, which approximate but do not exactly match what a person sees in the app. Model versions changed during the study. And the report describes specific errors it recorded, not how often errors occur, so it is not a reliability league table.
What does it mean for B2B shortlists?
It means AI can describe your business to a buyer from out-of-date evidence, and the buyer has no way to tell. Full Fact tested public-interest questions, not questions about companies, but the failure modes it recorded map directly onto what buyers ask about suppliers: the current service range, who leads the business, where it operates, which accreditations it holds and whether an announced change has actually happened.
Buyers already notice the problem without being able to diagnose it. In G2's survey of 1,076 business software buyers, published in April 2026, 64% said they encounter inaccurate AI chatbot recommendations often or very often, and 69% said AI guidance had led them to choose a different vendor from the one they first planned. Inaccuracy and influence sit side by side.
Two of Full Fact's findings matter most commercially. First, a superseded fact can survive long after it changed: the old benefit rate was still being quoted more than a year later. A retired product, a former managing director or an old office can persist in the same way. Second, answers drift from day to day, so a single check of what AI says about you proves little. One buyer may hear the current story and another the old one in the same week.
What does it mean for companies in Ireland and the UK?
It means the local context AI attaches to your business needs to be stated clearly and confirmed by local sources. Full Fact's questions were framed for a UK reader, yet some answers leaned on American sources, and the report asks AI companies to reflect differences between the UK's nations.
For an Irish or UK supplier, the equivalent risk is AI answering a buyer's question with the wrong jurisdiction, regulator or market in mind, or falling back on larger international sources that describe a competitor instead. Companies selling across Ireland and the UK should expect AI to blur the two unless their own pages and the independent sources about them are explicit about where they operate and whom they serve.
What should leadership teams do this quarter?
Start by finding out what AI currently says about you, over several days rather than once. Four practical steps follow.
- Ask the factual questions a buyer would ask about you. Who leads the company, what it offers, where it operates, which standards it holds. Ask on more than one AI platform, in a fresh chat with no history, and repeat on different days to see whether the answers hold.
- Retire the old story at its source. List every fact that has changed in the last three years, then update or withdraw the pages, profiles and directory entries that still state the old version. AI cannot stop quoting what is still published.
- Separate what is current from what is planned. Announcements of future launches, expansions or appointments should be dated and worded so they cannot be read as already done.
- Check which sources AI uses to describe you. If the sources are old, foreign or about someone else, that tells you where the correction effort belongs.
A Gap Check report shows who AI recommends to your buyers instead of you and how far behind you are, at no charge, walked through with Neil.