Ask a leading AI model a hard factual question with its search tools switched off and roughly one in three answers will be wrong — or an admission that the model simply doesn’t know. That is the central finding of AA-Omniscience, the benchmark maintained by Artificial Analysis. It also explains why every serious assistant reaches for the open web the moment precision matters, and why the pages it finds there still carry commercial weight.
The argument was sharpened in an analysis published on 5 October 2026 by SEO consultant Natzir Turrado: embedding knowledge inside a model is expensive, while retrieving it from outside is cheaper and fresher. Wherever a retrieval engine exists, someone will optimise to appear inside it.
- 66% at best: Claude Opus 5.5, the index leader, answers roughly two-thirds of factual questions correctly without web access.
- Abstention pays: The scoring penalises wrong answers but not silence, so cautious models climb the table.
- Retrieval stays essential: No model knows everything, which keeps search visibility a live concern for publishers.
What AA-Omniscience actually measures
The benchmark poses 6,000 questions across 42 topics in six domains: business, the humanities, finance, law, medicine and science. Scoring is deliberately blunt. The index runs from -100 to 100, adding a point for each correct answer, subtracting one for every error and leaving abstentions unpenalised, so a zero means as many hits as misses.
Two methodological details matter. The test runs without tools — no browsing, no document retrieval, no code execution — so what is being measured is what a model has internalised. And because abstention is free, the ranking rewards the model that stays quiet when out of its depth while punishing the one that bluffs.
A leaderboard where caution beats knowledge
The top of the table is tight. Claude Opus 5.5 leads with an index score of 46 and accuracy of 66%, followed by GPT-6 Astra in its high configuration on 44 and Claude Fable 5.1 on 43 — the latter posting the group’s highest raw accuracy at 67%.
The rest can be inferred from those two figures, and this is our own arithmetic rather than published data. If the index subtracts errors from correct answers, Opus 5.5 is wrong on roughly 20% of questions and abstains on the other 14%. Fable 5.1 errs on about 24% and abstains on 9%, which is precisely why it trails despite knowing more.
Turrado cites different numbers — a 23% error rate for Opus 5.5, 29% for Sonnet 5.5 and 23% to 43% across the twelve strongest models. Those could not be reproduced from the public table, which omits several configurations, so we attribute them to his reading. Either way, the shortfall is real: a mistake every four or five difficult questions.
Artificial Analysis captures the tension in a third metric, the hallucination rate — the share of incorrect answers among everything that was not correct. Applied to our arithmetic, Opus 5.5 would invent an answer in roughly 59% of the cases where it does not know, and Fable 5.1 in about 73%. At the far end sits MiniCPM5-1B, a tiny model with a hallucination rate of 1% — not because it knows more, but because it almost always says it doesn’t. A cautious, ignorant model can look more dependable than a brilliant, reckless one.
Why retrieval keeps search optimisation alive
The practical fix for imperfect internal knowledge is retrieval, and retrieval means reading web pages. A model that cannot know everything — and should not pretend to — needs fresh outside information, and somebody has to have published it in a form that can be found.
It is neither cheap nor easy. On 22 April 2025, during the antitrust trial against Google, ChatGPT product lead Nick Turley testified that OpenAI was years away from answering 80% of queries with its own search technology, and said Google had refused to grant access to its own. Google has since pushed its flagship product in the same direction: by July 2026, AI-powered search had become the default for 43% of searches, with fewer clicks reaching publishers. Models evidently need the web; nothing guarantees they will send traffic back to it.
What this means for publishers
TechNews DevTop has been publishing since 2005, and the job has shifted from writing for a search engine that ranked links to writing for systems that read, summarise and cite. Three lessons follow. Models search hardest for what changes and what is specific, so dates, figures and proper nouns matter more than ever. An answer that stands on its own, without the surrounding page, is easier to quote. And original data — the figure only you hold — is the one thing a model cannot obtain elsewhere.
The index also quantifies a legitimate worry about delegation: even the best available system, unassisted, misses one in five difficult questions and does not always flag its uncertainty. The sensible response is not to distrust AI outright but to ask whether it searched — an answer with linked sources and one without deserve different levels of confidence, even when both arrive in the same assured tone. Turrado’s core argument holds, with one condition attached: that work only makes economic sense if assistants cite and link in good faith. If they silently absorb content, the incentive to publish quality material erodes, and with it the raw material their own answers depend on.