Ask a general-purpose chatbot about an internal policy and it will often answer from memory — fluently, confidently and occasionally wrong. Retrieval-augmented generation, better known as RAG, is the fix the industry settled on: before composing a sentence, the system searches a document collection you supplied, pulls the passages that look relevant and drafts its reply from those. The technique powers Gemini Notebook, the product Google renamed from NotebookLM on 16 July 2026, and the expanded knowledge mode in Claude Projects. It narrows the error margin appreciably. It does not close it.
- Grounded answers: instead of leaning on what the model memorised during training, RAG feeds it the exact fragments it needs and can point back to the file they came from.
- Already mainstream: Gemini Notebook and Claude Projects both switch retrieval on automatically once a knowledge base grows large enough.
- Real money: 2026 pricing guides put a production RAG chatbot at $30,000 to $80,000, while the monthly model bill stays surprisingly small.
From a 2020 NeurIPS paper to the chatbot in your browser
The concept originates in Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, a paper by Patrick Lewis and eleven co-authors accepted at the NeurIPS 2020 conference. The problem it attacked is the one users still complain about: language models store knowledge in their parameters but struggle to access it precisely, to justify where an answer came from and to stay current.
The authors paired two memories — a parametric one, the model itself, and a non-parametric one, a vector index of Wikipedia queried by a neural retriever. Together they delivered the best results of the time on three open-domain question-answering tasks, and the output was judged more specific, more diverse and more factual than an equivalent model without retrieval. In plain terms: rather than asking the AI to remember, you hand it the library and ask it to look things up.
Inside the pipeline: chunk, index, retrieve, cite
OpenAI’s file-search documentation breaks the process down cleanly, and it generalises well beyond one vendor.
- Chunking. Each file is split into fragments — 800 tokens with a 400-token overlap by default — so a single idea is not severed in half.
- Indexing. Every fragment becomes an embedding, a numeric representation of meaning, stored in a vector database.
- Retrieval. A query triggers a hybrid of semantic and keyword search: semantic matching finds relevant passages even when they share no words with the question. Ten results come back by default, up to fifty on request.
- Answering with citations. The model receives those passages alongside the question and writes a response annotated with pointers to the source files.
Step three is where systems live or die. If retrieval surfaces the wrong passage, the model will write with great confidence on a flawed foundation. Surrounding choices matter too: a 2026 study evaluating 450 configurations across eight benchmarks and corpora of 5,000 to 200,000 documents found that three retrieval techniques predicted to deliver a combined 47% gain produced only 17% when stacked together.
Big context windows are not a substitute
The Lost in the Middle study, published in TACL in 2023, showed accuracy degrades markedly when the relevant detail sits in the middle of a long context; information at the start or end fares better, even in models built for long inputs. RAG dodges part of that by handing over a few fragments rather than a hundred pages. Anthropic notes that its Projects still prefer in-context processing when feasible and only fall back to retrieval near the limit, where capacity expands “up to 10 times”. For a short document, pasting it whole remains the sensible choice.
Hallucinations shrink — they do not vanish
The preregistered study Hallucination-Free?, in its May 2024 version, evaluated legal research tools built on RAG by LexisNexis and Thomson Reuters. Some vendors advertised them as capable of eliminating hallucinations; the researchers measured fabrication rates between 17% and 33% — better than a general assistant such as GPT-4, but far from zero. Citations are no safe conduct either: a link shows where the model started, not that it summarised faithfully. If a figure matters, open the citation and read the original paragraph.
The recurring failure modes are mundane: outdated documents that retrieval keeps surfacing, chunks sliced at awkward boundaries and ambiguous questions that drag in the wrong fragment. Clear file names, which Anthropic explicitly recommends, clean versioning and questions that name the specific document all help.
Budget tells a similar story. Published 2026 cost guides put a proof of concept at $10,000 to $25,000, a multi-source production build at $30,000 to $80,000 and an enterprise deployment with role-based access and audit logs at $80,000 to $200,000 or more. Infrastructure is the cheap part: at 10,000 questions a month, model usage runs roughly $5 to $100, and managed vector databases start at $20 to $50. Data preparation and evaluation consume the real budget — which is why two proposals that both say “RAG on GPT” can differ threefold in price and in reliability.
RAG is arguably the most useful AI idea of the decade and the worst marketed. It is sold as an antidote to invention when it is really a way of forcing a model to read before it speaks — which is already a great deal. The sane rule is to treat a retrieval-grounded answer like the work of a sharp intern with keys to your archive: fast, usually on the right track and always worth a second look when something important is at stake.