The problem: a general model will invent your prices
Ask ChatGPT or Claude what your company charges to install a commercial rooftop unit and you will get an answer. It will be fluent, specific, well formatted, and completely made up. The model has never seen your rate card. It has seen thousands of other companies' rate cards, so it produces something shaped like one.
Same for your refund policy, your warranty terms, and the reason invoice terms are net-45 for three specific accounts. You do not get a refusal. You get a plausible fabrication, which is worse, because a fabrication is only obviously wrong to someone who already knew the answer.
Retrieval-augmented generation, universally shortened to RAG, is the fix. Before the model answers, you find the relevant pieces of your own documents, hand them over, and instruct it to answer from those and cite them. The model stops being a source of facts and becomes a reader of your facts. It is the most requested AI build among small and mid-sized companies, because almost every business has documents nobody can search.
How RAG works, in plain language
Two phases. You index your documents once and re-index when they change. Then you answer questions. Here is each part, with the jargon defined as it appears, because you will hear all of these words from any vendor.
Chunking: cutting documents into retrievable pieces
You cannot retrieve a 90-page handbook as a unit, so it gets split into chunks of a few hundred words. Chunks overlap so a sentence at a boundary is not orphaned. OpenAI's Retrieval API defaults to a maximum of 800 tokens per chunk with 400 tokens of overlap, and lets you set chunk size from 100 to 4,096 tokens. Google's File Search tool in the Gemini API exposes the same two knobs.
This sounds like a technical detail and is actually the decision that most determines whether the system works. A chunk splitting a pricing table down the middle retrieves half a table. A chunk cutting a policy off before its exception clause is wrong in exactly the cases people ask about.
Embeddings and the vector store: search by meaning
Each chunk becomes an embedding — a long list of numbers encoding what the passage is about, so passages with similar meaning get similar numbers. That is what lets someone search for time off after a death in the family and match a chunk headed Bereavement Leave, which shares no keywords with the question.
Embeddings live in a vector store, a database built to answer which stored items are closest in meaning to this one. If you already run PostgreSQL, pgvector adds vector search to the database you have — BSD-licensed, HNSW and IVFFlat indexes, six distance metrics, up to 16,000 dimensions. For a purpose-built engine, Qdrant, Milvus and Chroma are Apache 2.0 and Weaviate is BSD 3-Clause, all four actively maintained as of September 2026. For no infrastructure at all, OpenAI and Google will host the index.
For most companies with a few thousand documents, pgvector inside the Postgres you already pay for is the right answer, and its being unglamorous is not an argument against it. Dedicated engines earn their keep at scale, under heavy metadata filtering, or when you need isolation features you would otherwise hand-roll.
Retrieval and reranking: finding the right pieces
At question time the system embeds the question and pulls the closest chunks. Meaning-based search is bad at exact strings — part numbers, invoice IDs, statute references — so production systems combine it with keyword search. This is hybrid search, now standard enough that OpenAI's retrieval endpoint exposes it as two weights you tune, one for the embedding match and one for the text match.
Then reranking: a second, more expensive model reads the question against each candidate and scores real relevance, which the first pass only approximated. Retrieve wide, rerank narrow — in Anthropic's contextual retrieval work, top 150 chunks reranked down to 20. Cohere sells reranking as an API, currently rerank-v4.0-pro and the faster rerank-v4.0-fast; MongoDB bundles it into Atlas after its Voyage AI acquisition. It is the highest-return addition to a mediocre RAG system.
Generation with citations: the answer, and where it came from
The retrieved chunks go into the prompt with the question and an instruction along the lines of answer only from these passages, and say you do not know if they do not contain the answer. What comes back should be an answer plus pointers to the source.
That last part has moved from do-it-yourself to platform feature. Anthropic's Citations returns the exact passages supporting each claim, at sentence granularity for plain text and PDFs, with page numbers for PDFs. Google's File Search returns citations with file references and page numbers.
What RAG is genuinely good for
Policy and SOP lookup is the best starting point: highest value, lowest risk. Someone asks how PTO accrues in their first year, or the escalation path for a damaged shipment, and gets the answer with a link to the paragraph. The answers are unambiguous, and the status quo costs you a senior person answering the same twelve questions forever.
Quoting from contracts is the high-value hard version. What is the termination notice period in the Henderson MSA, which vendor agreements auto-renew, does this SOW cover after-hours work. RAG is very good at finding and quoting the clause, but it is not a lawyer: the output is a pointer for a human to verify.
Customer support grounded in real documentation is the most common commercial deployment — deflection on your help center, or a panel drafting an agent's reply from your product docs. The grounding is what separates it from the chatbots everyone got cynical about; see how AI agents, chatbots and RPA differ.
Internal search that finally works is the quiet one. Documents sit across a shared drive, an email archive, a project tool and someone's laptop, and the search in each is keyword-only and bad. A retrieval layer across all of them is often what people use ten times a day when the flashier use case goes unused.
Do you even need RAG, now that context windows are enormous?
The honest question, and dodging it is how people buy infrastructure they did not need. Frontier models now take a million tokens of input — Claude Opus 5, Sonnet 5 and Fable 5.1 all sit at a 1M-token window as of September 2026, roughly 555,000 words. If your whole relevant corpus fits, skip retrieval and paste it in.
The rule runs on three axes. Volume: if the corpus fits with room for the conversation and you can afford to send it, you do not need RAG. A 60-page handbook does not need a vector store. Four thousand contracts do. Freshness: if documents change daily, retrieval from an index you re-embed on change beats maintaining a giant prompt. Cost and latency: you pay input tokens on every call, so stuffing the corpus into every question means paying for all of it every question.
There is a quality argument too. Chroma's Context Rot research, published July 2025 by Kelly Hong, Anton Troynikov and Jeff Huber, tested 18 models and found performance varies significantly as input length grows even on simple tasks — worse when the question needs semantic matching rather than exact wording, and worse with distractors present. A bigger window is not the same as reliable attention across it.
For most businesses the answer is both: retrieve to narrow the field, then let a long-context model reason across a generous set of retrieved material rather than three stingy chunks, with prompt caching keeping stable parts cheap. The related shift is agentic RAG, where the model gets search as a tool, decides what to look up, and searches again if the result was not enough. It handles multi-part questions single-pass retrieval mangles, at the cost of more calls and harder debugging. The tool is usually exposed over the Model Context Protocol; the broader pattern is in our guide to AI agents.
Why RAG projects fail
The demo always works. Someone indexes fifty documents, asks five questions, gets five good answers, and the project is approved. Then it meets the real document set.
The source documents are a mess
Scanned PDFs with no text layer are the classic killer. Anthropic's documentation says it plainly: PDFs that are scans without extractable text are not citable. A scanned contract is an image, and without OCR it is invisible to your index while still counting in your file total.
Behind it: tables that lose structure in conversion, so a rate card becomes a column of unlabeled numbers. Decks that are mostly images. Spreadsheets where meaning lives in cell position. Three versions in three folders. Budget real time for extraction and cleanup — usually the largest line item, and the one nobody scopes.
There is no single source of truth
If the current refund policy exists in four places and they disagree, retrieval finds one of them, and which one is essentially arbitrary. The system is now confidently authoritative about a superseded policy, and no amount of reranking fixes it. The fix is editorial: decide which repository is canonical per domain, index only that, exclude the rest. Doing it honestly surfaces that several departments have been working from different documents.
Content goes stale and nobody owns it
A RAG system is exactly as current as its index. Six months in, someone updates pricing, nobody re-indexes, and the assistant quotes last year's numbers with a citation that makes it look verified — stale content plus citations is more dangerous than stale content alone. Two fixes: automate re-indexing on document change rather than a schedule someone must remember, and assign a named owner per document set accountable for whether it is current. A person, not a team.
The chunking is wrong for the content
Fixed-size chunking on structured documents is the default and frequently wrong. Chunk on semantic boundaries — sections, clauses, headers, table rows — and keep surrounding structure with the piece.
The technique worth knowing is contextual retrieval, which Anthropic published with numbers. A chunk reading revenue grew 3% over the previous quarter is useless alone: no company, no quarter. The fix is a cheap model pass prepending 50 to 100 tokens of context to each chunk before embedding. Anthropic reported retrieval failure dropping from 5.7% to 3.7% with contextual embeddings, to 2.9% combined with contextual keyword search, and to 1.9% with reranking on top — their numbers on their evaluation set, but the direction is well established.
Nobody built an evaluation set
This failure hides all the others. Without a test set you cannot tell whether a change helped, so tuning becomes vibes and regressions ship silently.
Write 50 to 100 real questions with known answers and the documents they come from, drawn from actual traffic, including hard ones, adversarial ones, and ones your documents cannot answer. Measure separately: did retrieval surface the right chunk, and did the answer stay faithful to it. Ragas, now maintained under the vibrantlabsai organization and Apache 2.0 licensed, scores faithfulness, answer relevancy, context precision and context recall; DeepEval is a comparable Apache 2.0 alternative. Run it in CI.
Permissions: the failure that becomes a data breach
This gets its own section because the failure mode is categorically different. Everything above produces a bad answer. This produces disclosure.
Your files already have permissions — the salary review folder is restricted, the board deck is restricted. Index everything into a vector store and those permissions do not come along; the store holds text and numbers. Ask what are the salary bands for regional managers and, without explicit controls, retrieval finds the chunk and the model reads it out, because retrieval has no idea who is asking.
The critical detail is where the check happens. Filtering after retrieval, in the application layer, is not enough — the confidential text has already left the index and sits in the request. Microsoft's guidance for propagating SharePoint permissions into an AI search index is explicit: store allowed users and groups as filterable fields on each chunk using stable directory object IDs rather than emails, and apply security trimming with the asking user's identity as part of the query, so authorization runs before documents are returned.
Three consequences. Your index needs per-chunk access metadata from day one, because retrofitting means rebuilding it. Permission changes must propagate, revocations included, or someone keeps access through the assistant after losing it in the file system. And most organizations discover here that their file permissions are far more open than anyone believed — retrieval does not create oversharing, it makes years of permission drift instantly queryable in plain English.
For regulated data or client material under confidentiality obligations, this is also where hosting gets decided. Sending contract text to a third-party API may be fine, may need a data processing agreement, or may be off the table. Running retrieval and the model on infrastructure you control is a real option, with trade-offs in self-hosted AI for business.
Citations and verification: it sounds right is not a test
A business-grade system shows its sources on every answer, and the citation must resolve to something a human can open and read. Not a filename — the passage.
A grounded answer and a fabricated one are indistinguishable by reading them. Both are fluent, both confident, both use your vocabulary. The only visible difference is whether the claim traces to a document. Take that away and you have something producing unverifiable assertions about your own policies, which is worse than a search box, because a search box does not editorialize.
The platforms have made this cheap. Anthropic's Citations returns the cited passage itself, and cited text does not count toward output tokens, nor toward input tokens when passed back on later turns — so the verifiable version costs roughly what the unverifiable one does. If you build your own retrieval, Claude's search result content blocks take results from your tool with a source and title and cite them automatically.
Acceptance testing has to be more than a walkthrough. Run the evaluation set, confirm every claim traces to a retrieved chunk, and specifically test questions your documents cannot answer — the right output there is I do not have that, and a system that instead produces something plausible has failed the most important test in the suite.
What it costs to build and run
Four separate meters, and most people's intuition about which one dominates is wrong.
Embedding is a one-time cost per document version and the cheapest thing in the stack. OpenAI lists text-embedding-3-small at $0.02 per million tokens and text-embedding-3-large at $0.13 per million as of September 2026; a million tokens is roughly 750,000 words. Embedding a small company's entire library is a rounding error.
Storage is per-gigabyte and ongoing, though a gigabyte of embeddings is an enormous amount of text. OpenAI gives 1 GB free and charges $0.10 per GB per day beyond it; Google currently charges nothing for File Search storage or query-time embeddings, only for embeddings generated at indexing. Self-hosted on pgvector, it is whatever your database already costs.
Retrieval is per-query and billed differently by route. OpenAI's file search tool is $2.50 per 1,000 calls on the Responses API. Managed vector databases typically meter reads, writes, storage and egress as separate lines, so model your own query volume rather than trusting a headline rate. Self-hosted, it uses compute you already pay for, and hosted reranking adds a per-search charge.
Generation is where nearly all the money goes, because retrieved chunks are input tokens on every call. Retrieve twenty chunks instead of five and you have roughly quadrupled the per-question cost of the expensive meter. Two decisions dominate the bill: how many chunks you feed the model, and which model reads them — neither is embedding or storage. Running on your own provider account keeps that spend visible, which is the case for bring-your-own-key setups.
Build versus buy
Buy when your use case matches a product's default shape. AI search over a help center, a documentation site, or one well-maintained wiki — a hosted tool beats a custom build on time-to-value and probably on quality, because the vendor tuned chunking for exactly that content. Same if your documents already live in one platform with a decent retrieval feature.
Build when one of four things is true. Your documents live in five systems no single product connects. Your permission model is non-trivial and must be enforced inside retrieval. Your content has structure a generic chunker destroys — pricing tables, clause-level contracts, engineering specs. Or the answers feed something downstream, like drafting a quote or filing a ticket.
The middle path is usually right and rarely offered: assemble a pipeline from mature components instead of writing retrieval from scratch. Postgres with pgvector, an embedding model from a provider you already use, an off-the-shelf reranker, an open-source evaluation framework, and a model you can swap. That is a system you own and can move, not a platform that holds your index.
For scope: at AutomateNexus, custom AI builds start at $7,500 with a typical timeline around 30 days, and model costs run roughly $30 to $150 a month paid directly to the provider on your own key, no markup. That assumes documents in reasonable shape; if the first month is OCR and deduplication, it runs longer. For a read on whether retrieval is even your highest-value automation, the free audit is a three-minute self-serve questionnaire returning an automation health score and ranked quick wins, no call required.
Frequently asked questions
What comes up most often when businesses scope a retrieval project.
What is RAG in simple terms?
Looking up relevant passages from your own documents and giving them to an AI model along with the question, so it answers from your material instead of general training data. It is handing someone the file before asking the question, rather than asking them to remember.
Is RAG the same as fine-tuning?
No. Fine-tuning adjusts the model's weights to change behaviour — tone, format, task style. RAG changes what facts are in front of it at answer time. If the model does not know your information, that is RAG. If it does not write the way you write, that is prompting or fine-tuning. Facts that change should never live in fine-tuning, because updating them means retraining.
Can RAG work with PDFs and scanned documents?
Text-based PDFs, yes, with citation down to the page number. Scans are different: a scanned page is an image, and with no extractable text layer the content is invisible to retrieval — Anthropic's documentation states directly that scanned PDFs without extractable text are not citable. You need an OCR step first, and OCR quality on old faxes and handwritten notes decides whether your archive is usable.
How do I stop the AI from making things up?
You reduce it substantially rather than eliminating it. Instruct the model to answer only from the provided passages and otherwise say it does not know. Require citations on every claim and display them. Improve retrieval so the right passage is actually present. Test explicitly with questions your documents cannot answer. A system with no refusals in its logs is not accurate — it is not refusing.
Will my documents be used to train the model?
For the major providers' business and API tiers the default is that API inputs are not used for training, and enterprise agreements and zero-data-retention options go further. Consumer chat tiers can differ. Verify the terms on your specific plan rather than assuming, get it in writing for client-confidential material, and if obligations are strict enough, run the model on infrastructure you control.
Do I need a dedicated vector database?
Usually not at first. If you run PostgreSQL, pgvector gives you vector search in a database you already operate and back up, which for a few thousand to a few hundred thousand chunks is adequate. Qdrant, Milvus, Weaviate and Chroma earn their place at larger scale or under heavy metadata filtering. Migrating later is cheap — the embeddings are portable.
How long does a RAG build take?
A prototype over clean documents takes days. A production system with permission enforcement, automated re-indexing, an evaluation suite and real integration is typically four to eight weeks, and document condition moves that number more than engineering does. Contracts that are scans in a shared drive mean the first stretch goes to extraction before retrieval work starts.