Retrieval-augmented generation solves a real problem: it grounds a language model in your own content instead of its training data. It is a good technique. It is not a strategy, and it will not rescue a disorganised document estate.
Retrieval quality is a content problem
A RAG system retrieves what you have. If what you have is four versions of the same policy with no indication which is current, the model will confidently cite the wrong one — and it will sound exactly as certain as when it is right.
Before any retrieval pipeline: establish which repository is authoritative, remove superseded documents, and make ownership explicit.
Evaluate on questions people actually ask
Teams evaluate these systems on questions the team invented. Collect real questions from the helpdesk queue, the sales inbox, the onboarding channel. Build the evaluation set from those, and score against it before and after every change.
Know what it is bad at
Retrieval handles "what does our policy say about X". It handles aggregation across many documents poorly — "how many contracts renew next quarter" is a database question wearing a chatbot costume. Pointing it at the wrong class of question is the fastest way to lose user trust.
The failure mode is not a wrong answer. It is a wrong answer delivered in the same confident register as a right one.