I remember the pitch meeting like it was yesterday. The vendor demo was slick. Type a question, get a perfect answer from your company's own documents. No more hunting through Sharepoint, no more asking the same person the same question for the tenth time. We signed the contract the next week.
Fast forward three months and we had a system that answered questions like a confused intern who'd only skimmed the first page of the manual. Someone asked "What's our PTO policy?" and got a paragraph about how to submit expense reports. The problem wasn't the model. It was the retrieval. We'd dumped everything into the vector store without thinking about what good retrieval actually means.
Here's the thing nobody tells you about RAG. The retrieval part is harder than the generation part by a long shot. You can have the best language model in the world but if it's pulling the wrong chunks, you get confident nonsense. We were using cosine similarity on raw document chunks and it was a disaster. A chunk about vacation policy might be semantically close to a chunk about sick leave because they share words like "days" and "request" but they're completely different procedures.
We fixed it by being brutal about chunking strategy. Instead of splitting documents at arbitrary token limits, we chunked by logical sections. Each policy document got split at the second-level heading. Each procedure got its own chunk. We added metadata tags for document type, department, and last updated date. Then we filtered the retrieval by those tags before even running the similarity search. That alone cut the hallucination rate by more than half.
The next surprise was how bad the model was at handling contradictory information. We had two different versions of the same workflow in our knowledge base because someone had updated the process but not archived the old one. The model would sometimes blend them together into a Frankenstein procedure that made no sense. We had to build a simple deduplication step that flagged documents with similar titles and required a human to pick the current version.
What finally made it work was turning it into a conversation instead of a Q&A bot. We added a feature where the system showed the user which documents it was using and let them refine the search. Power users loved it. Regular users just wanted an answer, but they learned to trust it more when they could see the source. We also added a feedback button that logged every thumbs-down along with the user's optional comment. That gave us a training set of real failures.
The whole thing took twice as long as planned and cost more than the execs wanted. But a year in, it's handling about 60% of the internal support tickets without human intervention. The support team went from dreading the same questions every week to having time to actually improve the documentation. That's the win nobody talks about. RAG doesn't just answer questions. It forces you to clean up your knowledge base. And that cleaning is where the real value lives.
I still think about that initial vendor demo and how different the reality was. The tech worked eventually, but only after we stopped treating it like magic and started treating it like an engineering problem. If you're starting a RAG project today, spend your first month on data quality, not model selection. Your future self will thank you.