I remember the demo that sold the whole idea. We asked it how to submit a purchase order for lab equipment, and it spat out a perfect, step-by-step answer sourced from the latest SOP. The room was electric. We thought we'd cracked internal knowledge. Six months later, an engineer in the biotech division followed those steps and ordered a six-thousand-dollar centrifuge without the required safety review. Why? Because the AI had pulled its answer from a deprecated SOP that was still sitting in Confluence with no warning banner.
That was the first time I realized RAG isn't just about retrieval quality. It's about the absolute state of the documents you're retrieving from. We'd spent three months fine-tuning the embedding model and chunk size, testing different vector databases, and arguing about cosine similarity thresholds. We never once asked the question that actually mattered: is the underlying content reliable? Spoiler: it wasn't. Our Confluence space was a graveyard of drafts, duplicates, and decade-old pages that had been superseded by a Jira ticket someone forgot to link.
The second mistake was assuming that more retrieval was better. We set the top-k to 20, thinking the generator would be smart enough to synthesize from a rich set of sources. Instead, the response quality oscillated wildly. When a user asked about our VPN troubleshooting steps, the model would blend the current guide with instructions from a 2017 document that referenced network hardware we'd decommissioned years ago. The result read like coherent English but was operationally dangerous. We had to dial back to a much smaller k and invest a ridiculous amount of time in metadata filtering: tagging every document with a last-reviewed date, an owner, and a lifecycle status. Our DevOps team hated it, and honestly I didn't blame them.
What finally turned the corner was a simple human process. We identified the ten most frequently asked question types coming into IT support, found the canonical source for each, and locked them. Only those documents got indexed in the production RAG pipeline. Everything else went into a sandboxed version. It felt like cheating after all the automation we'd imagined, but it worked. The number of incorrect answers dropped to near zero overnight. We've since expanded the curated set, but we treat new entries like a software release, they go through a review before joining the index.
Looking back, I think we fell for the narrative that RAG is a tool that magically organizes chaos. It's not. If your knowledge base is a mess, RAG will just surface that mess faster and with a veneer of confident phrasing. The real work is the boring, pre-retrieval work: auditing your source material, deciding what should never be retrieved, and accepting that some manual curation is unavoidable. I still believe in the technology, but now I start every project by asking the team the same question: would you trust this content if you read it yourself? If the answer is no, no amount of vector math will fix it.