A few years back, I watched our internal knowledge base slowly rot. Teams would dump documentation into Confluence, and nobody ever revisited it. Search was terrible — you'd type a query and get twenty results from 2018, most of them irrelevant. People stopped using it. They'd just ask the person who'd been there longest, which worked until that person left.
When large language models started getting good, everyone jumped to the obvious idea: just dump all our docs into a chat interface. Let the model answer questions directly. We tried that. It was worse than the wiki. The model would hallucinate confidently, mixing up old process names with new ones, inventing policies that never existed. Internally we called it the Confident Liar problem.
That's when we pivoted to RAG. The basic idea is simple: instead of asking the model to memorize everything, you retrieve relevant chunks of real documentation first, then feed those chunks as context to the model when answering. The model becomes a reader and summarizer, not a memory bank.
The first version was rough. We chunked documents by page breaks and used a basic embedding model. The retrieval was noisy — you'd ask about onboarding steps for contractors, and it would pull a snippet from a safety manual from 2015. The model would then try to reconcile the irrelevant context with the question, producing answers that were technically wrong but grammatically perfect. Users lost trust fast.
I remember one incident where a junior engineer asked about the approved vendor list for cloud services. The RAG system pulled an old procurement policy and told them they needed three signatures for any purchase over five thousand dollars. The real threshold had been raised to twenty-five thousand a year prior. The engineer followed the old process, held up a procurement ticket for two weeks, and nearly missed a deadline. That was the moment I knew we had to fix the foundation.
We changed three things. First, we invested in chunking strategy — not by page, but by semantic sections. Each document was split on logical headers, and we kept chunks small enough that one chunk could answer a question directly. Second, we switched to a better embedding model that understood domain-specific language like change control, deployment windows, and SOC compliance. Third, we added a verification step: the system would show the user which document it was citing, with a direct link. If the model couldn't find a relevant chunk, it would say I don't know instead of making something up.
The difference was night and day. Users started coming back. They could ask a question and get an answer that cited the actual approved policy from last quarter. The link let them verify in seconds. Trust rebuilt slowly, but it rebuilt.
One thing that surprised me was the operational overhead. RAG isn't a set it and forget it solution. When a document gets updated, you have to re-embed that chunk. If you don't, the system retrieves the old version. We set up a webhook on our document repository so that any edit triggers a re-embed job. Simple in concept, but someone has to maintain that pipeline.
Looking back, the biggest lesson wasn't technical. It was that users will forgive a search engine that returns bad results. They won't forgive a chat interface that sounds confident and wrong. RAG works when you treat it as a retrieval problem first and a generation problem second. Get the retrieval right, and the model can do its job. Get it wrong, and you're just automating misinformation.
These days, when I see a team excited to bolt an LLM onto their knowledge base, I ask them one question: how do you handle the case where the answer isn't in the documents? If they don't have a plan for that, they're not ready for RAG.