When my team first pitched using RAG for our internal knowledge base, I was all in. The idea was beautiful: take thousands of Word docs, PDFs, and SharePoint pages, throw them into a vector store, and let people ask natural language questions. No more hunting through folders. No more "ask Bob" for the tenth time that week.
We built the prototype in two weeks. It worked great on the five documents we tested. The CEO asked "What's our data retention policy on customer PII?" and got a perfect answer with citations. We demoed it to the board. Everyone was thrilled.
Then we pointed it at the real knowledge base. All 15,000 documents. And it fell apart.
The first problem was chunking. We'd naively split documents by page, which worked fine for clean PDFs but destroyed tables, code snippets, and regulatory clauses that spanned page breaks. A contract termination clause got split in half, and the system started answering "Can we terminate early?" with the wrong half of the sentence. We had to rebuild our chunking logic to respect document structure — headers, sections, lists — and even then, it wasn't perfect.
The second problem was retrieval quality. The vector search kept pulling up documents that were semantically similar but practically useless. Someone asked "How do I request a software license?" and got back a five-year-old memo about the licensing audit process instead of the current request form. We had to add hybrid search — keyword matching on top of vector similarity — and boost the recency of documents in the ranking. That helped, but it meant we had to tag every document with a date and department, which nobody had done before.
Then came the hallucinations. Not the dramatic kind where the system invents facts out of thin air. The subtle kind where it combined two unrelated documents and produced a confident answer that was technically true but completely wrong in context. A user asked about travel policy for contractors, and the system stitched together the contractor policy from 2019 with the employee travel policy from 2023, producing a Frankenstein answer that approved per diem rates that didn't exist anymore. We added a guardrail that forced the system to only use documents from the same department and date range, and we started logging every query where the answer contradicted the highest-ranked source.
The real lesson was that RAG is not a set-it-and-forget-it solution. It's a continuous engineering effort. You need to monitor what people are asking, tune your chunking strategy, update your embedding models, and clean up the source documents constantly. We ended up building a feedback loop where users could flag bad answers, and those queries went straight to the top of our backlog.
Would I do it again? Yes. But I'd spend the first month just cleaning and structuring the knowledge base, not building the RAG pipeline. The model is only as good as the data it retrieves, and most enterprise knowledge bases are a mess. If you're thinking about this for your own org, start by asking who owns each document, when it was last updated, and whether it's actually accurate. If you can't answer those three questions, the RAG system won't either.