When our CTO first asked us to find use cases for large language models outside the usual customer service chatbot, I rolled my eyes. I'd seen plenty of demos, and most of them felt like parlor tricks. But then we started looking at the backlog of vendor security questionnaires. Every quarter, our procurement team spent two weeks manually filling out the same 200 questions about data handling, encryption, and access controls. We thought an LLM could draft responses based on our past answers and policy docs. That first attempt was a disaster.
I gave the model a folder of previous questionnaires and our security whitepaper, then asked it to answer a new one. The responses read beautifully, but they were wrong in subtle ways. It confidently stated we used AES-256 for data at rest when our standard was actually AES-128 for some legacy systems. It referenced an SOC 2 report we didn't have. We almost sent that to a client before a junior analyst caught the error. That was my mistake: I trusted the fluency of the output over the factual grounding.
So we changed the approach. Instead of letting the LLM generate freeform answers, we built a retrieval system that first searched our approved answer library and policy documents. The LLM's job was to match the question to the closest existing answer, then rephrase it slightly to fit the specific wording of the questionnaire. If no match existed, it flagged the question for human review. Suddenly, the two-week task became a two-hour review session. The LLM wasn't writing new content; it was routing and massaging what we'd already vetted. That distinction saved us from embarrassing compliance issues.
Next we turned to incident postmortems. Every major outage generated a 50-page Slack thread and a dozen scattered documents. Engineers dreaded writing the summary. We used an LLM to read the thread, pull out the timeline of events, identify the root cause candidate, and draft a first version of the postmortem. But we learned to keep a human in the loop for anything involving blame. The model once summarized a timeline in a way that made it look like one engineer caused the outage, when actually a misconfigured load balancer was the culprit. The engineer was furious. Now the LLM extracts facts, but the team lead writes the narrative.
The most surprising use case came from our legal team. They started using an LLM to compare old contracts against new ones, flagging clauses that had changed or were missing. It wasn't perfect, but it cut the initial review time by half. One lawyer told me she'd rather have the LLM miss something and check herself than wade through 80 pages of legalese from scratch. That's the practical bar: not perfect, just faster than the alternative.
The lesson I keep coming back to is that LLMs in the enterprise work best when they're constrained, not open-ended. Give them a narrow task, a source of truth, and a human checkpoint. Let them handle the grunt work of reading, matching, and drafting, but never let them be the final authority. If I'd tried to build a general-purpose "AI analyst" that answered any question about our infrastructure, we'd still be cleaning up the mess. Instead, we built a series of small, boring tools that save real hours every week.
That's the unglamorous truth: in the enterprise, the chatbot was the gateway drug. The real value is in the unsexy document triage, the questionnaire responses, the postmortem drafts. And the teams that figure that out first are the ones who stop treating LLMs as magic and start treating them as very fast, slightly unreliable interns.