I spent most of last spring trying to fine-tune a language model to answer internal IT support questions. We had thousands of past tickets, nicely categorized, and I was convinced that feeding them into a fine-tuning run would give us a custom model that knew our systems better than any off-the-shelf option. The data was there, the compute budget existed, so why not? Six weeks later, we had a model that was marginally better on our test set but still hallucinated when someone asked about a printer that didn't exist. The whole thing felt like a waste.
Then a junior engineer on my team built a simple script that pulled the five most similar past tickets and stuck them into the prompt along with the user's question. No fine-tuning, just a well-written system message and a few examples. On the same test set, it matched our fine-tuned model within two percentage points, and it took an afternoon to build instead of a quarter of a year. That was humbling. I had skipped the obvious step because fine-tuning felt more like real engineering, more permanent, more like something I could put in a slide deck.
The truth is prompt engineering gets you ninety percent of the way for most enterprise use cases that don't require deep domain expertise. If your task is classification, summarization, or simple question answering over a known knowledge base, a good prompt with retrieval augmented generation will often do the job. You can iterate in hours, adjust edge cases by tweaking instructions, and you keep the flexibility to swap the underlying model later without retraining anything. That last point matters more than people think when you're locked into a vendor contract.
But I've also been on the other side, where prompt engineering was never going to work no matter how clever we got. A client in regulatory compliance needed a model to extract specific clauses from contracts and output them in a strict JSON schema with no deviation. Prompts got us maybe eighty percent consistency, but the other twenty percent would occasionally drop a field or reorder the keys, and in a legal context that's unacceptable. We ended up fine-tuning a seven billion parameter model on a few thousand synthetically generated examples and it nailed the schema every time. The fine-tuning didn't make the model smarter, it made it more obedient. That's the key distinction.
So when do you actually need fine-tuning? When prompt engineering fails repeatedly on the same failure modes, when you need consistent output formatting down to the character, when you're handling a niche vocabulary that the base model has never seen, or when response latency is so tight that you can't afford long prompts with tons of examples. In those cases, stop patching prompts and start building a dataset. But don't start there. Start lazy.
I now keep a rule of thumb: two weeks of prompt engineering before I even open the fine-tuning docs. If after two weeks the model is still fighting me, then I have a clear list of failure cases to guide the dataset build, and I know exactly what I'm paying for. That's the mistake I made last spring, not trying the simple thing first because it didn't feel like enough effort. Sometimes the cheapest solution is the right one, and you don't get bonus points for overcomplicating it.