I once spent three weeks fine-tuning a Llama model for a customer support ticket classifier. We had hundreds of messy categories, internal jargon, and a legacy system that needed very specific tags. The baseline prompt with GPT-4 was getting about 82% accuracy on our test set. Management wanted 95%. I told them fine-tuning was the answer. I was so sure.

Looking back, I should have questioned that assumption. The training data was tiny—just 1,200 labeled examples—and heavily skewed toward common ticket types. I rented an A100, wrote some scripts, and launched training. The loss curve looked promising. But when we ran it in staging, accuracy actually dropped to 78%. Worse, the model became confidently wrong on rare categories. It would label a server outage as "password reset" because it had seen way more password reset examples.

That's when I realized prompt engineering might have been the better path. We had never really invested in systematic prompt optimization. We'd thrown together a prompt, added a few examples, and called it a day. So I sat down with a domain expert and we crafted a multi-step prompt: first identify the system, then the symptom, then match to a category using a lightweight rules layer. I embedded a few dozen carefully chosen examples directly in the prompt. Accuracy jumped to 91%. Not 95%, but close enough for production, delivered in a week instead of three.

The experience taught me that fine-tuning isn't a magic wand. It's a tool for when you have ample high-quality data and the base model's reasoning needs to be shaped toward very specific patterns. Prompt engineering is for when you can articulate the logic in words and examples, and the model just needs guidance. The crossover point isn't fixed. In my case, the handful of rare categories were the problem. A fine-tuned model could memorize them, but without enough positive examples it just overfit and hallucinated. A good prompt with explicit rules handled them gracefully.

I've since adopted a rule of thumb: exhaust prompt engineering first. Once you've got a prompt that uses chain-of-thought, few-shot examples, and maybe a retrieval step to fetch relevant context, measure the gap. If you're within striking distance and the remaining errors are consistent—like a specific pattern the model keeps missing—then sometimes a tiny fine-tune on just those edge cases works. But doing a full fine-tune without that groundwork is asking for trouble.

I've seen the opposite too, though. At another client, we needed to generate SQL from natural language in a very niche banking data model. Prompt engineering alone was brittle; the model would invent column names. We fine-tuned a model on a thousand query-log examples and got robust, schema-aware generation. So there's no one-size-fits-all. The key is to not treat fine-tuning as the next logical step after a "meh" prompt. It's a commitment, and you should only take it on when you've squeezed all you can out of instructions and examples.

Now when a project manager asks me which path to take, I say: show me your best prompt first. Let's push that as far as it'll go. Then we'll talk about training. It saves time, money, and embarrassment.

That's the approach: test, refine, measure, and only then invest in a model that might overfit your assumptions.