A few years back, I built what I thought was the perfect monitoring setup for our payment processing system. We had a Grafana dashboard that tracked latency, error rates, throughput, and a handful of infrastructure metrics. CPU wasn't spiking, memory was fine, and every five seconds our health check endpoint returned a cheerful 200. The whole thing glowed green for months. I'd check it during my morning coffee and think, yep, everything's fine.
Then one Monday at 10:42 a.m., the support team paged me. Customers couldn't complete checkouts. The website looked normal, but the final payment authorization step just hung. I pulled up our dashboard and stared at it. All green. The health check endpoint was still returning 200 faster than I could blink. It took us 45 minutes to realize a database deadlock was silently failing the actual checkout calls while the health check—a simple ping to the app server—kept humming along. I'd been monitoring the wrong thing, and I'd been doing it for six months without knowing.
That outage was the first time I understood that monitoring and observability aren't the same. Monitoring tells you when something predefined goes wrong. Observability lets you ask questions you didn't know you'd need to ask. Our health check was a perfect monitoring signal—it was a known known. The deadlock in a stored procedure that only triggered under a specific combination of coupon codes and high load? That was a known unknown at best. Without being able to explore what was actually happening inside individual requests, we were blind.
After that mess, we started instrumenting business transactions, not just infrastructure. We adopted OpenTelemetry and added custom spans that traced a checkout from click to confirmation, including calls to the coupon service, inventory check, and payment gateway. The difference was immediate. Now, when latency spiked, I could see that it wasn't the database generally—it was one particular query in the coupon validation step that was slowing down for users with loyalty points. That level of detail is what lets you fix things before customers notice.
I made another mistake on that journey, though. We got excited and instrumented everything. Every method, every database call, every cache hit. Our tracing backend bills by volume, and the bill in month one was comically large. Plus, we drowned in noise. I spent more time filtering out irrelevant spans than debugging actual problems. We learned to be surgical—focus on the critical paths that directly impact revenue or user experience. A missing span for a log statement is fine. A missing span for the payment authorization call is not.
What I do now is walk through the most important user journeys with the development team and ask: if this thing broke silently, what would we need to know? Then we instrument exactly that. No more, no less. I still have dashboards—they're useful for a quick gut check—but I don't trust any metric unless I can click through to a trace and see a single failed checkout from start to finish. That's the only ground truth. It turns out a green light is just a liar you haven't caught yet.