I remember standing on a factory floor in Ohio, staring at a conveyor motor that had just seized up, while our dashboard proudly displayed a 92% probability of failure—three hours after the line was already down. That moment summarized our entire first attempt at predictive maintenance.
We had all the right ingredients, or so we thought. Vibration sensors on every critical motor, temperature probes on the bearings, a data lake swallowing everything at one-second intervals. The data science team built a random forest model that looked great during validation. Precision, recall, F1 score—all above 90%. We were ready to save the plant millions in unplanned downtime.
The first month in production was humbling. Alerts would fire, but by the time a technician reached the machine, it was either already broken or running perfectly fine. The ones that were broken had always failed before the alert. We were essentially building a very expensive system to confirm what the operators already knew: the motor is dead.
The root cause wasn't the algorithm. It was our labeling. We had pulled maintenance records and tagged any work order with a failure code as a positive example. But many of those work orders were for scheduled replacements or inspections that happened at regular intervals, not unexpected breakdowns. The model had learned to pattern-match the clock and calendar, not the physics of degrading bearings.
We went back and re-labeled, this time with the help of two senior maintenance techs who had been at the plant for decades. They taught us that the real failure mode often started days before the machine stopped, with subtle changes in sound or heat that the sensors might capture but that their ears and hands already knew. We also realized we had a time-window problem. If we labeled a failure at 3 PM on Tuesday, but the model was trained to flag it based on data from Monday morning, we needed a buffer. Without it, the model would see normal Monday operation and associate it with a failure that hadn't happened yet, then miss the actual pre-failure signatures because those were too close to the event.
That's when I made a rule for myself: don't trust any label you didn't witness being created. For the next iteration, we did a supervised data collection run where techs manually tagged equipment state for two weeks. It was slow and expensive, but it gave us ground truth. We could then identify the exact sensor patterns that preceded a real breakdown, not just the ones that correlated with a shift change or a production rate adjustment.
We also learned to measure success the way the plant manager did. Accuracy meant nothing to her. She cared about two numbers: how many hours of downtime were avoided, and how much money was saved on emergency repairs. So we started tracking lead time—how early before failure the model gave a reliable alert—and false alarm rate. If the lead time was shorter than the time it took to safely stop the line, the prediction was useless, even if mathematically correct. We ended up tuning our model to be excessively cautious, sacrificing precision for recall, because a false alarm cost a few minutes of inspection while a missed alarm cost thousands.
Looking back, the technical modeling part was maybe twenty percent of the effort. The rest was data plumbing, operational context, and building trust with the people who actually run the equipment. Now when I start a predictive maintenance project, I spend the first week just watching the machines with the operators, asking them what they listen for, what they feel. It's not glamorous, but it prevents a dashboard that tells you what you already know.