I spent years wiring up supply chain systems with nightly batch jobs. It worked until a major client wanted real-time inventory visibility across three warehouses and a drop-ship network. That's when I proposed event-driven architecture. I'd read about it, seen conference talks, figured it was the answer.
The first mistake was assuming events would be clean. We put a message broker in place, had the warehouse management system publish order shipped events, and the inventory system subscribe. Within a week stock counts started drifting. The inventory team was furious. They'd see an item show zero in the warehouse, then a customer order would come in and suddenly there were five. Turns out the WMS published an event when a pick was created, not when it was physically scanned out the door. We had built a real-time system on top of a process that was still logically batch at the operational level.
We traced one discrepancy to a returns process. A pallet would arrive at dock door two, get scanned as received, then later moved to a quarantine area. Two events: pallet received and pallet quarantined. They often arrived out of order because the dock scanner and quarantine scanner used different network segments with different latency. The inventory service applied quarantine first, then received, and the item flipped to available. Classic out-of-order event problem. We hadn't included sequence numbers or a business timestamp that could be used for reconciliation. We thought the broker guaranteed order, but only per partition, and we had multiple producers writing to different partitions.
The fix wasn't glamorous. We added a correlation ID to every physical unit and a business timestamp from the scan device, not the broker's ingestion time. Then the inventory service kept a small state store for last seen event per unit, ignored out-of-order duplicates, and triggered a reconciliation job every hour instead of trusting the stream blindly. That hybrid approach saved us. Pure event streaming was too fragile for the messy reality of handheld scanners and flaky dock WiFi.
The bigger lesson was that event-driven architecture doesn't remove the need to understand business process boundaries. If your events don't match what actually happens on the floor, you've just built a faster way to propagate lies. The real value came later when we used events to predict delays, like a truck check-in event followed by no dock assignment for thirty minutes could flag a bottleneck before the warehouse manager noticed.
I still like event-driven systems, but now I ask one question before designing anything: what is the source of truth for this event, and what happens when two events arrive in the wrong order? If you can't answer that, keep your batch job a little longer. Maybe event-driven architecture is less about speed and more about being honest about how messy your operations really are.