Why shouldn't your data pipelines run on a timer?

It's the default because it's easy to set up, and it works — files do get processed. The problem is what it costs: a pipeline scheduled every one or two minutes runs whether or not there's anything to do, all day, every day. On a platform with dozens of feeds each polling independently, that adds up to a meaningful, constant tax on the orchestration layer for the vast majority of runs that find nothing.

We inherited exactly this pattern on a healthcare data platform: eligibility and claims-transfer pipelines checking inbound and NAS locations every minute or two, around the clock.

Event-driven triggers. A lightweight watcher process monitors the drop location and fires the specific pipeline the moment a file actually lands — no polling loop, no wasted runs. The pipeline still runs exactly as often as there's real work, just without the constant background cost of checking for it.

The other half of the fix is priority, not frequency: once pipelines are triggered by events rather than clocks, a queue can also become priority-aware, so the most business-critical run in the queue executes first instead of in arrival order. Neither change touches the actual transformation logic — it's purely an orchestration fix, which is exactly why it's low-risk to make.

In our experience, yes — disproportionately so relative to the effort involved. On the modernization project this came from, moving to event-driven triggers and priority queueing was one contributor (alongside a broader Spark/Airflow rebuild) to roughly $5M in annual cost reduction, and it's one of the cheaper changes to make relative to what it saves, because it's infrastructure, not business logic — you're not touching the rules that produce correct output, just how often the system wakes up to check for work.