Rules, ML, or LLMs — how do you choose for document automation?

Because most documents don't need one. On a project processing roughly 1.6 million faxed pharmacy forms a month across 700+ locations, running every page through an LLM would have been slow and needlessly expensive — and a large share of those forms are unambiguous enough that a keyword rule can classify them with total confidence in milliseconds.

The mistake is treating 'which technology' as a project-level decision. It's a per-document decision, and the right system makes it automatically, page by page.

Cheapest checks first. Keyword search attempts classification; a 100%-confidence match passes immediately with no model call at all. Only pages the rules can't confidently resolve go to an LLM with few-shot examples drawn from a reference set for that category. The same logic applies to extraction: regex handles structured fields, an LLM handles the complex or free-form ones.

Underneath all of it sits a confidence gate: anything — classification or extraction — that scores below a threshold (we used 90%) routes to a human review queue instead of flowing through automatically. The system doesn't have to be right about everything; it has to know when it isn't sure.

From real data, not assumptions. Before building the classifier for the pharmacy-forms project, we clustered a full month of actual production volume — about 1.6 million forms — and let the categories emerge from that (50+ distinct types surfaced this way). Guessing the taxonomy from a handful of samples is one of the most common ways these systems under-perform in production: the categories you didn't anticipate are exactly the ones that show up at 2am with no rule to catch them.