We build AI systems. An audit engine that runs unattended, retrieval over business documents, assistants that know a company's hours and prices. So take it seriously when we say: a large share of the "AI projects" businesses ask about would be worse — slower, costlier, less reliable — than the boring software equivalent.
This isn't AI skepticism. It's tool selection, which is most of what engineering judgment is.
The rule
If you can write down the rule, write the rule. Use a model only when you can't.
Conventional software executes rules. If the logic of a task can be stated precisely — "when an invoice is 30 days overdue, send this email"; "copy these five fields from the order into the spreadsheet"; "reject the form if the date is in the past" — then a script implements it perfectly, every time, for close to nothing. A model implements it approximately, for more money, with occasional creative failures you have to catch.
Models earn their keep where the rule cannot be written down: reading meaning out of messy human input, summarizing, classifying things that resist crisp definitions, extracting structure from documents that follow no template. "What is this customer actually asking for?" has no if-statement. That's model territory.
What this looks like in practice
Most real automation projects are sandwiches: deterministic bread, with a thin layer of model only where meaning has to be read.
- Invoice chasing: no model anywhere. Dates, amounts, and templates. A script.
- Inbox triage: a model classifies what each message wants ("quote request," "complaint," "spam") — then rules take over: routing, drafting from templates, logging. The model touches one step out of five.
- Document intake: a model extracts fields from PDFs that follow no standard layout — then validation code checks every extraction against hard rules before anything enters your records. Trust, but verify with an if-statement.
- A site assistant: a model handles phrasing; your actual hours, prices, and policies come from structured data it isn't allowed to improvise. The worst assistants are the ones allowed to guess.
Notice the pattern: the model is a component, not the architecture. Around every model sits ordinary software doing the things models are bad at — being exact, being repeatable, being auditable, being cheap.
The failure modes nobody demos
AI failures in production are rarely spectacular. They're quiet: the extraction that's right 94% of the time, in a workflow where the remaining 6% costs more than the automation saves. The classifier that drifts as your customers' language changes. The assistant that answers a pricing question confidently and wrong.
This is why evaluation is part of the build, not an afterthought. Before an AI feature goes anywhere near your operations, you need a way to measure whether it's actually correct on your data — not impressive on a demo. If a vendor can't show you their error rate on inputs like yours, they don't know it.
Questions that sort this quickly
- Can a competent employee write the steps on one page? If yes, it's a script.
- Does the input vary in meaning, or just in format? Format variance is parsing; meaning variance is a model.
- What does a wrong answer cost? If errors are expensive and hard to catch, keep the model away from the decision and use it only to draft for a human.
- Would this have been built in 2019? If the answer is "yes, it just cost too much then" — the economics changed, not the tool. That's a different, better story.
Why restraint is the tell
Here's a heuristic for evaluating anyone who wants to build AI for you: ask them to name a project where they'd recommend against AI. If they can't, they're selling the word, not the engineering. The tools are genuinely powerful — we use them daily, on our own products and in how we build. But power tools come with a torque setting, and knowing it is the difference between a craftsman and a guy with a new drill.
Technologies keep changing. The judgment about where to point them compounds.