AI is genuinely useful and it fails in ways ordinary software doesn't. Software that breaks stops working. AI that breaks keeps going and gives you an answer that looks fine.
Almost everything that goes wrong with AI in a business comes back to that.
1. A wrong answer looks exactly like a right one
This is the one that matters most and it's the hardest to design around.
Ordinary software fails loudly. A field is empty, a page errors, a total doesn't balance. You find out because something visibly stopped.
AI hands you a plausible answer with the same confidence whether it's right or wrong. A date read in the wrong order, a total out by a factor of ten, a supplier name that's nearly right. None of it looks like an error, and it all flows straight into whatever comes next.
What to do about it: check totals rather than fields. If five numbers are extracted and they're supposed to add up, add them up. An arithmetic check catches more than reading every field ever will.
2. It makes facts up
Ask it to draft a quote and forget to give it your rates, and you'll get a quote with rates in it. Plausible ones. Wrong ones.
It's built to produce a complete answer, so a missing input gets filled rather than flagged. This is the failure that reaches customers, because the draft reads well and nobody re-checks the parts that sound normal.
What to do about it: every fact comes from you or from a system, never from the draft. Prices, dates, availability, names, stock. Give it the facts and let it do the words.
3. It's confident in the middle, where it shouldn't be
On the obvious yes and the obvious no it's excellent. On close calls it's roughly a coin toss, and it doesn't tell you it's near the line.
A person looking at a borderline case asks somebody. That instinct is what's being replaced, and nothing takes its place by default.
What to do about it: build a third answer. Yes, no, and not sure, with the last one going to a person. That pile also turns out to be the best list you'll get of what to improve.
4. It drifts, and nobody is looking
It worked in testing. Six months later the documents coming in have changed slightly, or a supplier redesigned their invoice, or somebody started sending photos instead of scans.
Nothing errors. The accuracy just slides, and because it was right for six months nobody's checking any more.
What to do about it: sample it. Ten records a week, checked by a person, forever. It's twenty minutes and it's the difference between finding a problem in week one and finding it in month nine.
5. Nobody can say why it did that
A customer queries a decision. You go to look at why, and there's nothing to look at, because all that was kept was the answer.
This is a serious problem the moment a decision affects somebody, and it's completely avoidable. It's usually skipped because leaving it out is invisible at the start.
What to do about it: keep what went in, what came out, and which version of the software produced it. Log it from day one. Adding it after somebody complains means the complaint is the one case you can't answer.
What this means in practice
None of this is an argument against using AI. It's an argument for where to put it.
Point it at jobs where a mistake is cheap and visible, and where a person sees the output as part of doing their work anyway. Reading paperwork, sorting an inbox, drafting something somebody sends.
Keep it away from being the only check on money, safety, legal or medical decisions. It can do the first pass on any of those. A person signs it off.
The businesses that get burned aren't the ones that used it. They're the ones that put it somewhere unwatched and found out months later.