Ask the same question twice and you get two numbers. Now pick one for the board.
It’s 4:40 on a Thursday. The board pack goes out at six.
Priya, head of MI at a mid-sized lender, types a question into the new AI assistant: What was revenue from our top ten clients last quarter? A query flickers across the screen, a chart settles into place, and there it is: ₹142 crore.
It looks right. It’s formatted beautifully, and it arrived in four seconds instead of the four days it used to take.
She pastes it into slide seven.
The real number was ₹47 crore.
How a confident number goes wrong
Nothing broke. That’s the uncomfortable part.
The assistant did what it was built to do. It read the question, wrote a SQL query, and ran it. Along the way it joined the clients table to the accounts table to filter correctly. But each of those clients holds three accounts, one for each product line, so every transaction got counted three times.
No error message appeared, and there was no warning or red flag. The query ran cleanly and returned a number that was wrong by a factor of three, and nobody in the room could tell. (The figures here are illustrative, but the failure is not.)
This is the part of text-to-SQL that never makes it into the demo. A broken query is harmless because it announces itself. The dangerous one is the query that runs and quietly answers the wrong question.
It happens in three quiet ways
- The join that multiplies. One row becomes three, or twelve, and the total inflates by a factor just believable enough to pass.
- The sum that shouldn't be a sum. Revenue adds up across months. Balances don't. Add twelve month-end balances together and you get a figure twelve times too large that still looks like a balance. The model can't know this unless someone wrote the rule down somewhere it's forced to read.
- The question it decided you asked. Does "last quarter" mean Q2, or the last ninety days? Does "top clients" mean top by revenue or top by exposure? The model picks a reading and doesn't mention it. Your number is now the correct answer to a question you never asked.
Right once isn't good enough
Suppose the query is correct. There’s still a problem.
Language models are probabilistic by design. Ask the same question on Tuesday and again on Thursday and you may get two different queries, with a different join, a different filter, or a slightly different idea of “top ten.” Sometimes they agree. Sometimes they don’t.
For poking around in the data, that’s fine. For management reporting it’s fatal. If a number can’t be reproduced, it can’t be audited. If it can’t be audited, nobody will sign it off. That is precisely where most enterprise AI pilots go to die. They don’t fail for lack of accuracy. They fail because nobody can defend the output.
The model isn't the villain
It’s tempting to conclude that AI doesn’t belong anywhere near the numbers. That’s the wrong lesson.
Language models are remarkably good at two things analytics badly needs. The first is understanding what someone meant: they can take a rushed, ambiguous question and turn it into a precise request. The second is explaining what a number means once it exists, including what moved, why it matters, and what to check next.
The problem is the job in between. Arithmetic has to be exact, and it has to be identical every time. It has to follow rules the business agreed on once, not rules a model reinvents on every call.
That work belongs to plain, deterministic code running against measures that have been defined in advance. The same question should produce the same calculation and the same number, whether it’s Tuesday or Thursday and whether Priya asks it or her CFO does.
Let the model read and write. Just don’t let it count.
Three questions for your next demo
When a vendor shows you text-to-SQL, don’t ask whether it’s accurate. Ask this:
- "Can we run the same question five times and see if the query changes?"
- "Where does the system learn that balances can't be added across months?"
- "If this number is wrong, how would I find out before my board does?"
If the answers get vague, you’ve spotted the pitfall before it spots you.
As for Priya, she caught it at 5:52, because her gut said ₹142 crore was too good. Not everyone’s gut is that reliable, and nobody should have to rely on theirs.
