Who decides when an AI model is good enough?
It is easy to try an AI model on a few examples and decide it looks good. The chatbots of a Chevrolet dealership, Air Canada and DPD, and Google's flu-tracking model, all failed in live use in situations nobody had tested beforehand. Deciding what "good enough" means is a business decision, and it has to be made before the results come in.
In conventional software projects, acceptance criteria are written before development starts. The finished product passes or fails against them, and everyone knows in advance what “done” means. AI projects rarely work this way. A model is built, someone tries it on a few examples, the answers look reasonable, and “it looks good” quietly becomes the decision to go live.
That habit is risky for a simple reason. An AI model does not fail the way conventional software fails. It can answer most questions well and a few badly, behave differently after an update, or become less accurate over time as the data and the way people use it change. A handful of tries will not reveal any of this. Only an evaluation agreed in advance will: a set of test cases, the measures that matter, and the threshold the model must reach. Deciding what goes into that evaluation is a management question, not only a technical one.
Failures nobody had tested for
The best-known AI failures of recent years were rarely failures of the model itself. Each one exposed a question nobody had put to the system before it went live.
Requests outside the job. In December 2023, users discovered that the ChatGPT-based sales assistant on the website of Chevrolet of Watsonville, a California dealership, would do almost anything it was asked. It wrote Python code, recommended a competitor’s truck and, after a user instructed it to agree with everything, accepted an offer to sell a 2024 Chevy Tahoe for one dollar, calling it “a legally binding offer.”1 The vendor described the episodes as pranks and restricted the bot. What had not been tested was what the assistant should refuse to do.

The post that made the dealership’s chatbot famous. Chris Bakke on X, 18 December 2023.
Answers that contradict the company’s own rules. When Jake Moffatt asked Air Canada’s website chatbot about bereavement fares, it told him he could buy a full-price ticket and claim the discount within 90 days. The airline’s actual policy did not allow this. Before a British Columbia tribunal, Air Canada argued that the chatbot was a separate legal entity responsible for its own actions. The tribunal called that argument remarkable, rejected it, and ordered the airline to pay the difference in February 2024.2 The missing test was whether the chatbot’s answers matched the company’s own policies.

The Register, 15 February 2024.
Behaviour that changes after an update. In January 2024, a customer of the UK parcel company DPD asked its chatbot for help, got none, and then persuaded it to swear and to write a poem about how useless DPD was. His post spread widely. DPD said the error had occurred after a system update and that it had immediately disabled the AI element.3 A system that behaved acceptably before the update was not tested again after it.

Ashley Beauchamp on X, 18 January 2024. DPD disabled the AI element of its chat after the post spread.
A world that moves on. The problem is not limited to generative AI. Google Flu Trends estimated flu activity from search queries and was, for a time, held up as a model for using big data in public health. In February 2013 it predicted more than double the share of doctor visits for flu-like illness that the US Centers for Disease Control and Prevention reported.4 Researchers later pointed to changes in search behaviour and in Google’s own search engine, and to estimates that had not been checked against real-world data often enough to notice in time.

Nature, 13 February 2013.
Three decisions that define good enough
Each of these cases points to the same three decisions, and they should be made before the model is built.
| Part | The question it answers | Who should decide |
|---|---|---|
| Test set | Which cases must the model handle, including the hard and hostile ones? | Business experts and the technical team together |
| Measures | Which errors matter, and which are unacceptable? | The business owner |
| Threshold | What result is good enough to go live? | The business owner, before results are known |
The technical team builds the test set into an automated check and runs it. The business decides what goes into it and what counts as a pass, because only the business knows the difference between an awkward answer and an expensive one. A chatbot that sounds slightly stiff is a small problem. A chatbot that invents a refund policy is a legal one.
Morgan Stanley shows what this looks like when it is done well. Before rolling out an AI assistant to its financial advisors, the firm had advisors and engineers grade the model’s summaries for accuracy and coherence, and it ran daily regression tests with sample questions to catch weaknesses before they reached clients.5 The people who would rely on the answers helped decide what a good answer was.
The evaluation does not end at launch
DPD’s chatbot and Google Flu Trends failed after they had been working. That is why the same test set has to run again after every update and at regular intervals once the system is live, with results compared against what actually happens. The evaluation thus becomes the basis for tracking performance in production. As we discussed in an earlier piece, that tracking is one of the tasks a PoC never needs but production cannot do without.
In the stage-gate model we have described, the evaluation is the evidence at the feasibility gate and again before scaling. Four questions help executives test it. Who wrote the test cases, and do they include the situations the business fears most? Which errors are unacceptable, and who decided that? What result must the model reach, and was that threshold set before the results came in? When will the tests run again? An AI project that cannot answer these questions has not yet decided what “good enough” means, and someone else, a customer or a tribunal, may end up deciding for it.
Footnotes
-
Gizmodo, I’d Buy That for a Dollar: Chevy Dealership’s AI Chatbot Goes Rogue, December 2023; original post by Chris Bakke on X, 18 December 2023. ↩
-
Moffatt v. Air Canada, 2024 BCCRT 149, 14 February 2024; The Register, Air Canada must pay damages after chatbot lies to grieving passenger about discount, 15 February 2024. ↩
-
TIME, AI Chatbot Curses at Customer and Criticizes Company, January 2024; original post by Ashley Beauchamp on X, 18 January 2024. ↩
-
D. Lazer, R. Kennedy, G. King and A. Vespignani, The Parable of Google Flu: Traps in Big Data Analysis, Science 343, 14 March 2014; D. Butler, When Google got flu wrong, Nature, 13 February 2013. ↩
-
OpenAI, Morgan Stanley uses AI evals to shape the future of financial services. ↩