A data error does not cost what it costs to fix. It costs what it costs to fix, multiplied by how long it was allowed to live.
That is the number almost nobody calculates before a project starts, and it is the reason data quality loses every budget argument it enters right up until the week it becomes the only thing anyone wants to discuss.
What poor data quality actually costs
Gartner puts the average cost of poor data quality at $12.9 million per organisation per year. In AI, that figure hides well, because the damage rarely arrives labelled as a data problem. It shows up as a launch that slips two quarters, an evaluation that will not reproduce, or a model quietly making small wrong calls that only surface when the quarterly numbers land.
Average annual cost of poor data quality per organisation, according to Gartner research.
The error is cheap. The delay is what you pay for
A single mislabelled record costs almost nothing. What it costs depends entirely on where in the pipeline it is caught.
| Where it is caught | What it takes to fix | Realistic cost |
|---|---|---|
| Guideline stage, before labelling | One conversation and a rewrite | Hours |
| During review, within the first fortnight | Targeted relabel of a defined batch | Days |
| After training | Correct the data, retrain, re-evaluate | Weeks |
| In production | All of the above, plus every decision made while the error was live | Unbounded |
The error itself never gets worse. The number of things built on top of it does. This is why speed of detection is a more useful metric than accuracy rate, and why we treat it as the primary measure of a delivery team's health rather than a secondary one.
Quality is decided before the first label is applied
The most expensive failures in this field are not sloppy. They are decisions that sounded reasonable in the planning meeting.
The pattern we see most often is a definition problem wearing a data problem's clothes: every record is accurate, every annotator is consistent, and the wrong thing has been measured. A proxy variable stands in for the thing you actually care about, the model learns it perfectly, and nobody discovers the gap until the system meets a population or a scenario the proxy never described.
No amount of annotation throughput protects you from that. The protection is procedural. Before a single record is touched, someone senior enough to challenge the brief has to ask whether the label definition and the business decision are the same thing. Then the edge cases get written down. In our experience the exception taxonomy, meaning the documented rules for the awkward five percent, does more to determine final model quality than the label set itself.
Agreement is a measurement, not a promise
Any vendor will tell you they have a quality process. The question that separates them is whether they have a number.
Inter-annotator agreement measures how often two trained people, looking at the same item independently, reach the same conclusion. Across the projects we run at Impact Outsourcing we hold 0.94, and we hold it the boring way: calibration rounds before production, a real reviewer-to-annotator ratio, and adjudication time budgeted rather than borrowed.
Inter-annotator agreement sustained across production projects, tracked daily rather than quoted once.
Holding a score is harder than reaching one. Consistency tends to decay a few weeks into a project as interpretations of the guidelines quietly diverge. Teams that only sample randomly find the obvious errors and miss the systematic ones, which are the errors that actually damage a model. Tiered review, targeted at the items most likely to be ambiguous, is what catches drift while it is still cheap.
Four questions worth asking any data partner
Ask for the current inter-annotator agreement figure and whether they will report it to you continuously. Ask to see the exception taxonomy from a comparable project. Ask how review is structured and how ambiguous items are routed. Ask who is accountable by name, because a managed team remembers what you decided in week two and a marketplace does not.
A partner who can answer all four is running an operation. A partner who cannot is running a queue.
Where better data will not save you
This is the part worth being straight about.
When RAND studied why AI projects fail, the most cited cause was not data. It was leadership: 84 percent of the practitioners interviewed pointed to business objectives that were poorly defined or poorly communicated. Data problems came second, at 60 percent.
Clean data aimed at the wrong question is still a write-off. If nobody can state in one sentence what decision the model is meant to make and what a good outcome looks like, better annotation will only reach the wrong answer faster and with more confidence. That step costs nothing, and it has to happen first.
Frequently asked questions
What does poor data quality cost a company?
Gartner estimates at least $12.9 million per organisation per year on average. In AI the cost usually appears as delayed launches, model retraining and failed deployments rather than as a line item labelled data.
Why do most AI projects fail?
RAND found more than 80 percent fail, twice the rate of IT projects without AI. The most cited causes were poorly defined objectives, at 84 percent of interviewees, and inadequate or poor quality data, at 60 percent.
What is a good inter-annotator agreement score?
It depends on the task, since subjective judgement work scores lower than object detection. The more useful question is whether the score is measured continuously and reported to you, rather than quoted once during procurement.
Can bad training data be fixed after a model is trained?
Yes, but you pay twice. You correct the data, then retrain and re-evaluate, and you still carry every decision the model made while the error was live.