Your agreement score cannot see half your errors

Annotation error has two components, and most quality systems measure one of them.

The first is variance. Annotators see the same item and label it differently. That is what agreement scores capture, and it is what almost every QA process is built around.

The second is bias. Annotators see the same item and all label it the same way, and the way is wrong. Agreement cannot detect this, and not because the measurement is done badly. Agreement measures spread between people. Bias does not create spread. It moves everyone in the same direction at once.

So you can hold 0.95 agreement on a class that is systematically mislabelled, indefinitely, and the number will never move.

What agreement actually measures

Inter-annotator agreement is the degree to which independent annotators assign the same label to the same item, corrected for the agreement you would expect by chance. Nothing in that definition mentions being correct.

Read the definition again and the trap is obvious. A guideline with a gap in it does not produce chaos. It produces a convention. Somebody decides how to handle the ambiguous case, the decision spreads through the team, and inside a week everyone handles it identically. Agreement climbs. If the convention is wrong, accuracy falls at the same time, and the two movements look identical on a dashboard.

The number flatters you as well

Even the variance half is easy to overstate. Two annotators label 100 images for whether a truck is present. Ninety have no truck. Both say “no truck” most of the time, because most of the time that is right. They agree on 92 items.

92 percent agreement. Now correct for chance. If each says “no truck” around 92 percent of the time independently, they would land on the same answer by luck on roughly 85 percent of items. Cohen’s kappa is observed agreement minus expected agreement, divided by one minus expected agreement.

Identical data. One number reads as near perfect, the other as barely usable, and the second one is honest. Any vendor quoting raw percent agreement on an unbalanced task is quoting the flattering figure. See how our QA tiers report it.

The tell for bias

Since agreement cannot find bias, you need to know what bias looks like from the outside. The signal I trust most is unanimity plus speed on a class that should be hard. When a team is both fast and unanimous on a genuinely contested category, somebody has usually found a shortcut, and it has propagated.

Three practices follow from that. Agreement is reported per class, never as one account-level average, because a single broken class vanishes into a good mean. Gold sets are built from the cases where the guideline is thinnest, not from representative samples, since a gold set of easy items only proves that easy items are easy. And guidelines are re-audited when the data shifts rather than on a calendar, because new cameras and new locales are exactly where unwritten conventions form.

What it still will not tell you

Whether the taxonomy fits the problem. Whether the gold set is itself correct. Whether the data you chose to label is the data the model needed.

Agreement can be strong, accuracy against gold can be strong, and a model can still underperform because sampling under-weighted the cases that mattered. Annotation was not the constraint. No agreement figure would have said so, which is the same reason a data error costs what it costs.

That is the honest limit of the number. We publish it because it is a real measurement of a real property of our delivery, and because a buyer should be able to interrogate it rather than take it on trust.

Frequently asked questions