Category 02 / Validation & QA
An independent validation operation for AI training data. Multi-tier review, inter-annotator agreement scored weekly, gold-standard benchmarking, and written accuracy commitments. Runs over the data we label, or the data you bring us.
Operating model
The review pipeline
Quality is enforced on the way through the pipeline — not retrofitted at the end of a batch. Each stage has a defined output, an accountable role, and a gate criterion.
Operators train and pass gold-set calibration before production access. No pass, no production.
A schema-trained operator labels the batch inside the agreed tool, under the agreed convention.
A second operator reviews without seeing the first pass. Disagreements surface as measurable IAA.
A QA lead resolves disagreements against the schema and a gold reference — not against whichever pass was loudest.
Ambiguous frames and schema gaps route to your data lead. Drift by class surfaces in the next week’s standup.
The four capabilities
The pipeline is the scaffolding. These four practices are what the scaffolding is there to hold up.
Primary annotator, independent second pass, QA lead arbitration. The lead is not the delivery lead — the same person cannot sign off on their own throughput.
Inter-annotator agreement computed at the level that is actually useful — by operator, by class, by batch. Drift on a single class surfaces before it contaminates a release.
A gold-reference set drafted with your team becomes the benchmark operators calibrate against and that lead arbitration resolves to. Revisited as the schema evolves.
Schema gaps and ambiguous frames route to your data lead with context and a proposed resolution — not silently labelled wrong, not buried in a retrospective.
Where we apply it
The QA operation is self-contained. Most often it sits over our own annotation pipeline — but it applies just as cleanly to datasets you’ve labelled in-house, or received from another vendor.
The default mode. Our QA lead is a separate role from the annotation delivery lead, so the same team that ships the labels is not the team that signs off on them. Calibration, IAA, and exception flagging all run continuously, against your written accuracy threshold.
Bring us a dataset that was labelled in-house or by another vendor and we’ll run the same review pipeline over it — calibrate the reviewers to your schema, score agreement, surface drift, flag exceptions. Useful before a retraining run, an acquisition, or a vendor handoff.
Written commitments
Specific thresholds are agreed per engagement and live in the SOW. What doesn’t vary is the shape of the commitment.
Before production starts, we agree a written IAA threshold for the dataset — set against a gold reference your team co-drafts. Miss it, and rework is on us, not on your next release.
Weekly IAA and exception reports, delivered into your channel of choice. Monthly trend packs for your data lead. No "we’ll get numbers next week" — the numbers are the deliverable.
The five-stage pipeline is the same on week 1 and week 52. Headcount scales, techniques scale, the discipline does not get compressed away under delivery pressure.
Specific SLA numbers — threshold value, rework turnaround, report delivery day — are set per engagement. We’d rather hit an agreed number than publish a generic one. Treat these as shapes of commitment, not line items.
Applies across
The same review discipline applies whether the underlying labels are bounding boxes on a dashcam frame, point-cloud cuboids from a LiDAR rig, or entity spans in a legal document. The schema changes. The pipeline doesn’t.
What the record looks like
Inter-annotator agreement averaged across production accounts. Reported weekly, never retrofitted.
Passed through the review pipeline for our longest-running annotation client. Multiple pipeline generations, one discipline.
Across computer vision, NLP, and multimodal pipelines. Repeat engagements, not one-off audits.
Calibration, primary pass, independent 2nd pass, lead arbitration, exception & drift — the same five, every batch.
Scope with us
For new annotation work, we scope a calibrated team and a written accuracy threshold together. For existing datasets, we run the review pipeline over a sample and return a findings report before you commit to a full audit.