Model drift is when an AI model gets less accurate over time because the world has changed since it was trained. The model is not broken. It is out of date. It keeps answering with full confidence, using patterns that are no longer true, until someone checks it against fresh answers and retrains it.
Every AI model starts going out of date the day it goes live.
The week the world changed its shopping list
In early 2020, the top searches on Amazon were what you would expect: phone cases, phone chargers, Lego.
Within days, they were gone. By mid-April the top ten included toilet paper, face masks, hand sanitizer, paper towels and Clorox wipes, MIT Technology Review reported.
People changed their habits in a week. The AI systems behind stock forecasts and recommendations could not. One supplier's forecasting system, a consultant told the magazine, "was never trained on a spike like this, so the system was out of whack."
That is model drift at its most obvious. The world moved, and the model kept working for the old one.
What model drift means
An AI model learns from a snapshot of the world, then freezes. Drift happens in three steps:
- The model learns from a snapshot of past examples.
- The world moves on. Prices, slang, products, scams and habits keep changing.
- The answers go quietly wrong, because the model judges today's cases by yesterday's patterns.
Step three is the dangerous one. Nothing crashes and no alarm goes off. The model gives wrong answers in the same confident tone it uses for right ones.
Drift comes in two kinds. In data drift, the inputs change: customers start writing support tickets in Sheng instead of formal English. In concept drift, the right answer changes: fraudsters switch tactics, so yesterday's safe pattern is today's scam.
The flu tracker that slowly lost the plot
Google once built a model that estimated flu levels from what people searched for. The idea was clever. Sick people search for symptoms, and searches arrive days before official health data.
Then the world under it kept shifting. Google kept improving its own search engine, with 86 changes in June and July 2012 alone, and each one changed what people typed. Researchers who studied it in Science also found it had partly learned winter instead of flu. They called it "part flu detector, part winter detector."
Between August 2011 and September 2013, Google Flu Trends overestimated flu levels in 100 out of 108 weeks.
By February 2013, its estimate was more than double the doctor visits recorded by the US Centers for Disease Control and Prevention. No single change broke it. A long run of small ones did.
Almost every model ages
Google is not the exception. In 2022, researchers from Harvard Medical School and MIT tested four common types of model on 32 real datasets from healthcare, finance, transport and weather, then tracked them over time. In 91 percent of cases, the models got worse.
Drift is not a sign that something went wrong. It is what models do by default. The only question is whether you notice in week two or month ten.
How do you know your model is drifting?
Ask four questions. If the answer to any of them is yes, check your model against fresh human answers.
- Are people working around it? Agents overriding its suggestions, or analysts checking its numbers by hand. Users notice drift before dashboards do.
- Is it less sure of itself? Most models report a confidence score. A rising share of low scores means it is seeing cases it does not recognise.
- Has your world changed? New products, markets, languages, laws or customer habits all pull the data away from what the model learned.
- When did anyone last check it? If nobody can name the date, you do not know how accurate the model is today. You only know how accurate it used to be.
The fix is a loop. Pull a fresh sample of real cases, have trained people label the right answers, compare them with the model's answers, and retrain when the gap grows too wide. Software can watch the inputs and run the retraining. Only people can say what the right answer is in a world that just changed, which is why the human in the loop never really goes away. Those people also have to agree with each other, which is harder than it sounds, as we explain in Your agreement score cannot see half your errors.
It is also why we treat checking as a routine, not a one-off. At Impact Outsourcing, our review teams label fresh samples against a written rubric drafted with each client, and we measure how often reviewers agree every week. Our clients see drift as a number on a chart, not as a complaint from a customer.
The best AI teams do not expect a model to stay right. They plan for it to drift, and keep people in place to catch it.
Model drift FAQs
What is an example of model drift?
A shopping recommender trained before 2020 that kept suggesting phone cases while everyone was searching for face masks. Or a spam filter that lets through a new kind of scam email it has never seen.
What is the difference between data drift and concept drift?
Data drift means the inputs look different, such as new slang or new products. Concept drift means the right answer itself has changed, such as fraudsters switching tactics. Statistics can catch data drift. Concept drift needs fresh answers checked by people.
How often should a model be checked for drift?
It depends on how fast its world moves. A fraud or pricing model may need weekly checks. A model that reads medical scans may change more slowly, but a miss costs far more. Check more often than it would take a drifted model to do real damage.
Does retraining fix model drift?
Yes, if the model is retrained on fresh, correctly labelled data. Retraining on stale data, or on new data with poor labels, just teaches it the wrong lesson faster.