Data · 5 MIN READ
The recommendation bug you cannot see
Offline metrics look fine. Online behaviour does not match. The usual culprit is that your model sees a different world in production than it learned from.
AB19 Innovations
In short
Training/serving skew occurs when features are computed differently in training and inference. Fix it with a single shared feature definition consumed by both paths, and detect it by logging served feature vectors.
Training/serving skew is the most common serious bug in deployed machine learning and among the hardest to notice, because nothing errors. The pipeline runs. The model responds. The numbers are simply worse than they should be, and nobody can say why.
How it happens
The training job computes features with a SQL query over the warehouse. The serving path recomputes them in application code against the production database. Two implementations of the same idea, written months apart by different people, diverging on exactly the details nobody documented.
- Time windows: "last 30 days" measured from the event, or from now?
- Nulls: the training query drops them; the serving path substitutes zero.
- Categoricals: an unseen category maps to a different bucket in each path.
- Ordering: the warehouse deduplicates by latest; production takes the first row.
One definition, both paths
The durable fix is to define each feature once, as code, and have both the training job and the serving path call it. This is what a feature store gives you, but you do not need a feature store to get the benefit — a shared library and the discipline to use it covers most of the distance.
Detect it even after you have fixed it
Log the actual feature vector served with each prediction. Compare the distribution of served features against the training distribution on a schedule. When they drift apart, you find out from a monitor rather than from a quarterly review of declining performance.
It is unglamorous work. It is also the difference between a model that performs in production the way it did in evaluation and one that quietly under-delivers for a year.