AI Recommendation Engine

Recommendation quality was limited less by the model than by everything around it. Features were computed one way in a training notebook and another way in the serving path, so the model saw a different world in production than it learned from.

Cold start had no defined behaviour, which meant new users and new catalogue items — most of both, most of the time — received effectively arbitrary results.

We defined each feature exactly once and had both the training job and the serving path consume that definition. That single change removed an entire class of silent inconsistency.

Cold-start behaviour was made explicit: popularity and content-similarity fallbacks with a defined blend, rather than an implicit accident of whatever the model produced with no history.

Serving runs behind a typed FastAPI surface in containers, with a hard latency budget — if the personalised path exceeds it, the request falls back to a cached ranking rather than hanging.

Model artefacts are versioned alongside the feature definitions that produced them, so any deployment can be rolled back as a unit. New models go out against shadow traffic before they take real requests.

The system serves ranked, personalised results within a fixed latency budget, degrades to a sensible ranking instead of failing, and can be rolled back cleanly when a new model underperforms.

What we would tell another team

• Training/serving skew is the most common and least visible recommendation bug.
• Define cold-start behaviour deliberately — it is the majority case, not an edge case.
• Give the personalised path a latency budget and a fallback it can fail into.
• Version the model and its feature definitions together, or rollback is a fiction.