RecSys model inference

Industrial recommendation models (RecSys) are different from domains like NLP and CV in a few ways:

1. Massive amounts of user engagement data are readily available for training RecSys models. (e.g., ads clicks, search result page clicks, content browse). Most of the engagement data is sparse and categorical. Extensive training data leads to large model sizes and unique challenges in inference.
2. Model inference has to be low latency and high throughput as realtime recommendations are served to user at web-scale. Model compression to reduce model weight and improve serving performance has a tradeoff with lower revenue (due to loss in model accuracy). So, it is often not a choice.
3. User data is ever changing. To mitigate concept drift arising out of changing user behavior, model refresh should be near realtime.

image courtesy : Algoexpert

Here are a few deep-dives into various aspects of large scale model inference for recsys.

Topics:

1. Model architecture & serving paradigms
2. Serving embeddings
3. Online training & Serving
4. Model packaging
5. Using batching to optimize model execution.
6. Continuous testing for ML
7. Unified Virtual memory for large scale ML workloads

Let me know if you want any more topics to be covered.