Acing your ML interview #1

Unified virtual memory for large scale ML workloads Context: Recsys model training & inference are memory intensive. Generally, the servers for training & inference have the following hierarchy of memory: GPU RAM (High bandwidth, $$$) ~ 40–80 GB depending on GPU choice. CPU RAM (Low bandwidth, $$) ~ 1 TB & SSD memory (Low bandwidth, $) ~ multiple TBs…

Context: — Large enterprises have thousands of models in production serving important business needs. With such large scale ML deployment, continuous testing is a must have for robust model delivery. Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the…

Context : — RAMs in GPU are still an order magnitude lower than CPU counterpart. This makes GPU RAMs an expensive resource in training & inference. In this post we look at optimizing batch size as a way to maximize GPU RAM usage. Optimal batch sizes: GPU RAMs store the following during training & inference: Parameters…

Industrial recommendation models (RecSys) are different from domains like NLP and CV in a few ways: Massive amounts of user engagement data are readily available for training RecSys models. (e.g., ads clicks, search result page clicks, content browse). Most of the engagement data is sparse and categorical. …

Models as artifacts Typically, once a model is trained, a stand-alone artifact is created that can run the model independently from its training harness. …

Recsys models: online training & serving 1. Offline batch training vs online realtime training: In offline batch training model, the model is trained on historical datasets. A training worker reads one mini-batch of training examples from the storage at a time, computes a forward and a backward pass, and finally updates parameters for the model in training. Instead of reading mini-batch examples from the…

Recsys model serving: Embeddings Context: Production recommendation models have a large number of sparse features of high cardinality. For example, a video recommendation model will have a sparse feature for watched video ids. The cardinality of such a feature is O(billions). High cardinality resulting results in large embedding tables. This poses a challenge for…

We explore the architectural patterns used in recsys models and their implications on inference: Model Architectures Serving Paradigms Common architectural patterns ✔️ Model Architectures: Recsys models follow certain architectural patterns which helps in both training & inference. Here, we focus on inference only. Recsys models are built from two types…

Typical social network content has a subset of text, videos, audio, image, entities, etc. Understanding such content has led to a surge of interest in multimodal systems. Multimodal models have shown solid performance in multi-task setups and downstream tasks across different domains (content understanding, content generation, question answering etc), paving…