Video Recommendation: Data Strategy and Feature Engineering
Explore the critical data strategy and feature engineering components in designing a large-scale video recommendation system. Understand how implicit user behavior signals like watch time, replays, skips, and shares are combined into robust training targets. Learn to engineer features across user, video, and contextual dimensions while addressing freshness and scale. Discover how content embeddings solve the cold start problem for new videos and delve into the operational challenges of embedding billions of videos, including index rebuilding strategies. This lesson equips you with practical insights needed for ML system design interviews focusing on recommendation systems.
In a YouTube-scale recommendation system, fewer than 1% of users ever click a thumbs-up or thumbs-down button. The system serves billions of recommendation requests daily, yet it operates in a near-vacuum of explicit preference data. This forces a fundamental design choice: the model must learn what users want not from what they say, but from what they do. With business metrics mapped to ML objectives and scale constraints defined in the previous lesson, the next critical decision is what data to collect and how to transform raw behavioral traces into features the model can actually learn from.
This lesson covers three pillars that interviewers consistently probe. First, how implicit behavioral signals serve as satisfaction proxies and the noise they introduce. Second, how features are engineered across user, video, and contextual dimensions with different freshness requirements. Third, how the cold start problem for new videos is solved using content embeddings, and the operational reality of embedding billions of videos, including the painful index rebuild challenge when the embedding model changes.
Interviewers expect you to reason about why certain signals are chosen, what biases they carry, and how feature pipelines operate under latency constraints at a billion-item scale. Candidates who treat data strategy as an afterthought consistently underperform.
Implicit signals as satisfaction proxies
Explicit feedback, such as star ratings and thumbs-up actions, has two main limitations. It is sparse because most users do not provide explicit ratings, and it is skewed toward users with strong positive or negative reactions. As a result, it captures the extremes more often than neutral or mild preferences and underrepresents moderate preferences. This makes explicit feedback insufficient as the only training signal for a recommendation model that must make reliable recommendations across users and request contexts.
The system instead relies on four core implicit signals, each capturing a different facet of user satisfaction:
Watch time (normalized by duration): This is the strongest general engagement signal. However, raw watch time biases the model toward recommending longer videos. Normalizing by video duration fixes this. A user watching 95% of a 30-second clip signals stronger satisfaction than watching 10% of a 2-hour movie. The normalized ratio, sometimes called completion rate, becomes the actual feature.
Replays: ...