Search⌘ K
AI Features
Home
Courses
Grokking the Machine Learning System Design Interview

Retrieval and Ranking Pipelines

Explore how multi-stage retrieval and ranking pipelines efficiently narrow billions of candidates in under 200 ms for large-scale ML systems. Learn about candidate generation using two-tower ANN models, indexing strategies like HNSW and IVF, and ranking stages to balance recall, relevance, and latency constraints. Gain skills to design scalable recommendation systems that meet strict latency and quality requirements.

We'll cover the following...

When a user opens YouTube or scrolls through Instagram, the platform must select a handful of items from a corpus of hundreds of millions, sometimes billions, of candidates. The entire process completes in under 200 milliseconds. This is not a single model call. It is a carefully staged funnel where each layer narrows the candidate set while increasing model sophistication. With serving paradigms such as synchronous inference and batch precomputation covered in the previous lesson, this lesson explains how those serving patterns fit together in the multi-stage funnel used by search and recommendation systems in large-scale production systems.

Consider this interview prompt: “Design a video recommendation system that serves 1 billion users across 500 million videos with a 200 ms p99 latency SLA.” A single model cannot realistically score 500 million items per request within that latency budget. Instead, the system breaks the problem into three stages: candidate generation, coarse ranking, and fine-grained reranking. Each stage has its own candidate volume, latency budget, model architecture, and failure modes. You should explain what each stage does, why it exists, and what breaks if you remove it.

The following diagram illustrates how the funnel progressively narrows the candidate set from billions to the final ranked slate:

Three-stage ranking funnel trading breadth for depth to optimize model capacity usage
Three-stage ranking funnel trading breadth for depth to optimize model capacity usage

Each stage in this funnel operates under strict constraints, and the design decisions at one stage ripple through every downstream component. The next sections unpack each stage in detail, starting with candidate generation.

Candidate generation with two-tower ANN

The first stage must reduce billions of items to roughly 500–2,000 candidates in under 10–20 milliseconds. The dominant approach uses a two-tower architectureA neural network design where separate encoder networks (towers) independently produce embeddings for queries and items, enabling offline pre-computation of item embeddings and fast online retrieval via approximate nearest neighbor search.. ...