Retrieval vs ranking
You ship a much better ranking model. Offline NDCG jumps. Online, for many users, the items they would have loved still never appear. Why?
Large recommenders don't score every item for every request. They retrieve a few hundred candidates cheaply from millions, then rank only those with an expensive model. Each stage has its own job, its own model, its own metric and its own failure modes. The ranker can only reorder what retrieval handed it.
The ranker can't find what it never sees
Six. Retrieval's recall@K (here 6/10 at K = 500) is a hard ceiling on everything downstream. A ranking improvement measured offline on logged candidates can look great while the missing four never had a chance. Evaluate retrieval by recall, on held-out interactions, against the whole corpus.
Why retrieval models look the way they do
Retrieval must search millions of items in a few milliseconds. The standard answer: one network embeds the user (or query), another embeds each item, and the score is their dot product:
Item embeddings are computed offline and loaded into an approximate nearest-neighbor index (IVF, HNSW, or product quantization, as in lesson 6's follow-ups). At request time you compute one user embedding and ask the index for the top K dot products.
A user × item cross feature has to be computed per pair, and that's exactly what the index avoids. So retrieval gives up interaction features, and the ranker, which scores only a few hundred pairs, uses them freely (feature crosses, attention over the user's history for each candidate). That split is the reason for two stages. Late-interaction models (e.g. ColBERT-style multi-vector scoring) are a middle ground.
Budget the funnel
A corpus of 5,000 items. Each user has 10 relevant ones. You have 40 ms per request. Choose a retriever (bigger embeddings find relevant items more reliably, but the index search costs more), how many candidates to pass on, and a ranker (the big one is more accurate but costs 5× more per candidate). Goal: at least 6 relevant items in the top 10, averaged over 120 simulated users.
Negatives you didn't choose
The ideal retrieval loss is a softmax over the whole corpus: make the clicked item's score beat every other item's. That's millions of scores per example. The standard shortcut is in-batch negatives: in a batch of \(B\) (user, clicked item) pairs, each user's positive is scored against the other \(B-1\) users' items as negatives.
In-batch sampling draws negatives from the click distribution \(Q(i)\), not uniformly, so popular items are over-penalized relative to the full softmax. The logQ correction subtracts \(\log Q(i_c)\) from each logit before the softmax, which undoes that sampling bias (Yi et al., 2019). The other hazard is false negatives: another user's clicked item may be one this user would like too. It's labelled negative only because they were in the same batch.
Two more habits are worth stating in an interview. Mix in random negatives from the whole corpus, since in-batch negatives are all popular items. And treat missing interactions as unobserved, not disliked: an item the user never saw carries no label.
Where in the list matters
Toggle which of the 10 shown positions hold a relevant item. This user has 3 relevant items in total.
\(\mathrm{DCG@10} = \sum_{i=1}^{10} \frac{\mathrm{rel}_i}{\log_2(i+1)}\); NDCG divides by the best possible DCG with 3 relevant items. MRR = 1 / (rank of the first relevant item).
The discount \(1/\log_2(i+1)\) falls fast at the top: 1.00 → 0.63 for ranks 1 → 2, but 0.30 → 0.29 for ranks 9 → 10. NDCG and MRR are top-heavy, like user attention. Recall@10 ignores order entirely, which suits retrieval (did it make the list?) but not ranking.
Apply it somewhere else
Position bias. Fixes: include position as a training feature and set it to a constant at serving time, so the model explains position separately from relevance. Or weight examples by inverse examination propensity, estimated from randomized swaps. Or collect unbiased data with small randomization or interleaving.
ID-only embeddings can't represent an item with no history. Content features let the tower place it near similar items right away. Exploration (a small share of slots, bandit-style) gathers the interactions that refine it. Also make sure the index refresh cadence actually includes new items.
Evaluate each stage on the distribution it will actually see. The ranker should be evaluated on candidates the retriever produces, which are hard negatives. Retrieval should be evaluated by recall against the full corpus.
Write the metric and the loss
Plain editor, numpy, hidden tests. ⌘/Ctrl+Enter runs.
Exercise 1: NDCG@k
Exercise 2: in-batch softmax loss with logQ correction
Answer out loud, then check
- Several retrievers in parallel: a two-tower ANN over embeddings, co-engagement ("users who engaged with X also engaged with Y"), the social or follow graph, fresh and trending items. Union and dedupe, with per-source quotas.
- Budgets per stage: ANN top-K per source, a lightweight pre-ranker to cut thousands to hundreds, then the heavy ranker.
- Measure each source's recall and incremental recall on held-out engagements. Monitor index freshness and embedding drift, and keep a fallback when the index is stale.
- Pointwise (log loss per item): simple and calibrated, needed when probabilities feed auctions or blending, but it doesn't optimize order directly.
- Pairwise (BPR, RankNet): learns that the clicked item outranks a non-clicked one. It targets order and is robust to label scale.
- Listwise (softmax over the list, LambdaRank): LambdaRank weights each pair's gradient by the |ΔNDCG| of swapping them, so it focuses on errors near the top. Many production systems combine a calibrated pointwise head with a ranking objective.
- In-batch negatives follow item popularity \(Q(i)\), so the sampled softmax is biased against popular items.
- logQ correction: subtract \(\log Q(i)\) from each logit, estimating \(Q\) by streaming frequency counts.
- Complement it with mixed negatives (uniform random from the corpus) and watch for false negatives: dedupe the same item within a batch and consider hard-negative mining with care.
- The model only gets labels for what it showed, so its training data drifts toward its own preferences. Popular items get more exposure, then more clicks (rich-get-richer), and the catalog coverage narrows.
- Mitigations: exploration traffic, propensity-aware training, diversity and freshness constraints in the final blend, and holdouts that measure long-term engagement rather than immediate clicks.
Sources: Covington, Adams & Sargin, Deep Neural Networks for YouTube Recommendations (RecSys 2016); Yi et al., Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations (RecSys 2019); Järvelin & Kekäläinen, Cumulated gain-based evaluation of IR techniques (2002); Burges, From RankNet to LambdaRank to LambdaMART (2010); Joachims et al., Unbiased Learning-to-Rank with Biased Feedback (WSDM 2017).