# Overview A system which present a ranked list of items that we believe a user will engage with. The system's value is tied to an effective ordering and diversity in the results. The business objective of these systems is based on a user's decision and how that decision translates into revenue and exposure cost (i.e., surfacing the wrong item). # Key Considerations ## Challenges - **Evaluation horizon** - it is difficult to measure how short term actions (clicks) translate into long-term decisions (renewing a subscription). Techniques like [[counterfactual evaluation with important weighting]] help, but variance is still a factor. - Feedback loops ## Evaluation Using your selected metric(s), you can do offline evaluation through a [[lead-one-interaction-out test]], so each user appears in a train and test, but with different timestamps. For online evaluation, deploy in a shadow-rank mode. The model will reorder items, but the server order still comes from the baseline. Compare click-through on items that would change position. Once safe, graduate to an A/B buck. Measure at least one short-term metric (CTR) and one long-term metric (retention). If they diverge, you are in a trap. Alternatively, you can use [[paired difference tests]] over the A/B testing. ### Product Metrics - Session watch / purchase rate per user - [[Average Revenue Per User (ARPU)]] - [[Gross Merchandise Value (GMV)]] per user - Retention or churn-deferral rate over n-day windows - Inventory utilization - [[Dwell time]] ### ML Metrics - [[Mean Reciprocal Rank (MRR)]] - [[Normalized Discounted Cumulative Gain (NDCG)]] - [[Hit@K]] or [[Recall@K]] - Coverage - proportion of catalog shown over a time window—critical for cold-start sellers and long-tail content - Calibration / Expected Rating Error - does the score distribution match observed engagement probabilities? # Use Cases - [[Product Recommendations System]] - [[Video Recommendations System]] - [[Friend Suggestion System]] # Related Topics ## Reference #### Working Notes #### Sources - Excellent blog on a lot of recommendation system topics: [All Posts - Sumit's Diary](https://blog.reachsumit.com/posts/)