Recommendation Systems skill for Claude Code

A recommender is worth its complexity when it beats a popularity baseline in an online A/B test on the business metric (clicks that convert, watch time, retention), not just on offline recall. Most teams should ship a strong baseline first, log impressions properly, and only then add models.

Ground rules

• Log impressions, not just clicks. Without knowing what was shown, you cannot compute click-through, correct for position, or train a ranker.
• Split offline data by time. Random splits leak the future and inflate every metric.
• Always compare against popularity and “recently viewed / bought together” baselines.
• Define the objective in business terms before choosing a model: purchase, long-term engagement, retention, and what must never happen (recommending out-of-stock items, unsafe content, competitors’ products).
• Recommenders shape the data they learn from. Reserve some exploration traffic or the system converges on what it already shows.

Architecture: two stages

all items (10^5-10^8)
  -> candidate generation: several cheap sources, ~100-1000 candidates
     (co-visitation, MF/ALS neighbors, two-tower ANN, popular-in-segment, new items)
  -> ranking: feature-rich model scores candidates (GBDT or neural)
  -> re-ranking: business rules, diversity, freshness, dedupe, filters
  -> top N shown, impressions logged

Small catalogs (under ~10k items) can skip candidate generation and rank everything.

Choose the approach

Situation	Start with
No model yet, any data volume	Popularity (global and per segment), recently viewed, trending
“Customers also bought/viewed”	Item-item co-occurrence (co-visitation counts, normalized)
Implicit feedback, users x items matrix	ALS / BPR matrix factorization (implicit library)
Rich user and item features, large catalog	Two-tower embedding model + ANN index for candidates
Many signals to combine (recency, price, CTR, embeddings)	Learning-to-rank (LightGBM lambdarank) on top of candidates
Text-heavy items, cold start	Content embeddings (text or image) for item-item similarity
Session-based, anonymous users	Session co-occurrence or sequence models over recent events

Workflow

- [ ] 1. Define objective, surfaces, and guardrails
- [ ] 2. Fix logging (impressions, positions, context)
- [ ] 3. Build the time-based offline evaluation
- [ ] 4. Ship baselines; measure online
- [ ] 5. Add candidate generators; measure recall of candidates
- [ ] 6. Train a ranker; evaluate offline then online
- [ ] 7. Handle cold start and re-ranking rules
- [ ] 8. A/B test; monitor feedback loops and coverage

2. Logging

Each impression row: request ID, user (or session) ID, item ID, surface, position, timestamp, model version, candidate source, and score. Each interaction row joins back to the request ID (click, add to cart, purchase, dwell time, skip). Without the request ID join, attribution and ranker training labels are guesswork.

3. Offline evaluation

• Train on events before time T, evaluate on events in (T, T + window]. Roll T forward for several folds.
• For each user in the evaluation window, rank items and compare to what they actually interacted with.

Metric	Use for
Recall@k / HitRate@k	Candidate generation: did the relevant item make the cut?
nDCG@k, MAP@k	Ranking quality, position-aware
Catalog coverage	Share of items ever recommended; low = popularity collapse
Novelty / popularity bias	Average popularity rank of recommended items
Diversity	Intra-list dissimilarity (categories or embeddings)

Evaluate per segment: new vs returning users, heavy vs light users, item categories. Exclude items the user already interacted with before T when that is how production behaves.

4-5. Baselines and candidates

Co-visitation in SQL:

-- pairs of items viewed in the same session, last 30 days
SELECT a.item_id AS item, b.item_id AS related, COUNT(*) AS together
FROM events a JOIN events b
  ON a.session_id = b.session_id AND a.item_id <> b.item_id
WHERE a.ts > now() - interval '30 days' AND a.event = 'view' AND b.event = 'view'
GROUP BY 1, 2
HAVING COUNT(*) >= 3;
-- normalize by item popularity (e.g. together / sqrt(cnt_a * cnt_b)) before ranking

Matrix factorization with implicit:

import scipy.sparse as sp
from implicit.als import AlternatingLeastSquares

user_items = sp.csr_matrix((weights, (user_idx, item_idx)))   # weight = confidence, e.g. 1 + log(count)
model = AlternatingLeastSquares(factors=64, regularization=0.05, iterations=20)
model.fit(user_items)
ids, scores = model.recommend(u, user_items[u], N=200, filter_already_liked_items=True)
similar_ids, _ = model.similar_items(item_id, N=50)

Measure each candidate source’s recall@k (k = candidates kept) separately and as a union. The ranker cannot recover items that no source retrieved.

Two-tower models: train user and item encoders on positive interactions with in-batch negatives, index item embeddings in an ANN store (see vector-search), and query with the user embedding at request time. Use them once co-visitation and MF plateau.

6. Ranking

import lightgbm as lgb
ranker = lgb.LGBMRanker(objective="lambdarank", n_estimators=500, learning_rate=0.05)
ranker.fit(X_train, y_train, group=group_sizes_train,          # rows grouped per request
           eval_set=[(X_val, y_val)], eval_group=[group_sizes_val], eval_at=[10])

• Training rows: candidates from logged requests, label = graded interaction (purchase 3, add-to-cart 2, click 1, impression only 0).
• Features: user history aggregates, item stats (CTR, recency, price), user-item features (similarity scores from candidate sources, category affinity), context (time, device).
• Compute features with the same code online and offline; point-in-time correctness (no features computed after the event) prevents leakage.
• Position bias: clicks favor top slots. Add position as a training feature set to a constant at inference, or use randomized exploration data for unbiased labels.

7. Cold start and re-ranking

Case	Strategy
New user	Popular-in-context (geo, referrer, device), onboarding preferences, session signals within the first clicks
New item	Content-embedding neighbors, boosted exploration slots, editorial placement
Sparse category	Back off to parent category popularity

Re-ranking rules after the model: remove unavailable or already purchased items, cap items per brand or category, enforce freshness slots, apply safety and policy filters. Keep rules in config and log which rules fired.

8. Online testing and monitoring

• A/B test against the current system with the business metric as primary and guardrails (latency, revenue per session, complaint rate). See experiment-analysis.
• Offline gains often shrink online; trust the A/B test.
• Monitor: coverage, share of impressions going to the top 1% of items, cold-start item impressions, latency p95, and candidate source contribution over time.
• Keep 1-5% exploration traffic (randomized or epsilon-greedy slots) for unbiased data.

Gotchas

• Evaluating on interactions the current recommender caused rewards copying the current system. Exploration data or A/B tests break the loop.
• Popularity dominates implicit-feedback models; check coverage and novelty, not only recall.
• Bots and internal traffic pollute interaction logs; filter before training.
• Recommending what the user just bought (or the item they are viewing) is the most visible failure; filter it explicitly.
• Latency budget: candidate retrieval + feature fetch + ranking must fit the page budget (often under 100 ms server-side). Precompute where possible.
• Sensitive inferences (health, finances) from browsing data need product and legal review.

Output contract

**Objective:** <business metric>, guardrails <list>
**Logging:** impressions with request ID <yes>, positions <yes>
**Candidates:** <sources>, recall@<k> <x> (union)
**Ranker:** <model>, features <groups>, nDCG@10 <x> vs baseline <y>
**Coverage/novelty:** <x>% catalog, top-1% share <y>%
**Online:** A/B <result with CI>, exploration <x>% traffic