Pairwise ranking outperforms single-action RL for offline explanation selection: A practical lesson

We report a practical lesson from building a GPU-free explainable-recommendation serving stack: explanations are pre-generated offline into a per-item candidate pool, and a small CPU-resident model selects one at request time. Every candidate in the pool of size K carries an offline BERTScore-F1 label against a reference explanation, so we compare a pairwise learning-to-rank model (LightGBM LambdaRank) against a single-action RL formulation (DPO) and a distilled selector on the XRec Google Local benchmark (2,958 pairs, 5 seeds). LambdaRank outperforms both by 0.019–0.025 F1, a gap more than fifteen times the across-seed standard deviation. The advantage appears structural rather than algorithmic: LambdaRank's objective consumes the label of every candidate in the pool, whereas DPO's rollout only observes the label of the sampled side of each preference pair. A separate candidate-source design, selecting from knowledge-graph-grounded paths instead of the cached pool, trades this reference alignment for a Unique-Sentence Ratio (USR) of 1.000. We also encountered two failure modes: further RL fine-tuning on top of an already-distilled policy regresses F1, and an end-to-end RL fine-tune of the generator itself reward-hacks the metric within a few hundred steps. The practical takeaway is that whenever an offline metric can label every candidate in a fixed action set, a pairwise learning-to-rank baseline is worth evaluating before reaching for single-action RL. One caveat: LambdaRank's training signal differs in form from the evaluation metric we report, since it trains on quintile-binned labels while DPO trains on a blended reward, though both derive from the same underlying BERTScore-F1; a fully decorrelated evaluation remains future work.