Benchmarking Paper Recommendation Systems: What Does It Mean to Prove That They Work?

Abstract

This seminar paper was prepared for “Bachelor/Master Seminar – Current/Hot Topics in Applied AI, Business Process Management, and Blockchain” at the Technical University of Munich during Summer Semester 2026. It reviews how paper recommendation, citation recommendation, and literature discovery systems are evaluated. Drawing on 39 studies from 2020–2026, it examines evaluation choices such as candidate pools, relevance labels, baselines, metrics, and evidence from users or deployments. It proposes a checklist for matching evaluation evidence to claims about a system’s practical usefulness.