Searching the Literature by Meaning: Benchmarking Free and Commercial Word Embedding Models on Social Work Text

Abstract

<p><b><i><span>Purpose</span></i></b><span>: Locating relevant studies is the first step of evidence-based practice, yet most searching relies on keyword matching. Artificial intelligence (AI) tools called embedding models search by meaning, but the best-known options are paid commercial services. The study asked which free embedding models best search the social work literature, whether they match the commercial standard, and whether rerankers are needed.</span></p> <p><b><i><span>Materials and Methods</span></i></b><span>: Twelve free embedding models and two commercial OpenAI models were tested on 64,956 social work records (1989-2025) using 150 curated queries. Two frontier AI judges, from families unrelated to every tool evaluated, made 50,328 blind head-to-head comparisons (nDCG@10). A judge-free known-item test (496 queries) and a blind 120-pair expert human instrument provided validation.</span></p> <p><b><i><span>Results</span></i></b><span>: Free tools matched or beat the commercial standard. Two free models outperformed the paid flagship; a free 300-million-parameter model beat the paid default, essentially tied for first at finding specific papers (91.9%), and with a reranker was the best configuration overall (.846). Keyword search trailed every embedding model (.604 vs. .680–.842). Rerankers rescued weak models but added nothing to the strongest. Judges agreed on 85.5% of comparisons, and committee-to-rater agreement (69-76%) matched or exceeded rater-to-rater agreement (69-71%).</span></p> <p><span> <b><i>Discussion</i></b>: Score differences among leading models are too small to change what a searcher sees; tool choice should rest on size, speed, cost, and privacy.</span></p> <p><b><i><span>Conclusion</span></i></b><span>: High-quality, meaning-based search of the social work literature is achievable with free tools on an ordinary computer: no subscription, no queries sent to an outside company.</span></p>