Soumya Roy

Membership or Understanding? What the Imitation Game Measures When the Respondent Is a Language Model

Benchmarks measure whether a language model produces the right output. That is areasonable test of factual knowledge, but expertise is not mainly a stock of facts. Itis the ability to move fluently inside a working community. This review sets out analternative drawn from the soci...