Review: Perspectives from psychology could help quantify LLMs’ thinking capacity

A new Review discusses how concepts and methods borrowed from psychology can inform research to evaluate whether and how large language models (LLMs) can employ reasoning. “The ever-growing capabilities of these systems make them challenging but also increasingly important to understand and explain, especially since these expanding capabilities also bring greater potential for unforeseen harm,” Thilo Hagendorff and colleagues write. Their proposed approach – called “machine psychology” – advocates for seeking inspiration from the behavioral sciences. Since psychology’s protocols already assess emotional, cognitive, and social development, they could be modified to examine LLMs’ intelligence, biases, decision-making, social skills, and language. Hagendorff et al. urge caution, though: “Many psychological concepts are normatively laden and can foster mismatches in expectations between AI experts and the public regarding machine capability.”