Executives increasingly turn to generative AI to help evaluate strategic decisions, from ranking business models to weighing competing options — but new research published in Strategic Management Journal cautions that no single AI-generated evaluation should be trusted at face value. The study finds that individual generative AI evaluations of strategic decisions are inconsistent and often biased, simply by the order in which two options are presented — but aggregating many evaluations across different prompts, roles, and models produces judgments that closely resemble those of human experts.
The research team of Anil R. Doshi of University College London, J. Jason Bell of the University of Oxford, and Emil Mirzayev and Bart Vanneste, both of University College London, tested large language models by having them rank 60 business models through pairwise comparisons, repeated across multiple models, ten assumed roles — including journalist, strategy professor, and industry expert — and two prompting strategies. No human expert panel evaluates the same option set that many ways, giving the researchers a rare way to measure where AI’s judgment holds up, and where it doesn’t.
