Are Large Language Models Strategic Decision Makers? A Study of Performance and Bias in Two-Player Non-Zero-Sum Games
arxiv(2024)
Abstract
Large Language Models (LLMs) have been increasingly used in real-world
settings, yet their strategic abilities remain largely unexplored. Game theory
provides a good framework for assessing the decision-making abilities of LLMs
in interactions with other agents. Although prior studies have shown that LLMs
can solve these tasks with carefully curated prompts, they fail when the
problem setting or prompt changes. In this work we investigate LLMs' behaviour
in strategic games, Stag Hunt and Prisoner Dilemma, analyzing performance
variations under different settings and prompts. Our results show that the
tested state-of-the-art LLMs exhibit at least one of the following systematic
biases: (1) positional bias, (2) payoff bias, or (3) behavioural bias.
Subsequently, we observed that the LLMs' performance drops when the game
configuration is misaligned with the affecting biases. Performance is assessed
based on the selection of the correct action, one which agrees with the
prompted preferred behaviours of both players. Alignment refers to whether the
LLM's bias aligns with the correct action. For example, GPT-4o's average
performance drops by 34
"bigger and newer is better" does not hold for the above, where GPT-4o (the
current best-performing LLM) suffers the most substantial performance drop.
Lastly, we note that while chain-of-thought prompting does reduce the effect of
the biases on most models, it is far from solving the problem at the
fundamental level.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined