InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Context
arxiv(2024)
Abstract
Large language models (LLMs) have demonstrated the potential to mimic human
social intelligence. However, most studies focus on simplistic and static
self-report or performance-based tests, which limits the depth and validity of
the analysis. In this paper, we developed a novel framework, InterIntent, to
assess LLMs' social intelligence by mapping their ability to understand and
manage intentions in a game setting. We focus on four dimensions of social
intelligence: situational awareness, self-regulation, self-awareness, and
theory of mind. Each dimension is linked to a specific game task: intention
selection, intention following, intention summarization, and intention
guessing. Our findings indicate that while LLMs exhibit high proficiency in
selecting intentions, achieving an accuracy of 88%, their ability to infer the
intentions of others is significantly weaker, trailing human performance by
20%. Additionally, game performance correlates with intention understanding,
highlighting the importance of the four components towards success in this
game. These findings underline the crucial role of intention understanding in
evaluating LLMs' social intelligence and highlight the potential of using
social deduction games as a complex testbed to enhance LLM evaluation.
InterIntent contributes a structured approach to bridging the evaluation gap in
social intelligence within multiplayer games.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined