Human-like object concept representations emerge naturally in multimodal large language models
arxiv(2024)
Abstract
The conceptualization and categorization of natural objects in the human mind
have long intrigued cognitive scientists and neuroscientists, offering crucial
insights into human perception and cognition. Recently, the rapid development
of Large Language Models (LLMs) has raised the attractive question of whether
these models can also develop human-like object representations through
exposure to vast amounts of linguistic and multimodal data. In this study, we
combined behavioral and neuroimaging analysis methods to uncover how the object
concept representations in LLMs correlate with those of humans. By collecting
large-scale datasets of 4.7 million triplet judgments from LLM and Multimodal
LLM (MLLM), we were able to derive low-dimensional embeddings that capture the
underlying similarity structure of 1,854 natural objects. The resulting
66-dimensional embeddings were found to be highly stable and predictive, and
exhibited semantic clustering akin to human mental representations.
Interestingly, the interpretability of the dimensions underlying these
embeddings suggests that LLM and MLLM have developed human-like conceptual
representations of natural objects. Further analysis demonstrated strong
alignment between the identified model embeddings and neural activity patterns
in many functionally defined brain ROIs (e.g., EBA, PPA, RSC and FFA). This
provides compelling evidence that the object representations in LLMs, while not
identical to those in the human, share fundamental commonalities that reflect
key schemas of human conceptual knowledge. This study advances our
understanding of machine intelligence and informs the development of more
human-like artificial cognitive systems.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined