Are Large Language Models a Good Replacement of Taxonomies?
arxiv(2024)
Abstract
Large language models (LLMs) demonstrate an impressive ability to internalize
knowledge and answer natural language questions. Although previous studies
validate that LLMs perform well on general knowledge while presenting poor
performance on long-tail nuanced knowledge, the community is still doubtful
about whether the traditional knowledge graphs should be replaced by LLMs. In
this paper, we ask if the schema of knowledge graph (i.e., taxonomy) is made
obsolete by LLMs. Intuitively, LLMs should perform well on common taxonomies
and at taxonomy levels that are common to people. Unfortunately, there lacks a
comprehensive benchmark that evaluates the LLMs over a wide range of taxonomies
from common to specialized domains and at levels from root to leaf so that we
can draw a confident conclusion. To narrow the research gap, we constructed a
novel taxonomy hierarchical structure discovery benchmark named TaxoGlimpse to
evaluate the performance of LLMs over taxonomies. TaxoGlimpse covers ten
representative taxonomies from common to specialized domains with in-depth
experiments of different levels of entities in this taxonomy from root to leaf.
Our comprehensive experiments of eighteen state-of-the-art LLMs under three
prompting settings validate that LLMs can still not well capture the knowledge
of specialized taxonomies and leaf-level entities.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined