Comparing Template-based and Template-free Language Model Probing
CoRR(2024)
摘要
The differences between cloze-task language model (LM) probing with 1)
expert-made templates and 2) naturally-occurring text have often been
overlooked. Here, we evaluate 16 different LMs on 10 probing English datasets
– 4 template-based and 6 template-free – in general and biomedical domains to
answer the following research questions: (RQ1) Do model rankings differ between
the two approaches? (RQ2) Do models' absolute scores differ between the two
approaches? (RQ3) Do the answers to RQ1 and RQ2 differ between general and
domain-specific models? Our findings are: 1) Template-free and template-based
approaches often rank models differently, except for the top domain-specific
models. 2) Scores decrease by up to 42
template-free and template-based prompts. 3) Perplexity is negatively
correlated with accuracy in the template-free approach, but,
counter-intuitively, they are positively correlated for template-based probing.
4) Models tend to predict the same answers frequently across prompts for
template-based probing, which is less common when employing template-free
techniques.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要