In-Context Data Distillation with TabPFN
CoRR(2024)
摘要
Foundation models have revolutionized tasks in computer vision and natural
language processing. However, in the realm of tabular data, tree-based models
like XGBoost continue to dominate. TabPFN, a transformer model tailored for
tabular data, mirrors recent foundation models in its exceptional in-context
learning capability, being competitive with XGBoost's performance without the
need for task-specific training or hyperparameter tuning. Despite its promise,
TabPFN's applicability is hindered by its data size constraint, limiting its
use in real-world scenarios. To address this, we present in-context data
distillation (ICD), a novel methodology that effectively eliminates these
constraints by optimizing TabPFN's context. ICD efficiently enables TabPFN to
handle significantly larger datasets with a fixed memory budget, improving
TabPFN's quadratic memory complexity but at the cost of a linear number of
tuning steps. Notably, TabPFN, enhanced with ICD, demonstrates very strong
performance against established tree-based models and modern deep learning
methods on 48 large tabular datasets from OpenML.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要