Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation
arxiv(2024)
摘要
Dataset distillation has emerged as a promising approach in deep learning,
enabling efficient training with small synthetic datasets derived from larger
real ones. Particularly, distribution matching-based distillation methods
attract attention thanks to its effectiveness and low computational cost.
However, these methods face two primary limitations: the dispersed feature
distribution within the same class in synthetic datasets, reducing class
discrimination, and an exclusive focus on mean feature consistency, lacking
precision and comprehensiveness. To address these challenges, we introduce two
novel constraints: a class centralization constraint and a covariance matching
constraint. The class centralization constraint aims to enhance class
discrimination by more closely clustering samples within classes. The
covariance matching constraint seeks to achieve more accurate feature
distribution matching between real and synthetic datasets through local feature
covariance matrices, particularly beneficial when sample sizes are much smaller
than the number of features. Experiments demonstrate notable improvements with
these constraints, yielding performance boosts of up to 6.6
on SVHN, 2.5
state-of-the-art relevant methods. In addition, our method maintains robust
performance in cross-architecture settings, with a maximum performance drop of
1.7
https://github.com/VincenDen/IID.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要