How does promoting the minority fraction affect generalization? A theoretical study of the one-hidden-layer neural network on group imbalance
CoRR(2024)
Abstract
Group imbalance has been a known problem in empirical risk minimization
(ERM), where the achieved high average accuracy is accompanied by low accuracy
in a minority group. Despite algorithmic efforts to improve the minority group
accuracy, a theoretical generalization analysis of ERM on individual groups
remains elusive. By formulating the group imbalance problem with the Gaussian
Mixture Model, this paper quantifies the impact of individual groups on the
sample complexity, the convergence rate, and the average and group-level
testing performance. Although our theoretical framework is centered on binary
classification using a one-hidden-layer neural network, to the best of our
knowledge, we provide the first theoretical analysis of the group-level
generalization of ERM in addition to the commonly studied average
generalization performance. Sample insights of our theoretical results include
that when all group-level co-variance is in the medium regime and all mean are
close to zero, the learning performance is most desirable in the sense of a
small sample complexity, a fast training rate, and a high average and
group-level testing accuracy. Moreover, we show that increasing the fraction of
the minority group in the training data does not necessarily improve the
generalization performance of the minority group. Our theoretical results are
validated on both synthetic and empirical datasets, such as CelebA and CIFAR-10
in image classification.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined