How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model
Physical Review X(2023)
Abstract
Deep learning algorithms demonstrate a surprising ability to learn
high-dimensional tasks from limited examples. This is commonly attributed to
the depth of neural networks, enabling them to build a hierarchy of abstract,
low-dimensional data representations. However, how many training examples are
required to learn such representations remains unknown. To quantitatively study
this question, we introduce the Random Hierarchy Model: a family of synthetic
tasks inspired by the hierarchical structure of language and images. The model
is a classification task where each class corresponds to a group of high-level
features, chosen among several equivalent groups associated with the same
class. In turn, each feature corresponds to a group of sub-features chosen
among several equivalent ones and so on, following a hierarchy of composition
rules. We find that deep networks learn the task by developing internal
representations invariant to exchanging equivalent groups. Moreover, the number
of data required corresponds to the point where correlations between low-level
features and classes become detectable. Overall, our results indicate how deep
networks overcome the curse of dimensionality by building invariant
representations, and provide an estimate of the number of data required to
learn a hierarchical task.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined