The Difficulty of Training Deep Architectures and the Effect of Unsupervised Pre-Training

Whereas theoretical work suggests that deep ar- chitectures might be more e cient at represent- ing highly-varying functions, training deep ar- chitectures was unsuccessful until the recent ad- vent of algorithms based on unsupervised pre- training. Even though these new algorithms have enabled training deep models, many questions remain as to the nature of this di cult learning problem. ...