Predictive Modeling with Echo State Networks

A lot of attention is now being focused on connectionist models known under the name "reservoir computing". The most prominent example of these approaches is a recurrent neural network architecture called an echo state network (ESN). ESNs were successfully applied in several time series modeling tasks and according to the authors they performed exceptionally well. Multiple enhancements to standard ESN were proposed in the literature. In this paper we follow the opposite direction by suggesting several simplifications to the original ESN architecture. ESN reservoir features contractive dynamics resulting from its' initialization with small weights. Sometimes it serves just as a simple memory of inputs and provides only negligible "extra-value" over much simple methods. We experimentally support this claim and we show that many tasks modeled by ESNs can be handled with much simple approaches.