Coexistence of continuous attractors with different dimensions for neural networks.

This work briefly investigates the coexistence of continuous attractors in the neural networks with Rectified Linear Unit (RELU) transfer function. Memory is stored as a manifold of stable states, or a continuous attractor. Continuous attractors are some low-dimensional manifolds embedded in a high-dimensional state space. One neural network may possess more than one continuous attractors. More importantly, we found that these multiple continuous attractors may have different dimensional, some are 2-D plane attractors and others are 1-D line attractors. It is also an enlightenment to study the continuous attractors of high dimensional models.