VideoLSTM convolves, attends and flows for action recognition

•To exploit both the spatial and temporal correlations in a video, we hardwire convolutions in the soft-Attention LSTM architecture.•We introduce motion-based attention which guides better the attention towards the relevant spatial-temporal locations of the actions.•We demonstrate how the attention generated from our VideoLSTM can be used for action localization by relying on the action class label only.•We show the theoretical as well as practical merits of our VideoLSTM against other LSTM architectures for action classification and localization.