Gradient-Aligned convolution neural network

•We propose a general Convolution operation, called GAConv, which can replace conventional operations in CNN to help it achieve rotation invariance.•With GAConv, Gradient-Aligned CNN (GACNN) can achieve rotation invariance without any data augmentation, feature-map augmentation, and filter enrichment.•In GACNN, rotation invariance does not learn from the training set, but bases on the network model. Different from the vanilla CNN, GACNN will output invariant results for all rotated versions of an object, no matter whether the network is trained or not.•We conduct classification experiments on designed dataset and realistic datasets. The results show that with the same computation cost, GACNN achieved better results than conventional CNN and some rotational invariant CNN.