On the importance of severely testing deep learning models of cognition

•We argue that many unwarranted conclusions regarding deep neural network (DNN) and human similarities are drawn because of a lack of severe testing of hypotheses.•We attribute the lack of severe testing to bias of publishing ‘positive’ results that highlight DNN-human similarities.•We argue that a better appreciation of severe testing is needed at both the research and evaluation stages to obtain a better characterization of DNN-human similarities.