Deep-learning based reconstruction in optical scanning holography

•The manuscript propose a deep-learning network for high quality image reconstruction of digital holograms recorded by optical scanning holography (OSH) system for the first time.•A two-pupil optical heterodyne scanning system is used to obtain the training data where a five-fold cross validation method is applied to prevent from overfitting and produce enough images in the dataset.•Numerical simulations and optical experiments demonstrate that our proposed method can eliminate the defocus noise and generate high quality reconstruction results significantly if correctly trained with our dataset.