Sparse to Dense Depth Completion using a Generative Adversarial Network with Intelligent Sampling Strategies

ABSTRACTPredicting dense depth accurately is essential for 3D scene understanding applications such as autonomous driving and robotics. However, the depth obtained from commercially available LiDAR and Time-of-Flight sensors is very sparse. With RGB color guidance, modern convolutional neural network (CNN) based approaches can recover the missing depth information. However, there could be scenarios such as low-light environments where it might be difficult to get an associated RGB image with the sparse depth. In this work, we propose a Generative Adversarial Network (GAN) that can accurately predict the dense depth using only sparse samples without any RGB inputs. Generally, the sparsity in the depth samples is uniformly distributed and cannot guarantee capturing all intricate details. In this study, we also explore different variants of sparse sampling strategies from uniform to feature based directed sampling. We find that feature based intelligent sampling enjoys better compression ratio without sacrificing intricate details, saving data communication bandwidth. Compared to uniform sampling, depending on how aggressively the directed sampling is done, we observe about 3% to 25% reduction in size. We can easily reduce the size by 8% with directed sampling without sacrificing the reconstruction accuracy. Although such directed sampling strategies are not readily available with commercially viable depth sensors, we believe that our study paves the way for future intelligent sensing and sampling strategies. To further investigate data reduction and reconstruction accuracy trade-offs we deploy our GAN to generate higher resolution dense depth from 4 times smaller sparse samples. With slight decrease in accuracy, our GAN is able to recover the depth successfully which shows great promise in edge Internet of Things (IoT) applications where we have very tight constraint on data transmission bandwidth. Our source code along with examples is available at: https://github.com/kocchop/depth-completion-gan