Measuring reliability of object detection algorithms for automated driving perception tasks

ABSTRACTWe build a data-driven methodology for the performance reliability and the improvement of sensor algorithms for automated driving perception tasks. The methodology takes as input three elements: I) one or various algorithms for object detection when the input is an image; II) a dataset of camera images that represents a sample from an environment, and III) a simple policy that serves as a proxy for a task such as driving assistance. We develop a statistical estimator, which combines I)-III) and a data augmentation technique, in order to rank the reliability of perception algorithms. Reliability is measured as the chance of collision given the speed of the ego vehicle and the distance to the closest object in range. We are able to compare algorithms in the (speed vs distance-to-closest-object) space using p-values and use this information to suggest improved-safety algorithms.