The three R's of computer vision

•Bidirectional interactions between recognition, reconstruction and re-organization are very important.•Bottom-up grouping generates object candidates, which can be classified top down, following which the segmentations are refined again.•Recognition of 3D objects benefits from a reconstruction of 3D structure.•3D reconstruction benefits from object category-specific priors.•Reconstruction of 3D structure from video data goes hand in hand with the reorganization of the scene.