Hough-CNN: Deep Learning for Segmentation of Deep Brain Regions in MRI and Ultrasound.

•We propose Hough-CNN, a novel segmentation approach based on a voting strategy. We show that the method is multi-modal, multi-region, robust and implicitly encoding priors on anatomical shape and appearance. Hough-CNN delivers results comparable or superior to other state-of-the-art approaches while being entirely registration-free. In particular, it outperforms methods based on voxel-wise, semantic classification.•Hough-CNN is scalable to different modalities with little change in parameterisation. We demonstrate multi-region segmentation in MRI and midbrain segmentation in 3D freehand transcranial ultrasound (TCUS).•We propose and evaluate several different CNN architectures, with varying numbers of layers and convolutional kernels per layer. In this way we acquire insights on how different network architectures cope with the amount of variability present in medical volumes and image modalities.•We evaluate the impact of the number of annotated training examples on the final segmentations by training the networks with different amounts of data. In particular, we show how complex networks with higher parameter number cope with relatively small training datasets.•We adapted the Caffe framework to perform convolutions of volumetric data, preserving its third dimension across the whole network. We compare CNN performance using 3D convolution to the more common 2D convolution, as well as to a recent 2.5D approach.