Automatic medical image interpretation: State of the art and future directions

•Image interpretation is an emerging field of artificial intelligence.•A good amount of research has been published with different titles that may include caption generation, image interpretation, video captioning, deep captioning.•For medical image analysis and interpretation, the work is little at present and need attention of researchers to produce high performance algorithms in order to apply these methods in clinical practices.•This work reviews recent advances in describing medical images in natural and medical language.•The work compares and discusses the strengths and short coming of state of the art work and also proposes the dimensions that can be explored for future work.