Do humans integrate auditory and text information in a statistically optimal fashion?

Decreased speech perception in noise is a major problem, especially for older adults and for those with hearing loss. In order to improve speech perception in noise, the acoustic speech information can be supplemented with visually displayed speech text derived from the output of an automated speech recognition program. While such supplementation of auditory speech information with visual text has been shown to increase speech perception in noise, it is relatively unknown how listeners combine these two modalities of speech information to achieve improved speech perception. We aim to evaluate how listeners combine visual speech text and auditory speech information. We will evaluate subjects’ speech perception based only on auditory information in background noise. We will then evaluate subjects’ speech perception based only on visual text information that is derived from an automated speech recognition system. Using these two assessments of speech perception (auditory alone, visual text alone), we will predict listeners’ combined performance based on a maximum likelihood estimate model. We hypothesize that listener’s performance on joint presentation of auditory speech information with visual text information can be predicted based on the listener’s performance on separate presentations of auditory speech information alone and visual text information alone.