Communicative efficiency in multimodal language

The ecology of communication is face-to-face. In these contexts, speakers dynamically modify their communication across vocal (e.g., speaking rate) and gestural (e.g., co-speech gestures related in meaning to the content of speech) channels while speaking. What is the function of these adjustments? Here we test the hypothesis that speakers produce these multimodal adjustments to increase communicative success but also decrease cognitive effort moment-by-moment depending upon word predictability and audience (children or adults). We assess whether speakers modulate word durations and produce gestures depending on the predictability (surprisal) of each word they utter (computed from computational language models trained on corpora of Child-Directed, or Adult-Directed Language). Using data from a novel corpus of naturalistic interactions between adult-child (aged 3-4), and adult-adult, we show that surprisal predicts speakers’ multimodal adjustments and that these effects were modulated by whether the comprehender was a child or an adult. Thus, communicative efficiency applies generally across vocal and gestural communicative channels not being limited to structural properties of language or vocal modality.