The Multimodal Kappa Effect: Context-dependence of Sensory Dominance in Time Perception

When processing information from multiple sensory sources, inputs seem to be weighted according to the reliability of their estimate (Driver & Spence, 2000; Ernst & Bülthoff, 2003). This differential weighting typically leads to a dominance of one sense over another for a given percept. For example, in temporal perception, the auditory system dominates over the visual system (Repp & Pennel, 2002; Romei, De Haas, Mok, & Driver, 2011), whereas in spatial perception, the visual system dominates (Bertelson & Aschersleben, 1998). However, temporal decisions are not independent of spatial features. For example, in the Kappa effect, the time interval between two stimuli appears longer when they originate from spatially distant rather than the same source (Cohen, Hansel, & Sylvester, 1954; Price-Williams, 1954). We sought to investigate this Kappa effect in a bimodal context, wherein temporal information was always congruent across modalities but spatial information was not. In Experiment 1, participants reproduced a temporal interval between two auditory markers across three different conditions. In the unimodal condition, the auditory markers were presented either from the same or different spatial sources. In the two bimodal conditions, visual stimuli were presented synchronously with the auditory markers. In bimodal congruent trials, the spatial source of visual and auditory stimuli was identical. In bimodal incongruent trials, auditory stimuli always originated from the same spatial source, but visual stimuli came from different sources. Consistent with previous results, a Kappa effect was observed for unimodal auditory intervals. This effect was also observed in bimodal trials. Most interestingly, even when the task-relevant auditory markers came from the same spatial source, reproductions were prolonged if the irrelevant visual stimuli came from different spatial sources (i.e., in incongruent trials). Thus, despite the typical dominance of the auditory modality, the irrelevant visuo-spatial information biased the perception of the relevant audio-temporal information. Yet, one might alternatively argue that participants based their reproductions simply on the visual stimuli instead of the auditory ones. Thus, in Experiment 2, the main results were replicated by employing a temporal discrimination task, in which two intervals were marked by three successive auditory stimuli. The first two stimuli marking the first interval came from the same spatial source, whilst the third could be either from the same or a different source. In the two bimodal conditions, two visual stimuli were presented simultaneously with the first and third auditory markers, thus providing spatial information, but no temporal information about the duration of each interval. Again, these visual stimuli could be either spatially congruent or incongruent with the auditory marker locations. As in Experiment 1, a Kappa effect was found in all conditions. In summary, the Kappa effect found in the bimodal incongruent conditions suggests that visuospatial information dominated the audio-spatial information, and thereby even overruled the typical dominance of the auditory modality for temporal perception. Thus, these results suggest an important role for context in sensory dominance. That is, when information from different modalities is present, the extent to which one sense drives the overall percept depends on the interaction between multiple perceptual domains, such as space and time.