The use of active learning systems for stimulus selection and response modelling perception experiments

To study the role of perceptual cues on categorization and decision making, participants are typically tested in (perception) experiments with a fixed set of randomized or pseudo-randomized trials. In linguistics and psycholinguistics, for instance, studies often investigate the relative weighting of different cues for a linguistic contrast (e.g., intonation vs. word order). For categorization beyond the segmental level (e.g., /p/ vs. /b/), it is important to establish that results generalise to different words or sentences, which necessitates the use of a range of different items. This may limit the number of conditions (cues and cue combinations) that can be sensibly tested in the same experiment. We show that Active Learning (AL) systems provide a solution: Since stimulus selection is informed by the system's learning mechanism (presenting certain conditions less often than uncertain conditions), they allow for efficient testing of numerous conditions and different items in the same experiment. In this paper, we compared two weighting approaches (average probability-based vs. regression-based) to model the outcome of three simulated scenarios with three binary factors each. Results show that valid results (i.e., little error between predicted values and the actual responses at the end of the experiment) are obtained after about a third of the trials of an original psycholinguistic experiment we replicated. For simulations with interactions between factors, the regression-based approach performed better. Our findings bear implications for the application of AL in psycholinguistic research (extraction of cue weights, inferential statistics, and a stopping criterion during an on-going experiment), which we will discuss.