Investigating immersion and migration decisions for agent-based modelling: A cautionary tale

Background: Agent-based modelling provides an appealing methodological choice for simulating human behaviour and decisions. The currently-dominant approaches based on static transition rates or unverified assumptions are restrictive, and could be enhanced with insights from cognitive experiments on actual decision making. Here, one common concern is that standard surveys or experiments may lack ecological validity, limiting the extent to which research findings can be generalised to real-life settings. For complex, highly emotive decision-making scenarios, such as those related to migration, the typically-used short, methodical survey questions may not appropriately map onto complex real-world decisions of interest. Immersive contexts may offer more accurate representations of reality, potentially enhancing the usefulness of experimental information in multi-disciplinary modelling endeavours. Methods: This pre-registered study of migration decisions, aimed at informing a multi-disciplinary construction of an agent-based model of migration, presents a choice-based interactive fiction game in which players make migration decisions to advance through a story. Participants (N = 1000 Prolific users) were randomly assigned to one of four experimental conditions, three involving different renditions of the game attempting to create immersion, with the last condition presenting the decisions in standard survey format. Results: Although addressing the lack of ecological validity in survey data is important for improving agent-based modelling methodology, the experimental design used to tackle this issue, while responding directly to modelling needs, proved too complex. The created experimental conditions ended up too distinct from each other, involving stimuli that differed in quantity and content. This introduced several unintended and uncontrolled confounds, making it impossible to meaningfully interpret the results of this experiment on its own. Our results act as a cautionary tale for agent-based modellers, highlighting that the modelling needs should not override the principles of experimental design, and provide motivation for more rigorous research on this topic.