Combating Short Circuit Behavior in Natural Language Reasoning: Crossover and Mutation Operations for Enhanced Robustness

In this study, we delve into the “short circuit” phenomenon observed in multiple-choice natural language reasoning tasks, where models tend to make accurate choices without properly considering the context of the question. To better understand this phenomenon, we propose white-box and black-box proxy tests as investigative tools to detect short circuit behavior, confirming its presence in fine-tuned NLU reasoning models. To tackle the short circuit issue, we introduce biologically inspired “crossover” and “mutation” operations. These operations are applied to augment the training data for popular models such as BERT, XLNet, and RoBERTa. Our results demonstrate that these data augmentation techniques effectively enhance the models’ robustness and mitigate the short circuit problem.