Methodology Matters: Is There a Method ChoiceBias in Software Engineering?

As software engineering is a socio-technical research field, there are a myriad of research methods that researchers need to consider. Method choice determines different tradeoffs in terms of generalizabity, realism and control, among other attributes. In this paper, we reflect on the methods which tend to dominate software engineering research, and consider how humans tend to be involved in our research. We consider two years of ICSE proceedings and find that a majority of studies use computer simulation methods relying on trace measures rather than active human participation. This choice leads to a low level of control over extraneous factors. We question whether our method choice is out of convenience, or whether this choice reflects the kinds of research results our community values.