Relevant Information in TDD Experiment Reporting

Experiments are a commonly used method of research in software engineering(SE). Researchers report their experiments following detailed guidelines.However, researchers do not, in the field of test-driven development (TDD) atleast, specify how they operationalized the response variables and themeasurement process. This article has three aims: (i) identify the responsevariable operationalization components in TDD experiments that study externalquality; (ii) study their influence on the experimental results;(ii) determineif the experiment reports describe the measurement process components that havean impact on the results. Sequential mixed method. The first part of theresearch adopts a quantitative approach applying a statistical análisis (SA)of the impact of the operationalization components on the experimental results.The second part follows on with a qualitative approach applying a systematicmapping study (SMS). The test suites, intervention types and measurers have aninfluence on the measurements and results of the SA of TDD experiments in SE.The test suites have a major impact on both the measurements and the results ofthe experiments. The intervention type has less impact on the results than onthe measurements. While the measurers have an impact on the measurements, thisis not transferred to the experimental results. On the other hand, the resultsof our SMS confirm that TDD experiments do not usually report either the testsuites, the test case generation method, or the details of how external qualitywas measured. A measurement protocol should be used to assure that themeasurements made by different measurers are similar. It is necessary to reportthe test cases, the experimental task and the intervention type in order to beable to reproduce the measurements and SA, as well as to replicate experimentsand build dependable families of experiments.