How Often Do I Agree: an Experimental Test of Item Format Method Variance in Stress Measures

Three studies experimentally investigated response formats of agreement (A) and frequency (F) to determine their effect on correlations between often used measures of job stressors and strains. Study 1 used a within-subject design where respondents were given two sets of the same 6 stressor measures that varied in A versus F formats, as well as measures of 6 strains. Study 2 replicated Study 1 using a between-subject design where each respondent was randomly assigned to either the A or F stressor scale format. Study 3 utilized a 2 (A vs. F for stressors) by 2 (A vs. F for strains) design to test the assumption that matched response formats would result in higher correlations than unmatched due to common method variance. The first two studies provide little evidence for a systematic difference between A and F formats in predicting strains. Study 3 found that matching formats more often resulted in lower, not higher, correlations between stressor and strain pairs. Taken together, these experimental studies suggest that response format has little effect on correlations, means, standard deviations, and internal consistencies, and is not a method variance concern, at least with these commonly used measures of job stressors and strains.