Building and Revising an Assessment to Measure Students’ Self-Efficacy in Systems Thinking

Given the growing interest in teaching systems thinking and systems engineering in traditional engineering majors like mechanical engineering, a critical goal involves creating valid and reliable methods for assessing students’ development in systems thinking and systems engineering. To measure students’ development, it is important to distinguish changes in performance from changes in attitudes and beliefs. This paper discusses ongoing refinements to the first half of an existing Systems Thinking Skills Survey (STSS; Muci-Kuchler, 2017), which is designed to measure students’ self-efficacy (i.e., belief in one’s capacity, cf. Bandura, 1997; 2006) to apply their knowledge and skills to systems engineering problems. Across multiple semesters, pre- and post STSS datasets from two universities were analyzed to gauge the validity and internal consistency of 44 self-efficacy items. Results guided changes to the items, while also enabling a reduction in the instrument’s length. With a noticeable reduction to only 30 items, the new version showed higher internal consistency (as measured by Cronbach’s alpha), while maintaining validity. In addition, specific subscales were identified to measure students’ self-efficacy for particular systems engineering topic/skill areas, e.g., concept generation, setting target specifications, identifying customer needs, and systems architecture. Finally, results from administering the new version as a pre/post-test showed that students had significant gains in self-efficacy after taking a mechanical engineering course that incorporated systems thinking/systems engineering material. These significant gains were present in students’ overall self-efficacy ratings, as well as their ratings for each self-efficacy subscale. Overall, these results highlight the value of iterating on assessment practices as part of curriculum development.