Using student perceptions in development , validation and application of an assessment questionnaire

This paper reports on a two-stage study aimed at developing, validating and applying an instrument that can be used to assess secondary students’ perceptions of assessment. In the first stage, following a review of literature, a six-scale instrument of 48 items was trialed with a sample of 470 students from grades eight, nine and ten in 20 science classrooms in three Western Australian schools. Based on internal consistency reliability data and exploratory factor analysis, refinement decisions resulted in a fivescale instrument that was named the Student Perceptions of Assessment Questionnaire (SPAQ). The scales of the SPAQ are Congruence with Planned Learning, Authenticity, Student Consultation, Transparency and Diversity. In the second stage, the SPAQ was used with five scales of the What is Happening in this Class (WIHIC) questionnaire, an attitude scale, and a self-efficacy scale. This survey was administered to a larger sample of nearly 1,000 students from 40 science classes from the same grades as in the first stage. Statistical analyses confirmed the validity and reliability of the SPAQ. The mean score ranged from 2.16 for the scale of Student Consultation to 3.17 for the scale of Congruence with Planned Learning on a four point Likert type scale. Cronbach Alphas ranged from 0.62 to 0.83. Significant correlations (p < 0.01) were found among all the scales used in the instrument, for example, Congruence with Planned Learning was positively related with all the other scales of SPAQ. Introduction: Despite the growth in emancipatory conceptualizations of classrooms that embrace a constructivist epistemology, little contemporary evidence exists to support the view that students are genuinely involved in decision-making about their assessment tasks. That is, forms of Assessment and specific assessment tasks employed in schools are overwhelmingly decided by teachers and administrators. Furthermore, even though reports like The Status and Quality of Teaching and Learning in Australia (Goodrum, Hackling, & Rennie, 2001) have asserted that assessment is a key component of the teaching and learning process, teachers tend to utilize a very narrow range of assessment strategies on which to base feedback to parents and students. In practice, there is little evidence that teachers actually use diagnostic or formative assessment strategies to inform planning and teaching (Randnor, 1996). Teachers feel that they need to ‘sacrifice learning with understanding for the goal of drilling students in the things for which they will be held accountable’ (Hobden, 1998). Historically, teachers have received substantial levels of advice on assessment practices. Harlen (1998) advises teachers that both oral and written questions should be used in assessing student’s learning. The inclusion of alternative assessment strategies, such as teacher observation, personal communication, and student performances, demonstrations, and portfolios, have been offered by experts as having greater usefulness for evaluating students and informing classroom instruction (Dorr-Bremme & Herman, 1986; Stiggins, 1994). Tobin (1998) asserted that assessment can be used to provide opportunities for students to show what they know. Reynolds, Doran, Allers & Agruso (1995) argued that for effective learning to occur, congruence must exist between instruction, assessment and outcomes. This paper represents contextspecific investigation of this congruence. An effective assessment process should involve a two-way communication system between teachers and their students. Historically, teachers have used testing instruments to transmit to the student and their parents what is really important for the student to know and do. While this reporting tends to be in the form of a grade, the form and design of the assessment can send subtle messages on what is important. There has been a substantial amount of research into types of assessment but very little research into students’ perceptions of assessment (Black & Wiliam, 1998; Crooks, 1998; Plake, 1993, Popham, 1997). Use of Student Perceptual Data: Until the late 1960s. a very strong tradition of trained observers coding teacher and student behaviors dominated classroom research. Indeed, it was a key recommendation of Dunkin and Biddle (1974) that instruments for research on teaching processes, where possible, should deal with the objective characteristics of classroom events. Clearly, this low-inference approach to research which often involved trained observers coding teacher and student behaviours was consistent with the behaviourism of the 1960s. The study of classroom psychosocial environments broke this tradition and used student perceptual data in the late 1960s. Since then, the strong trend in classroom environment research has been towards this high-inference approach with data collected from both the teachers and students. Walberg (1976) supports this methodological approach where student learning involves student perceptions acting as mediators in the learning process. Walberg (1976) also advocates the use of student perception to assess environments because students seemed quite able to perceive and weigh stimuli and to render predictively valid judgments of the social environments of their classes. Classroom Learning Environment: Recent reviews (e.g. Fraser, 1998, 1994) show that science education researchers have led the world in the field of classroom environment research over the last three decades, and that this field has contributed much to understanding and improving science education. For example, classroom assessments provide a means of monitoring, evaluating and science teaching and curriculum. A key to improving student achievement and attitudes is to create learning environments that emphasize those characteristics that have been found to be linked empirically with student outcomes. The WIHIC questionnaire brings parsimony to the field of learning environment by combining modified versions of the most salient scales from a wide range of existing questionnaires with additional scales that accommodate contemporary educational concerns (e.g., equity and cooperation, Fraser, (1998). Based on the previous studies, (Fraser, Fisher, & McRobbie, 1996) developed this new learning environment instrument. The WIHIC questionnaire has been used to measure the psychosocial aspects of the classroom learning environment in various contexts since its development. In certain cases, the questionnaire has been adapted without any modifications, while as in other cases modifications were made to suit the specific context. Until now, the original questionnaire in English has been translated into Chinese for use in Taiwan (Aldridge, Fraser, & Huang, 1998) and Singapore (Chionh & Fraser, 1998)and Korean for use in Korea (Kim, Fisher, & Fraser, 2000). Academic Efficacy: Over the past two decades the broad psychological concept of self-efficacy has been the subject of interest (Bandura, 1997; Schunk, 1995) Within this field, one particular strong area of interest is that of academic efficacy, which refers to personal judgments of one’s capabilities to organize and execute courses of action to attain designated types of educational performances (Zimmerman, 1995). Research studies have provided consistent, convincing evidence that academic efficacy is positively related to academic motivation (e.g. Schunk & Hanson, 1985), persistence (Lyman, Pretice-Dunn, Wilson & Bonfilio, 1984), memory performance (Berry, 1987), and academic performance (Schunk, 1989). Aim: The overall aim of the study was to investigate relationships among students’ perceptions of their assessment tasks, classroom learning environments, academic efficacy and attitude to science in years eight, nine and ten. Objectives of this study were: 1. to develop and validate an instrument to assess students’ perceptions of assessment tasks for use with middle school students; 2. to investigate associations between students’ perceptions of assessment tasks and their attitudes towards science. 3.to investigate gender and year level differences in students’ perceptions of assessment tasks in science classes; Instruments and Procedure Used: The study was carried out in phases over a period of three years using a multi-method research approach: 1.In the first phase Perceptions of Assessment Tasks (PAT) a six-scale instrument of 48 items from a 55 item questionnaire developed by Schaffner, Burry, Cho, Boney and Hamilton (2000) was administered to 470 students from grades eight, nine and ten in 20 science classrooms in three Western Australian schools. Close-ended interviews were conducted with 40 students to provide further data on the validation of the instrument. Internal consistency reliability data and exploratory factor analysis, refinement decisions resulted in a five-scale instrument that was named the Student Perceptions of Assessment Questionnaire (SPAQ). 2. In second phase based on The SPAQ was used with an attitude scale, and a self-efficacy scale. This survey was administered to a larger sample of nearly 1,000 students from 41 science classes from the same grades as in the first phase. Students’ Perceptions of Assessment Questionnaire (SPAQ) The SPAQ contains 30-items assigned to internally consistent scales, namely, Congruence with Planned Learning, Authenticity, Student Consultation, Transparency and Diversity. Table 1 shows a description of the scales with sample items.. Responses to the SPAQ items are recorded on a four point Likert type format of Almost Never, Sometimes, Often, and Almost Always. Validation statistics along with confirmatory factor analysis performed on the data collected are presented in results section Two outcome scales namely Attitude to Science and Academic Efficacy were also employed in the present study. Attitude to Science was assessed on an 8-item scale adopted from the Test of Science-Related Attitudes (TOSRA: Fraser, 1981).