The Quest for the Holy Grail of Validity in Science Assessments: A Comment on Kampa and Köller (2016) “German National Proficiency Scales in Biology: Internal Structure, Relations to General Cognitive Abilities and Verbal Skills”

Scherer, Ronny

doi:10.1002/sce.21278

Cited by 4 publications

(6 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Second, we aimed to apply the RALP to an ecologically valid sample of prospective science teachers to gain qualitative insights into their strengths and weaknesses. As suggested by Scherer (2017), referring to Pellegrino et al's (2016) Assessment Triangle (Figure 2) facilitates building a sound and persuasive validity argument since it automatically requires considering different participants (e.g., science teacher educators, PSTs, TSTs). Overall, various pieces of empirical evidence for validity were collected, and thus, a persuasive validity argument was built.…”

Section: Discussionmentioning

confidence: 99%

“…that reflect on the competency-based pedagogy. In this paper, we aim to build a validity argument for the RALP, and thereby, we illustrate that testing an instrument for validity is a challenging and complex endeavor (Kane, 2013); it is even understood as "the quest for the Holy Grail" (Scherer, 2017). Testing for validity needs to cover multiple aspects of validity (AERA, APA, & NCME, 2014) to build a sound and convincing validity argument and, thus, ensure that the scores can be interpreted validly.…”

Section: How the Ralp Can Be Beneficial For Science Teacher Educationmentioning

confidence: 99%

See 1 more Smart Citation

Assessing the quality of science teachers' lesson plans: Evaluation and application of a novel instrument

Großmann,

Krüger

2023

Science Education

View full text Add to dashboard Cite

Lesson planning is a core part of teachers' professional competence. Written lesson plans play a significant role in science teacher education as a preparation for demonstration lessons during the final teacher certification exam. However, the few existing scoring rubrics on lesson plans are not particularly theoretically sound and are barely tested for the validity of score interpretations. In response to the demand for transparent and applicable criteria, we developed the rubric to assess science lesson plans (RALP) to assess science teachers' lesson plan quality. We employed a mixed‐methods approach: First, we present multiple sources of validity evidence (based on test content, internal structure, relations to other variables, and consequences of testing) as mainly quantitative indicators for the quality of the RALP. Based upon that, we applied the RALP to lesson plans written by preservice and trainee science teachers (N = 100) and provided a qualitative analysis of six cases to illustrate common patterns in these lesson plans. Results indicate that teacher educators consider the RALP criteria (N = 24) relevant and objectively applicable. Correlation analyses of the scores and two teacher educators' holistic quality assessment of all lesson plans provide convincing evidence that the RALP can discriminate lesson plan quality levels. Moreover, comparisons between preservice science teachers and trainee science teachers reveal that trainee teachers score significantly higher than preservice teachers, indicating that the RALP is sensitive to differences in teaching and planning experience. The application and in‐depth analysis of three criteria of the RALP illustrate these differences in levels of planning quality. We discuss possible applications of the RALP in science teacher education and research in science teaching.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: How the Ralp Can Be Beneficial For Science Teacher Educationmentioning

confidence: 99%

Assessing the quality of science teachers' lesson plans: Evaluation and application of a novel instrument

Großmann,

Krüger

2023

Science Education

View full text Add to dashboard Cite

show abstract

“…Besides these, students’ background, for instance, indicated by their socioeconomic status, represents another source of individual differences ( Conley et al, 2004 ). It is therefore important to consider background variables when examining the adaptability-epistemological beliefs link and the invariance of the epistemological beliefs measure ( Scherer, 2017 ).…”

Section: Theoretical Frameworkmentioning

confidence: 99%

“…This question concerns the validity of the measure ( Pellegrino et al, 2016 ). Knowledge about the invariance of epistemological beliefs measures along the continuum of adaptability provides test developers and assessment specialist with information about the functioning of the measures and the cautions associated with the interpretation of the resultant test scores ( Scherer, 2017 ).…”

Section: Introductionmentioning

confidence: 99%

Observing the World Through Your Own Lenses – The Role of Perceived Adaptability for Epistemological Beliefs About the Development of Scientific Knowledge

Scherer

Guttersrud

2018

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

Students are exposed to vast amounts of information and knowledge that is rapidly changing. This exposure requires them to be adaptive, that is, to constantly adjust their thinking, behavior, and even their affect to successfully solve information-rich and knowledge-lean problems. Considering these developments, the purpose of the present study is twofold: First, it is aimed at exploring the link between students’ beliefs about their adaptability in an ever-changing world and their beliefs about the changing nature of scientific knowledge, thus linking two educationally relevant belief systems. Second, this study further explores validity issues related to the well-established and commonly used “Epistemological Beliefs about the Development of Scientific Knowledge (EBDE)” scale. Performing structural equation modeling on a large-scale data set of 1,662 Norwegian tenth-grade students, we estimated the correlations among different aspects of adaptability (i.e., cognitive-behavioral and affective-emotional adaptability) and EBDE. Moving beyond these correlations, we tested whether students’ perceived adaptability had an impact on the functioning of EBDE items by means of moderated factor analysis. Our analyses revealed that adaptability was associated with sophisticated EBDE in science, and the EB scale functioned differently with respect to different adaptability scores. The results of this study indicate that students perceive the development of scientific knowledge through the lenses of their own adaptability. Furthermore, the differential functioning of the EBDE scale challenges its validity.

show abstract

“…http://onlinelibrary.wiley.com/doi/10.1002/sce.21278/abstract () offers useful commentary about the varieties of “validity” that should become more widely appreciated by those in the Science Education community. Without criticizing the work that compelled his commentary, Scherer reminds us about multiple evidence sources beyond the standard approaches used by those developing large‐scale assessments.…”

Section: Theory‐like Constructs and Perspectivesmentioning

confidence: 99%

Doors labeled as times: Frames of observing and understanding

Settlage¹,

Southerland²

2017

Sci. Ed.

View full text Add to dashboard Cite

The Quest for the Holy Grail of Validity in Science Assessments: A Comment on Kampa and Köller (2016) “German National Proficiency Scales in Biology: Internal Structure, Relations to General Cognitive Abilities and Verbal Skills”

Cited by 4 publications

References 33 publications

Assessing the quality of science teachers' lesson plans: Evaluation and application of a novel instrument

Assessing the quality of science teachers' lesson plans: Evaluation and application of a novel instrument

Observing the World Through Your Own Lenses – The Role of Perceived Adaptability for Epistemological Beliefs About the Development of Scientific Knowledge

Doors labeled as times: Frames of observing and understanding

Contact Info

Product

Resources

About