German National Proficiency Scales in Biology: Internal Structure, Relations to General Cognitive Abilities and Verbal Skills

Abstract

National and international large‐scale assessments (LSA) have a major impact on educational systems, which raises fundamental questions about the validity of the measures regarding their internal structure and their relations to relevant covariates. Given its importance, research on the validity of instruments specifically developed for LSA is still sparse, especially in science and its subdomains biology, chemistry, and physics. However, policy decisions for the improvement of educational quality based on LSA can only be helpful if valid information on students’ achievement levels is provided. In the present study, the nature of the measurement instruments based on the German Educational Standards in Biology is examined. On the basis of data from 3,165 students in Grade 10, we present dimensional analyses and report the relationship between different subdimensions of biology literacy and cognitive covariates such as general cognitive abilities and verbal skills. A theory‐driven two‐dimensional model fitted the data best. Content knowledge and scientific inquiry, two subdimensions of biology literacy, are highly correlated and show differential correlational patterns to the covariates. We argue that the underlying structure of biology should be incorporated into curricula, teacher training and future assessments.

Articulating the validity evidence for a science alternate assessment

“…Although assessment developers present validity evidence and arguments (e.g., Gotwals & Songer, ; Kampa & Koller, ; Opfer, Nehm, & Ha, ), few provide explicit theories of action that describe how a summative assessment functions within an educational system (Goldstein & Behuniak, ; Perie & Forte, ; Quenemoen, ; Reeves & Marbach‐Ad, ). Such theories of action are particularly important for assessments of students with SCD because validity arguments need to evaluate the plausibility of precursors and assumptions about students with SCD that contribute to score interpretation, such as their opportunities to learn science or abilities to interact with the assessment system (Marion & Pellegrino, ; Perie & Forte, ).…”

The Quest for the Holy Grail of Validity in Science Assessments: A Comment on Kampa and Köller (2016) “German National Proficiency Scales in Biology: Internal Structure, Relations to General Cognitive Abilities and Verbal Skills”

“…Table provides an overview of these sources for studies that primarily focus on studying item and test performance. On the basis of these sources and facets of validity, I review the evidence Kampa and Köller () tap in their paper for crafting a validity argument.…”

“…Performing elaborate statistical analyses based on uni‐ and multidimensional item response theory models, they find that both constructs can be empirically distinguished, although they are highly correlated. It is noteworthy that Kampa and Köller () did not attempt to study the internal structure of the two subscales by differentiating between, for instance, different processes and skills of scientific inquiry or different types of content knowledge (e.g., with respect to different scientific contexts or concepts); instead, they tried to disentangle the overall relation between content knowledge and scientific inquiry skills. Although a further differentiation of the two concepts may provide more fine‐grained information on the relations among the subprocesses and skills and therefore reveal instructionally relevant information about the specific strengths and weaknesses of performance or conceptual understanding (Hartig & Höhler, ; Leighton & Gierl, ; Wind & Gale, ), the reported findings nicely serve the purpose insofar that evidence for the empirical distinction and relation is provided.…”

“…In their recently published paper, Kampa and Köller () present an assessment of students’ content knowledge and scientific inquiry skills in the domain of biology and describe the validation thereof. This study showcases the creation of a validity argument of science assessments by focusing on the internal structure and the relations to other constructs, namely general cognitive abilities and verbal skills.…”

“…Which sources of evidence can inform the quest for validity? This essay briefly reviews Kampa and Köller's () validity argument and extends on potential ways to supplement it. Along these lines, it sketches approaches to validity science educators may want to take in order to examine different sources of evidence.…”