A Framework for Conceptualizing and Evaluating the Validity of Instructionally Relevant Assessments

Pellegrino, James W.; DiBello, Louis V.; Goldman, Susan R.

doi:10.1080/00461520.2016.1145550

Cited by 139 publications

(144 citation statements)

References 22 publications

Supporting

Mentioning

127

Contrasting

Order By: Relevance

“…In developing, refining, and applying our design process to multiple performance expectations in the physical and life sciences, we have been very cognizant of generalization and validation issues. Our validation efforts have been guided by the framework presented by Pellegrino, DiBello, and Goldman (2016) for instructionally supportive assessments. That framework discusses three major components of validity for which systematic evidence should be sought above and beyond articulation of the ECD process: Cognitive, Instructional, and Inferential.…”

Section: Validation Of the Design Process And Productsmentioning

confidence: 99%

Designing Knowledge‐In‐Use Assessments to Promote Deeper Learning

Harris

Krajcik

Pellegrino

et al. 2019

Educational Measurement

Self Cite

102

View full text Add to dashboard Cite

Contemporary views on learning highlight that deep learning occurs not simply by accumulating knowledge, but by using and applying knowledge as one engages in disciplinary activity. Increasingly, those concerned with education policy and practice are shifting priorities toward supporting deeper learning by emphasizing the importance of students’ ability to apply knowledge in subject areas. Designers of student assessments are following suit and are taking on the challenge of creating a new generation of assessments. We present a principled approach for designing classroom‐based assessments that not only assess deeper learning, but also provide teachers with critical information about how students are progressing toward achieving ambitious new learning goals. Our approach follows the evidentiary reasoning of evidence‐centered design and builds on research about the important role of knowledge‐in‐use to support student learning. We illustrate our approach in the context of creating tasks that assess students' science proficiency as reflected in the Next Generation Science Standards that are gaining prominence in the United States.

show abstract

Section: Validation Of the Design Process And Productsmentioning

confidence: 99%

Designing Knowledge‐In‐Use Assessments to Promote Deeper Learning

Harris

Krajcik

Pellegrino

et al. 2019

Educational Measurement

Self Cite

102

View full text Add to dashboard Cite

show abstract

“…A valid assessment design for professional and complex competencies like IP implies an authentic modeling of the world of work where examinees can think and act as if at real workplaces (Darling-Hammond et al 2013;Janesick 2006;Pellegrino et al 2016;Shavelson 2012;Wiggins 1998;Winther and Achtenhagen 2009). Therefore, all 22 diagnostic tasks reflect typical IP situations as they were identified by domain analysis.…”

Section: Tasks: Diagnostic Assessment Tasks For the Domain Of Ipmentioning

confidence: 99%

Developing and validating a technology-based diagnostic assessment using the evidence-centered game design approach: an example of intrapreneurship competence

Bley¹

2017

Empirical Res Voc Ed Train

View full text Add to dashboard Cite

Background: Educational experts commonly agree that tailor-made guidance is the most efficient way to foster the learning and developmental process of learners. Diagnostic assessments using cognitive diagnostic models (CDMs) have the potential to provide individual profiles of learners' strengths and weaknesses on a fine-grained level that can enable educators to assess the current position of learners. However, to obtain this necessary information a strong connection has to be made between cognition (the intended competence), observation (the observed learners' responses while solving the tasks), and interpretation (the inferences made based on the observed responses of learners' underlying competencies). To secure this stringent evidencebased reasoning, a principled framework for designing a technology-based diagnostic assessment is required-such as the evidence-centred game design (ECgD). Aim:With regard to a diagnostic assessment, three aspects are of particular importance according to the ECgD approach: (I) the selection of a measurable set of competence facets (so-called skills) and their grain-size, (II) the constructed pool of skill-based tasks, and (III) the clear and valid specified task to skill assignments expressed within the so-called Q matrix. The Q matrix represents the a priori assumption for running the statistical CDM-procedure for identifying learners' individual competence/skill profiles. These three prerequisites are not simply set by researchers' definition nor by experts' common sense. Rather, they require their own separate empirical studies. Hence, the focus of this paper is to evaluate the appropriateness and coherence of these three aspects (I: skill, II: tasks, and III: Q matrix). This study is a spin-off project based on the results of the governmental ASCOT research initiative on visualizing apprentices' workrelated competencies for a large-scale assessment-in particular, the intrapreneurship competence of industrial clerks. With the development of a CDM I go beyond the IRT-scaling offering the prerequisites for identifying individuals' skill profiles as a point of departure for an informative individual feedback and guidance to enhance students' learning processes.Methods: Therefore, I shall use a triangulated approach to generate three empirically based Q matrix models from different sources (experts and target-group respondents), inquiry methods (expert ratings and think-aloud studies), and methods of analyses (frequency counts and a solver-non-solver comparison). Consequently, the four single Train (2017) Train (2017) 9:6 Q matrix models (researchers' Q matrix generated within the task construction process and the three empirically based Q matrix models) were additionally matched by different degrees of overlap for balancing the strengths and weaknesses of each source and method. By matching the patterns of the four single Q matrix models, the appropriateness of the set of intrapreneurship skills (I) and the pool of intrapreneurship tasks (II) were investigated. ...

show abstract

“…The top part of the figure (Instructionally Relevant Assessment) is adapted from Pellegrino et al. (). The bottom part of the figure (Psychologically Relevant Assessment) is the author's creation.…”

Section: The Problem: Lack Of Understanding Of Students’ Interpretatimentioning

confidence: 99%

“…Teaching and assessment scholars may disagree with this latter statement but the research bears this out (see Patrick, Anderman, Bruening, & Duffin, 2011). As shown in the top part of Figure 1, teachers and assessment scholars may have the expertise to generate instructionally relevant formative assessment (Pellegrino et al, 2016) but not to investigate whether the assessment is psychologically relevant for students. Instructionally relevant assessment is necessary, and feeds into psychologically relevant assessment for students.…”

Section: Psychological Expertise Required To Explore Student Interprementioning

confidence: 99%

Students’ Interpretation of Formative Assessment Feedback: Three Claims for Why We Know So Little About Something So Important

Leighton

2019

J Educational Measurement

View full text Add to dashboard Cite

If K‐12 students are to be fully integrated as active participants in their own learning, understanding how they interpret formative assessment feedback is needed. The objective of this article is to advance three claims about why teachers and assessment scholars/specialists may have little understanding of students’ interpretation of formative assessment feedback. The three claims are as follows. First, there is little systematic research of K‐12 students’ interpretations of feedback. Systematic research requires gathering substantive evidence of students’ cognitive and emotional processes using psychological methods and tools. Second, there is an overemphasis on the external assessment process at the expense of uncovering learners’ internal reasoning and emotional processes. This overemphasis may be due to vestiges of behavioral approaches and lack of training in social cognitive methods. Third, there are psychological tools such as the clinical interview, pioneered by Piaget and used by psychologists to “enter the child's mind,” which may be helpful in uncovering students’ interpretation of feedback and associated behavioral responses. If the purpose of formative assessment is to change student learning, and feedback is delivered as a conduit to help with this long‐term change, understanding students’ interpretation of feedback plays a central role in the validity of the process.

show abstract

A Framework for Conceptualizing and Evaluating the Validity of Instructionally Relevant Assessments

Cited by 139 publications

References 22 publications

Designing Knowledge‐In‐Use Assessments to Promote Deeper Learning

Designing Knowledge‐In‐Use Assessments to Promote Deeper Learning

Developing and validating a technology-based diagnostic assessment using the evidence-centered game design approach: an example of intrapreneurship competence

Students’ Interpretation of Formative Assessment Feedback: Three Claims for Why We Know So Little About Something So Important

Contact Info

Product

Resources

About