No abstract

Background Concept inventories (CIs) are commonly used in engineering disciplines to assess students' conceptual understanding and to evaluate instruction, but educators often use CIs without sufficient evidence that a structured approach has been applied to validate inferences about student thinking. Purpose We propose an analytic framework for evaluating the validity arguments of CIs. We focus on three types of claims: that CI scores enable one to infer (1) students' overall understanding of all concepts identified in the CI, (2) students' understanding of specific concepts, and (3) students' propensity for misconceptions or common errors. Method We applied our analytic framework to three CIs: the Concept Assessment Tool for Statics (CATS), the Statistics Concept Inventory (SCI), and the Dynamics Concept Inventory (DCI). Results Using our analytic framework, we found varying degrees of support for each type of claim. CATS and DCI analyses indicated that the CIs could reliably measure students' overall understanding of all concepts identified in the CI, whereas SCI analyses provided limited evidence for this claim. Analyses revealed that the CATS could accurately measure students' understanding of specific concepts; analyses for the other two CIs did not support this claim. None of the CI analyses provided evidence that the instruments could reliably measure students' misconceptions and common errors. Conclusions Our analytic framework provides a structure for evaluating CI validity. Engineering educators can apply this framework to evaluate aspects of CI validity and make more warranted uses and interpretations of CI outcome scores.

No abstract

No abstract

This paper defines Bayesian network models and examines their applications to IRT‐based cognitive diagnostic modeling. These models are especially suited to building inference engines designed to be synchronous with the finer grained student models that arise in skills diagnostic assessment. Aspects of the theory and use of Bayesian network models are reviewed, as they affect applications to diagnostic assessment. The paper discusses how Bayesian network models are set up with expert information, improved and calibrated from data, and deployed as evidence‐based inference engines. Aimed at a general educational measurement audience, the paper illustrates the flexibility and capabilities of Bayesian networks through a series of concrete examples, and without extensive technical detail. Examples are provided of proficiency spaces with direct dependencies among proficiency nodes, and of customized evidence models for complex tasks. This paper is intended to motivate educational measurement practitioners to learn more about Bayesian networks from the research literature, to acquire readily available Bayesian network software, to perform studies with real and simulated data sets, and to look for opportunities in educational settings that may benefit from diagnostic assessment fueled by Bayesian network modeling.