Modeling Diagnostic Assessments with Bayesian Networks

Almond, Russell G.; DiBello, Louis V.; Moulder, Brad; Zapata‐Rivera, Diego

doi:10.1111/j.1745-3984.2007.00043.x

Cited by 63 publications

(43 citation statements)

References 15 publications

(25 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In the same publication, Wendler and Walker (2006) discussed IRT methods of scoring, and Davey and Pitoniak (2006) discussed designing CATs, including use of IRT in scoring, calibration, and scaling. Almond et al (2007) described Bayesian network models and their application to IRT-based CD modeling. The paper, designed to encourage practitioners to learn to use these models, is aimed at a general educational measurement audience, does not use extensive technical detail, and presents examples.…”

Section: Explanation Evaluation and Application Of Irt Modelsmentioning

confidence: 99%

Item Response Theory

Carlson

Davier

2017

Methodology of Educational Measurement and Assessment

View full text Add to dashboard Cite

Item response theory (IRT) models, in their many forms, are undoubtedly the most widely used models in large-scale operational assessment programs. They have grown from negligible usage prior to the 1980s to almost universal usage in largescale assessment programs, not only in the United States, but in many other countries with active and up-to-date programs of research in the area of psychometrics and educational measurement.Perhaps the most important feature leading to the dominance of IRT in operational programs is the characteristic of estimating individual item locations (difficulties) and test-taker locations (abilities) separately, but on the same scale, a feature not possible with classical measurement models. This estimation allows for tailoring tests through judicious item selection to achieve precise measurement for individual test takers (e.g., in computerized adaptive testing, CAT) or for defining important cut points on an assessment scale. It also provides mechanisms for placing different test forms on the same scale (linking and equating). Another important characteristic of IRT models is local independence: for a given location of test takers on the scale, the probability of success on any item is independent of that of every other item on that scale. This characteristic is the basis of the likelihood function used to estimate test takers' locations on the scale.Few would doubt that ETS researchers have contributed more to the general topic of IRT than individuals from any other institution. In this chapter we briefly review most of those contributions, dividing them into sections by decades of publication. Of course, many individuals in the field have changed positions between

show abstract

Section: Explanation Evaluation and Application Of Irt Modelsmentioning

confidence: 99%

Item Response Theory

Carlson

Davier

2017

Methodology of Educational Measurement and Assessment

View full text Add to dashboard Cite

show abstract

“…Although these efforts are not explicitly concerned with constructed responses, they support such assessments by accommodating polytomous responses. One approach is based on Bayesian networks (Almond et al 2007), whereas the second approach follows a latent variable tradition (von Davier 2013).…”

Section: Diagnostic and Formative Assessmentmentioning

confidence: 99%

A Historical Survey of Research Regarding Constructed-Response Formats

Bejar

2017

Methodology of Educational Measurement and Assessment

View full text Add to dashboard Cite

This chapter chronicles ETS research and development contributions related to the use of constructed-response item formats. 1 The use of constructed responses in testing dates back to imperial China, where tests were used in the selection of civil servants. However, in the United States, the multiple-choice format became dominant during the twentieth century, following its invention and use by the SAT ® examinations created by the College Board in 1926. When ETS was created in 1947, post-secondary admissions testing was largely based on tests consisting of multiplechoice items. However, from the start, there were two camps at ETS: those who believed that multiple-choice tests were sufficiently adequate for the purpose of assessing "verbal" skills and those who believed that "direct" forms of assessment requiring written responses had a role to play. For constructed-response formats to regain a foothold in American education several hurdles would need to be overcome. Research at ETS was instrumental in overcoming those hurdles.The first hurdle was that of reliability, specifically the perennial issue of low interrater agreement, which plagued the acceptance of constructed-response formats for most of the twentieth century. The second hurdle was broadening the conception of validity to encompass more than predictive considerations, a process that began with the introduction of construct validity by Cronbach and Meehl (1955). Samuel Messick at ETS played a crucial role in this process by making construct validity relevant to educational tests. An inflexion point in the process of reincorporating constructed-response formats more widely in educational tests was marked 1 Constructed responses to a prompt or question can range in scope and complexity. Perhaps the most common constructed response is the written essay. However, short written responses to questions are also considered to be constructed, as are spoken answers in response to a prompt, mathematical responses (equations, plotted functions, etc.), computer programs, and graphical responses such as architectural designs.

show abstract

“…Bayes Nets are also well understood models previously presented extensively in the research literature as stand-alone models and used for instance in classroom-based assessments (Almond, DiBello, Moulder, & Zapata-Rivera, 2007;Almond, Mulder, Hemat, & Yan, 2009 Bayes Nets are acyclic directed graph structures, represented with nodes and arcs, which are shown in the Results section in an example. To generate the joint probabilities in the network, a strong assumption is made in Bayesian Network approaches that P, the local probability distribution of variable X i , is made conditional only on the value of the parent nodes connecting to a given node from above.…”

Section: Relevant Scholarshipmentioning

confidence: 99%

Hybrid Measurement Models for Technology-Enhanced Assessments Through mIRT-bayes

Scalise

2017

IJSP

View full text Add to dashboard Cite

Technology-enhanced assessments (TEAs) are rapidly emerging in educational measurement. In contexts such as simulation and gaming, a common challenge is handling complex streams of information, for which new statistical innovations are needed that can provide high quality proficiency estimates for the psychometrics of complex TEAs. Often in educational assessments with formal measurement models, latent variable models such as item response theory (IRT) are used to generate proficiency estimates from evidence elicited. Such robust techniques have become a foundation of educational assessment, when models fit. Another less common approach to compile evidence is through Bayesian networks, which represent a set of random variables and their conditional dependencies via a directed acyclic graph. Network approaches can be much more flexibly designed for complex assessment tasks and are often preferred by task developers, for technology-enhanced settings. However, the Bayesian network-based statistical models often are difficult to validate and to gauge the stability and accuracy, since the models make assumptions regarding conditional dependencies that are difficult to test. Here a new measurement model family, mIRT-bayes, is proposed to gain advantages of both latent variable models and network techniques combined through hybridization. Specifically, the technique described here embeds small Bayesian networks within an overarching multidimensional IRT model (mIRT), preserving the flexibility for task design while retaining the robust statistical properties of latent variable methods. Applied to simulation-based data from Harvard's Virtual Performance Assessments (VPA), the results of the new model show acceptable fit for the overarching mIRT model, along with reduction of the standard error of measurement through the embedded Bayesian networks, compared to use of mIRT alone. Overall for respondents, a finer grain-size of inference is made possible without additional testing time or scoring resources, showing potentially promise for this family of new hybrid models.

show abstract

Modeling Diagnostic Assessments with Bayesian Networks

Cited by 63 publications

References 15 publications

Item Response Theory

Item Response Theory

A Historical Survey of Research Regarding Constructed-Response Formats

Hybrid Measurement Models for Technology-Enhanced Assessments Through mIRT-bayes

Contact Info

Product

Resources

About