Applications of Bayesian Decision Theory to Sequential Mastery Testing

Abstract

The purpose of this paper is to formulate optimal sequential rules for mastery tests. The framework for this approach is derived from empirical Bayesian decision theory. Both a threshold and linear loss structure are considered. The binomial probability distribution is adopted as the psychometric model involved. Conditions sufficient for sequentially setting optimal cutting scores are presented. Optimal sequential rules will be derived for the case of a beta distribution representing prior true level functioning. An empirical example of sequential mastery testing for concept-learning in medicine concludes the paper. (Contains 6 tables and 51 references.) (SLD) ******************************************************************************** Reproductions supplied by EDRS are the best that can be made from the original document. ******************************************************************************** The purpose of this paper is to formulate optimal sequential rules for mastery tests. The Applications of Bayesian Decision

A new adaptive testing algorithm for shortening health literacy assessments

“…the cost of continuation of the test compared to the reduction in cost of misclassification (both false positives and false negatives) that can be expected to result from administering additional items. Vos[28], Welch and Frick[29] and Rudner[30,31] have discussed the extension of decision theory to item-selection.…”

An investigation of a Bayesian decision-theoretic procedure in the context of mastery tests

“…The decision is based on minimizing the posterior expected loss at each stage of testing. Vos (1997Vos ( , 2000Vos ( , 2002 has applied a similar procedure at the item level instead of the testlet level by using a binomial probability model. His procedure was named the Bayesian Sequential Rule (Bayes).…”

“…However, the Bayes procedure has drawn some attention recently for its ability to take the relative costs of erroneous testing outcomes into consideration explicitly. That is, the relative seriousness of false negative errors, false positive errors or administering additional items can be considered in advance in the Bayes procedure by specifying the appropriate values, which are called loss parameters (Vos, 1997(Vos, , 2000(Vos, , 2002.…”

“…The Bayesian method is a relatively new approach, which was developed by Lewis and Sheehan (1990) and expanded by Glas and Vos (1998). There are two basic elements in the application of Bayesian methods to decision making: the psychometric model used to relate the examinee's observed test scores and true ability level, and the loss structure employed to evaluate the costs of all possible decision outcomes (Vos, 1997). Most classification tests use an IRT model to connect the examinee's responses and his/her true level of ability.…”

“…Generally, the threshold loss function is less desirable since it assumes a constant loss for all examinees. Some authors (van der Linden et.al, 1977;Vos, 1999) noted that this assumption is probably unrealistic in some applications. It seems more realistic to assume that loss is an increasing function of theta for nonmasters and a decreasing function of theta for masters (van der Linden, 1996, Vos, 1997.…”