The Optimization of Multivariate Generalizability Studies with Budget Constraints

Marcoulides, George A.; Goldstein, Zvi

doi:10.1177/0013164492052002005

Cited by 10 publications

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several authors have studied how one can optimize generalizability coefficients, reliability coefficients, or validity coefficients, or how one can minimize decision error rates. In many cases this was studied in the context of a budget constraint (e.g., Allison et al, 1997 ; Ellis, 2013 ; Liu, 2003 ; Marcoulides, 1993 , 1995 , 1997 ; Marcoulides & Goldstein, 1990a , 1990b , 1992 ; Meyer et al, 2013 ; Sanders, 1992 ; Sanders et al, 1991 ) or fixed total testing time (e.g., Ebel, 1953 ; Hambleton, 1987 ; Horst, 1949 , 1951 ). Alternatively, one may minimize the costs of measurement given constraints on the generalizability coefficient or error variance (e.g., Peng et al, 2012 ; Sanders et al, 1989 ), which is a similar problem.…”

Section: Previous Optimization Approachesmentioning

confidence: 99%

A Simple Model to Determine the Efficient Duration of Exams

Ellis

2020

Educational and Psychological Measurement

View full text Add to dashboard Buy / Rent full text

This study develops a theoretical model for the costs of an exam as a function of its duration. Two kind of costs are distinguished: (1) the costs of measurement errors and (2) the costs of the measurement. Both costs are expressed in time of the student. Based on a classical test theory model, enriched with assumptions on the context, the costs of the exam can be expressed as a function of various parameters, including the duration of the exam. It is shown that these costs can be minimized in time. Applied in a real example with reliability .80, the outcome is that the optimal exam time would be much shorter and would have reliability .675. The consequences of the model are investigated and discussed. One of the consequences is that optimal exam duration depends on the study load of the course, all other things being equal. It is argued that it is worthwhile to investigate empirically how much time students spend on preparing for resits. Six variants of the model are distinguished, which differ in their weights of the errors and in the way grades affect how much time students study for the resit.

show abstract

Section: Previous Optimization Approachesmentioning

confidence: 99%

A Simple Model to Determine the Efficient Duration of Exams

Ellis

2020

Educational and Psychological Measurement

View full text Add to dashboard Buy / Rent full text

show abstract

“…However, they cannot apply the Lagrange multiplier to some more complicated designs (i.e., the mixed design and multivariate conceptual design and etc. ), which is deficient Goldstein, 1991, 1992;Macrolides, 1994Macrolides, , 1995 [27][28][29][30].…”

Section: Introductionmentioning

confidence: 99%

How Many Students and Items Are Optimal for Teaching Level Evaluation of College Teachers? Evidence from Generalizability Theory and Lagrange Multiplier

2022

Sustainability

View full text Add to dashboard Buy / Rent full text

Budget and cost are two of the problems that cannot be ignored when conducting a measure study. Based on the application of generalizability theory, combined with Lagrange multiplier, this paper explores how many students and items are optimal for teaching level evaluation of college teachers under budget constraints to maintain the sustainable development of higher education. A total of 397 students are required to evaluate 10 teachers’ teaching level using the Teaching Level Evaluation Questionnaire for College Teachers, and we make different generalizability designs (i.e., (s:t) × i, (s:t) × (i:v) and (s:t) × (i:v) × o) for the collected data. The study unifies the Lagrange multiplier formula, derives the optimal sample size formula of different designs under budget constraints in generalizability theory, and calculates the optimal sample size for teaching level evaluation of college teachers in different designs with the estimated variance components. Results indicate that: (1) the unified formula of Lagrange multiplier has a stronger robustness and can be applied to different study designs under budget constraints in generalizability theory; (2) the occasion has a great effect on teaching level evaluation for college teachers; (3) the (s:t) × (i:v) × o design has a high efficiency in estimating the optimal sample size of teaching level evaluation for college teachers; (4) the design of (s:t) × (i:v) × o is the optimal generalizability design of teaching level evaluation for college teachers under budget constraints in generalizability theory; and (5) under budget constraints of teaching level evaluation for college teachers in generalizability theory, the optimal sample size of students is 31 for each teacher and the optimal sample size of items is 7 for each dimension.

show abstract

“…Approaches have been proposed to maximize reliability within a budget constraint. Woodward and Joe (1973) derived equations for their constrained optimization; Saunders et al (1989) deployed discrete optimization, which was further updated by Saunders (1992) with the Cauchy–Schwartz inequity approach; Marcoulides (1993, 1995) as well as Marcoulides and Goldstein (1990, 1991, 1992) had developed the LaGrange multiplier approach and other related variants to handle the optimization within both univariate and multivariate G-theory. Meyer et al (2014) extended the LaGrange multiplier approaches to G-theory with nested designs.…”

Section: Introductionmentioning

confidence: 99%

A Short Note on Optimizing Cost-Generalizability via a Machine-Learning Approach

Jiang

Shi

DiStefano

2021

Educational and Psychological Measurement

View full text Add to dashboard Buy / Rent full text

The costs of an objective structured clinical examination (OSCE) are of concern to health profession educators globally. As OSCEs are usually designed under generalizability theory (G-theory) framework, this article proposes a machine-learning-based approach to optimize the costs, while maintaining the minimum required generalizability coefficient, a reliability-like index in G-theory. The authors adopted G-theory parameters yielded from an OSCE hosted by a medical school, reproduced the generalizability coefficients to prepare for optimizing manipulations, applied simulated annealing algorithm to calculate the number of facet levels minimizing the associated costs, and conducted the analysis in various conditions via computer simulation. With a given generalizability coefficient, the proposed approach, virtually an instrument of decision-making supports, found the optimal solution for the OSCE such that the associated costs were minimized. The computer simulation results showed how the cost reductions varied with different levels of required generalizability coefficients. Machine learning–based approaches can be used in conjunction with psychometric modeling to help planning assessment tasks more scientifically. The proposed approach is easy to adopt into practice and customize in alignment with specific testing designs. While these results are encouraging, the possible pitfalls such as algorithmic convergences’ failure and inadequate cost assumptions should also be avoided.

show abstract

The Optimization of Multivariate Generalizability Studies with Budget Constraints

Abstract: The optimization of multivariate measurement designs when limited resources are available is not a simple task. This paper presents a method for determining the optimal number of conditions to use in measurement designs when resource constraints are imposed.

Cited by 10 publications

References 5 publications

A Simple Model to Determine the Efficient Duration of Exams

A Simple Model to Determine the Efficient Duration of Exams

How Many Students and Items Are Optimal for Teaching Level Evaluation of College Teachers? Evidence from Generalizability Theory and Lagrange Multiplier

A Short Note on Optimizing Cost-Generalizability via a Machine-Learning Approach

Contact Info

Product

Resources

About