A Practical Solution to Optimizing the Reliability of Teaching Observation Measures Under Budget Constraints

Abstract

Researchers often use generalizability theory to estimate relative error variance and reliability in teaching observation measures. They also use it to plan future studies and design the best possible measurement procedures. However, designing the best possible measurement procedure comes at a cost, and researchers must stay within their budget when designing a study. In this study, we applied the LaGrange multiplier method to obtain facet sample size equations that minimize relative error variance (hence maximize reliability) under budget constraints. We did this for a crossed design and three nested designs that are more typical of data collection in teaching observation studies. Using an example budget and variance components similar to those found in practice, we demonstrate the use of these equations. We also show the way variance components from fully crossed designs can be combined to use our equations for a nested design.

A Simple Model to Determine the Efficient Duration of Exams

“…Several authors have studied how one can optimize generalizability coefficients, reliability coefficients, or validity coefficients, or how one can minimize decision error rates. In many cases this was studied in the context of a budget constraint (e.g., Allison et al, 1997;Ellis, 2013;Liu, 2003;Marcoulides, 1993Marcoulides, , 1995Marcoulides, , 1997Marcoulides & Goldstein, 1990a, 1990b, 1992Meyer et al, 2013;Sanders, 1992;Sanders et al, 1991) or fixed total testing time (e.g., Ebel, 1953;Hambleton, 1987;Horst, 1949Horst, , 1951. Alternatively, one may minimize the costs of measurement given constraints on the generalizability coefficient or error variance (e.g., Peng et al, 2012;Sanders et al, 1989), which is a similar problem.…”

How Many Students and Items Are Optimal for Teaching Level Evaluation of College Teachers? Evidence from Generalizability Theory and Lagrange Multiplier

“…At present, the focus of higher education assessment turns into the assessment of students' learning outcomes (Le and Xin, 2015) [2]. In the evaluation of teaching level of college teachers, the "students' evaluation of teaching" has become an important part of the evaluation of teaching qualities in most colleges (Le and Xin, 2015;Meyer et al, 2014;Wolbing and Riordan, 2016) [2][3][4]. Usually, students evaluate the teacher on several indicator dimensions using the teaching level evaluation questionnaires assigned by colleges, and the average score of the evaluation is used to express the teaching level of college teachers.…”

“…In general, the reliability of the evaluation will also increase or decrease as the number of the students who assess their teachers increases or decreases (Lakes, 2013) [7]. However, it is necessary to consider the budget and cost in the process of evaluating the teaching level of college teachers when conducting survey research (Hill et al, 2012;Meyer et al, 2014) [3,8]. In fact, the researchers should consider how to design a measurement program with relatively high feasibility and reliability under budget constraints in designing research procedures Marcoulides, 1991, 1991) [9,10], because the cost will also be higher when the number of students evaluating teachers is larger.…”

“…Meyer et al (2014) [3] applied the Lagrange multiplier (estimating the optimal sample size of different designs in generalizability theory under budget constraints proposed by Macrolides et al (1993Macrolides et al ( , 1997 [24,25] to actual measurement research through an empirical study of the teaching level evaluation of college teachers. Meyer and other people list the application of the Lagrange multiplier in the simple crossover design and present the application of the Lagrange multiplier in the nested design, showing that the Lagrange multiplier has a wide applicability in the generalizability theory.…”

A Short Note on Optimizing Cost-Generalizability via a Machine-Learning Approach

“…Woodward and Joe (1973) derived equations for their constrained optimization; Saunders et al (1989) deployed discrete optimization, which was further updated by Saunders (1992) with the Cauchy–Schwartz inequity approach; Marcoulides (1993, 1995) as well as Marcoulides and Goldstein (1990, 1991, 1992) had developed the LaGrange multiplier approach and other related variants to handle the optimization within both univariate and multivariate G-theory. Meyer et al (2014) extended the LaGrange multiplier approaches to G-theory with nested designs. Devising these approaches to a particular G-theory design requires mathematical deriving procedures, which may be a challenge to many applied researchers.…”