Same, Similar, or Something Completely Different? Calibrating Student Surveys and Classroom Observations of Teaching Quality Onto a Common Metric

Abstract

Using item response theory, this study explores whether student survey and classroom observation items can be calibrated onto a common metric of teaching quality. The data comprises 269 lessons of 141 teachers that were scored on the International Comparative Analysis of Learning and Teaching (ICALT) observation instrument and the My Teacher student survey. Using Rasch model concurrent calibration, items from both instruments were calibrated onto a common one‐dimensional metric of teaching quality. Most items were found to fit the model. Challenges pertain mainly to items measuring teaching students learning strategies and differentiation. Explanations for these difficulties are discussed.

Students’ perceptions of teaching behaviour in Turkish secondary education: a Mokken Scaling of My Teacher Questionnaire

“…When the MS polytomous DMM was applied, the resulting MTQ10 showed strong psychometric characteristics, internal consistency and construct validity. Its unidimensional structure is consistent with previous findings and the original version of MTQ (Maulana et al 2015a(Maulana et al , 2019van der Lans et al 2019) and is consistent across groups (random samples, gender, subject, grade level). MTQ10 met all the MS polytomous DMM assumptions.…”

“…Second, the Graded Response Model (GRM), an extended IRT model for ordered polytomous observed variables, was applied to understand the response behaviours and how the set of items performed (Samejima 1968;Perner and Imiya 2005). The MTQ items were visualised using R package 'psychotree' to explore the unidimensionality assumption further (Maulana et al 2015a(Maulana et al , 2019van der Lans et al 2019). The total information estimated by this model indicated the presence of a nonnormal distribution with the highest frequency towards the maximum scores (3-4) in the data (Fig.…”

“…"My teacher teaches me to check my solutions", α = 0.85). Prior studies have shown that the items of the MTQ (Maulana et al 2016c;van de Grift et al 2014b) can be ordered in a unidimensional structure (Maulana et al 2015a(Maulana et al , 2019van der Lans et al 2019) and is valid and reliable across countries (Maulana et al 2019).…”

“…Empirical studies show that the six domains of teaching behaviour follow a stagelike order on a unidimensional continuum (van de Grift et al 2014b;van der Lans et al 2018van der Lans et al , 2019. More-complex teaching behaviours require sufficient experience, practice and knowledge, even though a small number of novice teachers are cabaple of displaying highly-skilled teaching behaviours.…”

Teachers’ interpersonal relationships and instructional expertise: How are they related?

“…The ICALT model also summarizes teachers' classroom practices using six clusters, namely: safe and stimulating learning climate, efficient classroom management, clear and structured explanation, intensive and activating instruction, teaching learning strategies, and differentiation (Maulana et al, 2015;Van de Grift et al, 2014). In addition, the ICALT model orders these six clusters hierarchically on a single progressive continuum of instructional expertise (Van de Grift et al, 2014;van der Lans et al, 2015van der Lans et al, , 2019. This continuum starts with least the complex cluster of teaching behaviors, which are learned by novice teachers, and ends with the most complex cluster of teaching behaviors, which are only mastered by expert teachers.…”