An observation tool for instructor and student behaviors to measure in-class learner engagement: a validation study

Abstract

BackgroundEfforts are made to enhance in-class learner engagement because it stimulates and enhances learning. However, it is not easy to quantify learner engagement. This study aimed to develop and validate an observation tool for instructor and student behaviors to determine and compare in-class learner engagement levels in four different class types delivered by the same instructor.MethodsObserver pairs observed instructor and student behaviors during lectures in large class (LLC, n=2) with third-year medical students, lectures in small class (LSC, n=6) and case-based teaching sessions (CBT, n=4) with fifth-year students, and problem-based learning (PBL) sessions (~7 hours) with second-year students. The observation tool was a revised form of STROBE, an instrument for recording behaviors of an instructor and four randomly selected students as snapshots for 5-min cycles. Instructor and student behaviors were scored 1–5 on this tool named ‘in-class engagement measure (IEM)’. The IEM scores were parallel to the degree of behavior's contribution to active student engagement, so higher scores were associated with more in-class learner engagement. Additionally, the number of questions asked by the instructor and students were recorded. A total of 203 5-min observations were performed (LLC 20, LSC 85, CBT 50, and PBL 48).ResultsInterobserver agreement on instructor and student behaviors was 93.7% (κ=0.87) and 80.6% (κ=0.71), respectively. Higher median IEM scores were found in student-centered and problem-oriented methods such as CBT and PBL. A moderate correlation was found between instructor and student behaviors (r=0.689).ConclusionsThis study provides some evidence for validity of the IEM scores as a measure of student engagement in different class types.

The effectiveness of TBL with real patients in neurology education in terms of knowledge retention, in-class engagement, and learner reactions

“…This is a written form for observing and recording behaviors of the instructor and four randomly selected students as snapshots for 5-min cycles in classes. The in-class engagement measure (IEM) was created on the basis of a previously developed observation tool called STROBE (18) and validated in a study conducted among Turkish medical students (1). Instructor and student behaviors were scored 1 to 5 on this tool.…”

“…Additionally, the number of questions asked by the instructor and students was recorded. A sample of the IEM is provided in online supplementary materials (1).…”

Peer-led, transformative learning approaches increase classroom engagement in care self-management classes during inpatient rehabilitation of individuals with spinal cord injury

“…However, to identify constructs of positive engagement, we consulted with nurse educators about the behaviors they consider indicative of active engagement in class. The criteria they identified as 'positive' were similar to the highest rated student behaviors on the In-class Engagement Measure (IEM), 38 an observation tool to record details of the learning environment and student behaviors. Moreover, direct observation of student behavior, with adequate checks on inter-observer reliability (as employed here), is accepted practice in studies of classroom behavior.…”

Competence and training in interventional pulmonology

“…21 in Problem Based Learning (PBL) the teacher presents a problem to the class and acts as a facilitator or mentor, rather than as the provider of solutions. 36 in medicine, these exercises help learners gain cognitive, technical and experiential skills in a patient-and learner-centric environment. One such model, the "Practical approach (Pa) to interventional Bronchoscopy", is based on a four-box format which has been developed to complement traditional learning (Table iV).…”