Developing an Observation System to Capture Instructional Differences in Engineering Classrooms

Abstract

In 1999, bioengineering and learning science faculties at four research universities began collaboration on a National Science Foundation-supported Engineering Research Center. The Vanderbilt-Northwestern-Texas-Harvard/MIT Engineering Research Center for Bioengineering Education and Technology is the first such Center to have a specific focus on postsecondary education. Within this project, educators, learning scientists, and bioengineers collaborated to develop the VaNTH Observation System, an assessment tool to capture quantitatively teaching and learning experiences of the bioengineering classroom. The four components of the instrument address the professor's interaction with students, students' lesson engagement, narrative notes, and research-based measures of effective teaching. As professors redesigned lessons to incorporate current learning theory as a guide, observers measured differences in bioengineering classroom experiences resulting from these innovations. Results indicate that the observation instrument captures differences in classroom experiences, and these differences relate to a lesson's incorporation of current learning theory. I. INTRODUCTIONFor the past three years, the bioengineering and learning sciences faculties at Vanderbilt University, Northwestern University, the University of Texas at Austin, and the Health Sciences and Technology Program of Harvard and Massachusetts Institute of Technology have collaborated in an NSF-supported Center for Research in Bioengineering Educational Technologies. The mission of this Center, named VaNTH for the collaborating institutions, is to "…integrate learning science, learning technologies, and the domains of bioengineering in order to develop effective educational resources to prepare for the future of bioengineering" [1]. The research efforts of the Center involve four thrusts: learning science, assessment and evaluation, learning technology, and bioengineering domains. Of particular interest within the assessment and evaluation group has been the development of an observation instrument to capture instructional differences in bioengineering classrooms.As bioengineers and learning scientists began working together, they determined to draw on and incorporate into bioengineering classrooms the most current, research-based knowledge on effective teaching and learning. Much of this knowledge is laid out in How People Learn: Brain, Mind, Experience, and School [2]. This monograph of the National Research Council explains the "How People Learn" (HPL) theory as applied to mainly K-12 classrooms. At the heart of HPL theory are four types of "centeredness" that occur both singularly and concurrently in effective classrooms: elements of a lesson may be knowledge-centered, learner-centered, assessment-centered, and/or community-centered. Knowledge-centered elements focus on information and procedures that constitute a domain of knowledge. Learner-centered elements cause students to make linkages to prior knowledge and experience. Assessmentcentered elem...

High-Quality Learning Environments for Engineering Design: Using Tablet PCs and Guidelines from Research on How People Learn

“…The VaNTH Observation System (VOS) was used to systematically assess HPL framework implementation in EI-100 classrooms (Harris & Cox, 2003;Gazca, Palou, López-Malo, & Garibay, 2009a) . VOS is an assessment tool developed to capture qualitative and quantitative classroom observation data from teaching and learning experiences in the bioengineering classroom.…”

Research and Teaching: A New Tool for Measuring Student Behavioral Engagement in Large University Classes

“…There are a handful of published classroom observation instruments that have been developed for postsecondary education settings with large classes: the Reformed Teaching Observation Protocol (RTOP; Piburn et al, 2000), VaNTH Observation System (VOS; Harris & Cox, 2003), STROBE (O'Malley et al, 2003), and the Classroom Observation Protocol for Undergraduate STEM (COPUS; Smith, Jones, Gilbert, & Wieman, 2013). RTOP does not explicitly deal with student engagement and is more focused on the teachers' implementation of the lesson plan.…”

Challenge-based instruction in biomedical engineering: A scalable method to increase the efficiency and effectiveness of teaching and learning in biomedical engineering

“…Harris and Cox [29] have developed a method-the VaNTH observation system (VOS)-based on a hand-held personal computing device that can be used to make direct observations of classroom behavior. The VOS captures three fundamental aspects of teaching and learning: (1) teacherstudent interactions in terms of who-to-whom-did what-howwith what-media; (2) a "snapshot" of student engagement in a variety of sanctioned and unsanctioned activities (e.g., with the professor, with other students, or working alone); (3) the context of the class, captured in a section of the VOS called "Narrative Notes".…”

“…However, when the learning objectives focus on students' ability to solve complex problems, then we need a performance assessment that requires students to be more generative in their solution. Table 2 details a range of assessment and evaluation measures that have been used to test the general hypotheses of VaNTH relative to the HPL frameworks and more specific studies of students' cognition [33][34][35]29]. …”