The Pragmatist in Context of a National Science Foundation Supported Grant Program Evaluation: Guidelines and Paradigms

Abstract

Background: The philosophical underpinnings of evaluation guidelines set forth by a funding agency can sometimes seem inconsistent with that of the intervention. Purpose: Our purpose is to introduce questions pertaining to the contrast between the instructional program’s underlying philosophical beliefs and assumptions and those underlying our evaluation approach. Drawing heavily on Scriven, we discuss these from a pragmatist evaluation stance in light of issues defined by Lincoln and Guba (2000). The discussion is couched in the evaluation of an innovative approach to teaching computer science. Setting: Auburn University, Auburn, AL Intervention: The evaluation is designed to investigate the effects of a studio-based teaching approach in computer science education. The evaluation framework employs a rigorous design that seeks to provide evidence to support or refute some assumed truth about the object (or construct) investigated. The program evaluated is steeped in a constructivist framework which assumes that no universal truth or reality exists, but rather, is constructed by the individual. Research Design: Our evaluation design, to a good extent, reflects a post-positivist, quasi-experimental position. We also include a qualitative component using student interviews. Data Collection and Analysis: Evidence of the effectiveness of the instructional approach for learning is assessed quantitatively using pre- and post-test and pre- and post-survey data group comparisons (mixed design ANOVA). Interviews provide the basis for qualitative theme analysis. Findings: Quantitative results were somewhat weak but consistent in support of the studio-based teaching. Interview data suggest that most students did find working in groups enjoyable and a valuable experience.