Special Considerations When Using Statistical Analysis in Engineering Education Assessment and Evaluation

Abstract

Two special considerations are discussed which frequently arise when conducting statistical analyses for the assessment and evaluation of engineering education programs. The first concerns the multiple comparison problem and Type I errors, specifically when should the significance level be adjusted and which adjustment procedure is most appropriate? Three scenarios are presented to illustrate three different applications of the classical Bonferroni procedure, one of the most extensively used adjustment procedures. A scenario is also presented for when an adjustment is not necessary. The second consideration is: when evaluating a predictive model should a tree diagram be used as an alternative to a classification table? For example, how does one assess a model's predictions when certain of its "recommendations" are not followed? For this type of case, a classification table may yield incomplete information. The use of a tree diagram to present more information on model performance is discussed. I. INTRODUCTIONEngineering education has witnessed an explosion of noteworthy research during the past decade spurred on by two converging initiatives. The first is the current evolution of curricula in response to the changes in the engineering accreditation criteria by the Accreditation Board for Engineering and Technology (ABET). The second is the National Science Foundation's (NSF) bold move to fund the Engineering Education Coalitions in concert with a series of other NSF educational initiatives that have generated a number of the ideas currently being implemented in response to the new ABET criteria. In addition, the former has highlighted the critical need for assessment within the engineering education community, while the latter has triggered a call for wide-scale dissemination and implementation of many of these NSF sponsored educational innovations. Indeed, research in engineering education is becoming a "respected" endeavor as more educators and administrators are using such results to make educational and policy improvements.That said, as educators turned to this type of research they must also become more cognizant of the statistical ramifications of their results. This paper presents two special considerations that frequently arise when statistical analyses are employed as part of the assessment and evaluation of engineering education programs and courses. It has been our experience that researchers often overlook these considerations, and consequently may draw inaccurate conclusions at worst or faulty analyses at best. Although these two issues are separate in nature, they both deal with reaching potentially erroneous conclusions with regards to Type I and II errors.The first is associated with the multiple comparison problem, specifically adjusting the Type I error level (i.e., the probability of rejecting the null hypothesis, when, in fact, it is true) and determining which adjustment procedure is appropriate for a particular multiple comparison situation. That is, when should the significance leve...

Predicting STEM Degree Outcomes Based on Eighth Grade Data and Standard Test Scores

“…There were also significant differences in attitudes among majority students and their African-American, Asian Pacific, and Hispanic counterparts. Larpkiattaworn et al (2003) examined issues associated with employing statistical analysis to assess and evaluate engineering education initiatives. Their article presented several techniques for managing Type I errors when conducting multiple comparisons and provided guidance in selecting from among the techniques, e.g., the classic Bonferroni method, Scheffe's procedure, Tukey's test, and the sequentially rejective Bonferroni procedure (SRBP).…”

The Effect of College Cost and Financial Aid on Access to Engineering

“…Since we conduct multiple statistical tests, we use an adjusted alpha level for each test to keep the experimentwise error rate at .05 level. Calculated from Bonferroni adjustment 31 , the adjusted alpha level for each test is .004 (≈ .05/12). Table 2 shows the rising paces of out-of-state tuition and fees, in-state tuition and fees, and grant aid per student in both nominal value and constant dollar.…”

The Effects of Merit-based Scholarships on First-year Engineering Student Characteristics and Academic Behaviors

“…Conducting multiple statistical tests, this study uses an adjusted alpha level to keep the experimentwise error rate at .05 level. The adjusted alpha level for each test is .001 based on calculation from the conservative Bonferroni adjustment 34 .…”