Coefficient Alpha: An Engineer's Interpretation of Test Reliability

Abstract

Reliability is a fundamental concept of test construction. The most common measure of reliability, coefficient alpha, is frequently used without an understanding of its behavior. This article contributes to the understanding of test reliability by demonstrating that questions which lower reliability are inconsistent with the bulk of the test, being prone to test-taking tricks and guessing. These qualitative characteristics, obtained from focus groups, provide possible causes of lower reliability such as poorly written questions (e.g., the correct answer looks different from the incorrect answers), questions where students must guess (e.g., the topic is too advanced), and questions where recalling a definition is crucial. Quantitative findings confirm that questions lower reliability when students who answer correctly have lower overall scores than students who answer incorrectly. This phenomenon is quantified by the "gap" between these students' overall scores, which is shown to be highly correlated with other item metrics. An increasing number of concept inventory tests are being developed to assess student learning in engineering. Scores and student comments from the Statistics Concept Inventory are used to make these judgments.

Investigating the impact of pedagogical agent gender matching and learner choice on learning outcomes and perceptions

“…The three identified factors related to 1) evaluations of the program or content matter (eight items, such as "I would recommend this program to other students" and "I would like to learn more about electrical circuits", with factor loadings ranging from .47 to .77), 2) evaluations of the graphics used in the program (four items, such as "The graphics made the lesson easier to understand" and "The graphics in the program helped me to learn", with factor loadings ranging from .55 to .75), and 3) difficulty ratings (six items, such as "The lesson was difficult" and "The topics that were covered in the lesson were difficult", with factor loadings ranging from .41 to .90). As measured by Cronbach's alpha (Allen, Reed-Rhoads, Terry, Murphy, & Stone, 2008), the internal reliability of the program rating scale was .91, internal reliability of the graphics rating scale was .86, and internal reliability of the difficulty ratings was .89. A program ratings score, a graphics ratings score, and a perceived difficulty score were computed by averaging the ratings from the respective questions which loaded on these factors.…”

Technological Literacy Learning With Cumulative and Stepwise Integration of Equations Into Electrical Circuit Diagrams

“…How large is the current flow?" The pretest was designed to measure the participant's domain-specific prior knowledge before entering the instructional session and had an internal reliability as measured by Cronbach's of .68 [39].…”

Analyzing an Abbreviated Dynamics Concept Inventory and Its Role as an Instrument for Assessing Emergent Learning Pedagogies

“…The increase in alpha from the pre-to post-test is indicative of the class as a whole answering the questions more consistently and a likely artifact of reduced instances of guessing on the post-test 21 . The post-test alpha of 0.607 is lower than those published by Gray et al (0.640-0.837) and the team led by Jorion (0.74) for the full 29-question DCI 3,12 .…”

“…The post-test alpha of 0.607 is lower than those published by Gray et al (0.640-0.837) and the team led by Jorion (0.74) for the full 29-question DCI 3,12 . However, Cronbach's alpha is directly dependent on the number of questions on a given test, so it is difficult to determine if the difference in alpha between the aDCI and the DCI is mostly due to the difference in exam length or other causes 21 .…”