Increased Accuracy in Measuring Leadership Impacts

Abstract

This study sought to determine the change in level of leadership skills of participants in a community leadership program and the degree of response shift (change in level of understanding of leadership skills) in the self-report measures employed. While most of the participants reported increases in their leadership skills at the end of the program, differences were found between their self-report ratings using the pretest/posttest approach and the then/posttest approach. The degree of response shift (pre/then comparison) was also evident. The findings from this study together with other studies cited suggest that when employing self-report measures, the then/post approach provides a less conservative and more accurate means of assessing leadership skill development than would the traditional pretest/posttest approach. Suggestions for reducing the effects of response shift are also offered.

Understanding Commuter Student Self-Efficacy for Leadership: A Within-Group Analysis

“…Cross-sectional designs with retrospective questions provide a stronger indication of student gains by reducing concerns associated with response-shift bias when measuring self-reported leadership outcomes (Howard, 1980;Rohs, 2002;Rohs & Langone, 1997). Furthermore, a study of self-and peer-reported leadership behaviors and the quality of those behaviors found self-reports of leadership to be generally accurate (Turrentine, 2001).…”

The Measurement of Socially Responsible Leadership: Considerations in Establishing Psychometric Rigor

“…The use of retrospective questioning allows for the creation of quasi-pretests affording some degree of control when examining change across the SRLS measures. Additionally, some research on leadership outcome measurement suggests the appropriateness of cross-sectional designs due to concerns related to response-shift bias (Dailey, 1979;Howard, 1980;Rohs & Langone, 1997;Howard & Rohs, 2002). These scholars argue that accuracy in longitudinal measurement is predicated on the assumption that:…”

Cultural Transferability of Socially Responsible Leadership: Findings From the United States and Mexico

“…Additionally, the model was adapted for use in cross-sectional research by asking participants to reflect retrospectively on their precollege knowledge. This approach is appropriate for use in the study of educational outcomes possessing cognitive attributes such as leadership capacity as it limits error associated with response shift bias (Howard, 1980;Howard & Dailey, 1979;Rohs, 1999Rohs, , 2002Rohs & Langone, 1997).…”

“…Cross-sectional designs are traditionally less ideal for use in college impact research, given results do not represent true causal relationships and potentially introduce biased assessments of institutional quality due to a lack of control measures (Astin & Lee, 2003). However, as explained in the introduction of the conceptual model, this research used retrospective questions to capture precollege data which scholars suggest is more appropriate when measuring dependent variables with cognitive dimensions like leadership development (Howard, 1980;Howard & Dailey, 1979;Rohs, 1999Rohs, , 2002Rohs & Langone, 1997). The use of selfreported data may also be of concern as it can vary in accuracy of measurement depending on the degree to which researchers ensure both methodological and psychometric quality (Gonyea, 2005).…”