Beyond Composite Scores and Cronbach’s Alpha: Advancing Methodological Rigor in Recreation Research

Gagnon, Ryan J.; Stone, Garrett A.; Garst, Barry A.

doi:10.18666/jorel-2017-v9-i4-7959

Cited by 7 publications

(5 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The point of this article is to provide an example from adventure education research that illustrates how more advanced methodologies and statistical analyses can provide a more complex assessment of change on a given variable. Gagnon et al (2017) state, "More simply, the analytic approaches necessary to tell the story of leisure and recreation research are a moving target, likely requiring a career-long commitment to methodological training on the part of researchers at conceptual and applied levels. Researchers should not only be able to "do" the appropriate analyses for the question they are asking, but also (and arguably more important) be prepared to translate that story to the community of recreation and leisure practitioners they are charged with serving" (p. 400).…”

Section: Discussionmentioning

confidence: 99%

The Benefits of Longitudinal Data and Multilevel Modeling to Measure Change in Adventure Education Research

Shirilla

Solid²,

Graham

2021

Journal of Experiential Education

View full text Add to dashboard Cite

Background: A common critique of adventure education research methodology is the overreliance on pre-/post-study designs to measure change. Purpose: This paper compares and contrasts two methods of data analysis on the same adventure education data set to show how these distinct approaches provide starkly different results and interpretation. Methodology/Approach: Using secondary data analysis, we employ a longitudinal data set of the social skill development of urban middle school students who participated in an adventure education program over the course of three academic years. First, change was assessed using a pre-/post-design by a traditional analysis of variance (ANOVA). Next, change was assessed with a six-wave, longitudinal design using multilevel modeling. Findings/Conclusions: Results show that the multilevel modeling approach revealed nonlinear change in the social skill development of middle school students, resulting in more accurate, nuanced estimation of change in social skill development than the ANOVA approach. Implications: The use of longitudinal data and multilevel modeling can be a useful methodological approach and statistical tool for adventure education researchers to not only address the criticisms of quantitative research in adventure education, but more importantly, provide a more thorough understanding of the impact of adventure education on human development.

show abstract

Section: Discussionmentioning

confidence: 99%

The Benefits of Longitudinal Data and Multilevel Modeling to Measure Change in Adventure Education Research

Shirilla

Solid²,

Graham

2021

Journal of Experiential Education

View full text Add to dashboard Cite

show abstract

“…Problematic item loadings (e.g., λ ≤ .4) were observed across several items that were reverse-coded. The rationale for reverse-coded items (e.g., variance improvement, maintaining respondent attention, reducing response bias) is beyond the scope of this study, but the primary outcome of reverse coding is nearly always measurement error (Gagnon, Stone, & Garst, 2017). With the removal of these items, the final CFA 44-item seven-factor model indicated close fit, S/B χ 2 (864) = 1,181.801, p ≤ .001, non-normed fit index [N-NFI] = .919, comparative fit index [CFI] = .926, root mean square error of approximation [RMSEA] = .045, 90% CI [.038, .051] (see Table 2).…”

Section: Methodsmentioning

confidence: 99%

Division-I Student-Athletes’ Affective and Cognitive Responses to Receiving Confirmation From Their Head Coach

Cranmer

Gagnon

Mazer

2019

Communication & Sport

Self Cite

View full text Add to dashboard Cite

Coach confirmation—a behavior that encompasses coaches’ recognition, endorsement, and acknowledgment of athletes—has been forwarded as an effective coaching behavior that redresses ineffective and antisocial patterns of coaching. Empirical evidence of its effectiveness, however, has been limited to athlete affect and based on data from retrospective samples of former high school athletes. This study addresses these limitations by exploring the effectiveness of coaches’ use of confirmation with collegiate student-athletes and considers its influence on their satisfaction with coaches and sport, competitiveness, and cognitive learning. Data obtained from 177 Division-I student-athletes revealed that coaches’ use of challenge increases student-athletes’ satisfaction, motivation, competitiveness, and learning, whereas the use of acceptance only determines satisfaction. These findings demonstrate that confirmation is an effective coaching behavior, its dimensions function independently, and challenge best fulfills the multidimensional nature of coaching effectiveness. The implications of this data reveal that confirmation theory functions uniquely within the athletic context, and coach effectiveness is largely determined by the implementation of strategies and efforts to refine skill sets and assist in development.

show abstract

“…However, the support for many current measures and the techniques underpinning their "validity" are somewhat suspect. Specifically, many measures of outcomes associated with the broader OST field reflect outdated techniques for justifying their validity (Gagnon, Stone, & Garst, 2017) and can result in a misalignment of what the scales purport to measure when compared with what they actually measure. For instance, a common approach is the utilization of composite scores for assessing camp-related outcomes; indeed, the author team has utilized similar techniques within OST and camp settings (Gagnon & Bumpus, 2016;Garst, Baughman, Franz, & Seidel, 2013).…”

Section: Journal Of Outdoor Recreation Education and Leadershipmentioning

confidence: 99%

“…For instance, a common approach is the utilization of composite scores for assessing camp-related outcomes; indeed, the author team has utilized similar techniques within OST and camp settings (Gagnon & Bumpus, 2016;Garst, Baughman, Franz, & Seidel, 2013). Further, escalating expectations for methodological rigor within the broader social sciences have highlighted limitations of composite-score-based validation approaches (Freire & Caldwell, 2013;Gagnon et al, 2017). The fundamental challenge embedded within composite-score approaches (i.e., summing the items composing a subscale and dividing this sum by the number of items to create a composite score) is the assumption that items composing the composite are measured error-free (Gagnon et al, 2017), whereas more contemporary approaches such as confirmatory factor analyses (CFA) do not assume perfect measurement (Brown, 2015).…”

Section: Journal Of Outdoor Recreation Education and Leadershipmentioning

confidence: 99%

“…Further, escalating expectations for methodological rigor within the broader social sciences have highlighted limitations of composite-score-based validation approaches (Freire & Caldwell, 2013;Gagnon et al, 2017). The fundamental challenge embedded within composite-score approaches (i.e., summing the items composing a subscale and dividing this sum by the number of items to create a composite score) is the assumption that items composing the composite are measured error-free (Gagnon et al, 2017), whereas more contemporary approaches such as confirmatory factor analyses (CFA) do not assume perfect measurement (Brown, 2015). Further, within the few camp-related studies using CFA, utilization of the corresponding findings is difficult if not impossible in the practical sense (e.g., for site-specific program evaluation) due to the level of training and resources necessary for conducting these analyses.…”

Section: Journal Of Outdoor Recreation Education and Leadershipmentioning

confidence: 99%

See 1 more Smart Citation

Research and Practice: An Applied Examination of the Parental Perceptions of Developmental Outcomes Scale

Gagnon

Garst

2019

JOREL

View full text Add to dashboard Cite

To support practitioners' need for reliable and valid measures of the youth camp experience, this study compares the results of two analytic approaches-(1) composite-based paired samples t tests and (2) a latent structural equation model approach-using the Parental Perceptions of Developmental Outcomes (PPDO) scale in a sample of 930 parents of youth attending residential summer camps. Through a retrospective pretest (RPT) design, parents reported statistically significant growth across the five dimensions of the PPDO (responsibility, exploration, self-regulation, attitude, and communication) in both analytic approaches. The study results indicated similar effect sizes and conclusions were reached using both analytical approaches, suggesting a more practitioner-friendly approach (composite) may tell the same story as a more complex and research-oriented (latent) approach. This finding, as well as the revalidation of the PPDO in this sample, indicates the PPDO's usefulness in applied and research contexts.

show abstract

Beyond Composite Scores and Cronbach’s Alpha: Advancing Methodological Rigor in Recreation Research

Cited by 7 publications

References 28 publications

The Benefits of Longitudinal Data and Multilevel Modeling to Measure Change in Adventure Education Research

The Benefits of Longitudinal Data and Multilevel Modeling to Measure Change in Adventure Education Research

Division-I Student-Athletes’ Affective and Cognitive Responses to Receiving Confirmation From Their Head Coach

Research and Practice: An Applied Examination of the Parental Perceptions of Developmental Outcomes Scale

Contact Info

Product

Resources

About