Meta-Analysis in Higher Education: An Illustrative Example Using Hierarchical Linear Modeling

Abstract

The purpose of this article is to provide higher education researchers with an illustrative example of meta-analysis utilizing hierarchical linear modeling (HLM). This article demonstrates the step-by-step process of meta-analysis using a recently-published study examining the effects of curricular and co-curricular diversity activities on racial bias in college students as an example (Denson, Rev Educ Res 79:805-838, 2009). The authors present an overview of the meta-analytic approach and describe a meta-analysis from beginning to end. The example includes: problem specification; research questions; study retrieval and selection; coding procedure; calculating effect sizes; visual displays and summary statistics; conducting HLM analyses; and sensitivity analyses. The authors also offer guidelines and recommendations for improving the conduct and reporting of research which in turn can provide the information necessary for future and more comprehensive meta-analytic reviews.Keywords Meta-analysis Á Effect size Á Random-effects model Á Hierarchical linear modeling Á Moderator analysis While meta-analytic techniques have been available for over a century (see Cooper and Hedges (1994) for a history of meta-analysis), meta-analysis as a method of inquiry for integrating quantitative results from a stream of research began to become popular approximately 30 years ago (Glass 2000). While conducting a meta-analysis can be very time consuming when done properly, advances in computing and statistical software have facilitated conducting literature searches and applying meta-analytic procedures. The purpose of this article is to provide higher education researchers with a pedagogical example of a meta-analysis utilizing hierarchical linear modeling (HLM) 123Res High Educ (2011( ) 52:215-244 DOI 10.1007 more widespread use of meta-analytic methods. In this article, we begin by first providing an overview of the meta-analytic approach. We then provide an example of meta-analysis using Denson's Review of Educational Research (2009) study of the effects of curricular and co-curricular diversity activities on racial bias in college students. Finally, we provide guidelines and recommendations for improving the conduct and reporting of future research. Overview of the Meta-Analytic ApproachPut simply, meta-analysis is a statistical technique of combining the findings of a set of studies that address common research hypotheses (Cooper and Hedges 1994;Lipsey and Wilson 2001). As Glass (1976) defines it, ''meta-analysis refers to the analysis of analyses…the statistical analysis of a large collection of analyses results from individual studies for the purpose of integrating the findings'' (p. 3). Conducting a meta-analysis is much like conducting a primary research study. In a meta-analysis, instead of students or participants being the unit of analysis, the primary studies themselves become the unit of analysis. In meta-analysis, the researcher collects research studies from a particular domain (e.g., studies of the ef...

A Meta-Analysis of Electronic Word-of-Mouth Elasticity

“…Second, we estimate the impact of the aforementioned factors on eWOM volume and valence elasticities. In the context of quantitative meta-analysis, data have a nested or hierarchical structure (i.e., subjects nested within studies; Denson and Seltzer 2011), making traditional regression analyses such as ordinary least squares (OLS) inappropriate because nested data structures may lead to heteroskedasticity in the errors (Krasnikov and Jayachandran 2008). Thus, to account for within-study error correlations between eWOM elasticities, we perform the meta-analysis with hierarchical linear modeling (HLM), 5 as Bijmolt and Pieters (2001) suggest.…”

Meta-Analysis of the Influence of Instructional Media on Speaking Learning

“…They learn about the role of speaking in developing knowledge. The effect size, which shows the magnitude of the effect of a treatment or the strength of the relationship between two variables, is an essential component in the meta-analysis because it provides information from the summary results (Denson & Seltzer, 2011). By determining the effect sizes of each research, the average overall effect sizes can be determined (Oktradiksa & Fitriansyah, 2017).…”

Contemporary applications of moderation analysis in counseling psychology.

“…Researchers are typically encouraged to report measures of effect size to aid in interpretation of results and understanding of practical significance of findings (Kelley & Preacher, 2012). Reporting effect size can also be useful for other researchers conducting power analysis or meta-analysis on the given topic (Denson & Seltzer, 2011; Kelley & Preacher, 2012). Furthermore, because of this, many educational and psychological journals are beginning to require the reporting of effect size, in addition to reducing the emphasis on p values and null hypothesis significance testing (Kelley & Preacher, 2012).…”