Meta-Analysis, Replication, Social Skills, and Learning Disabilities

Abstract

This article examines Ochoa and Olivarez's (1995)replication study of Swanson and Malone's (1992) meta-analysis of sociometric research for children with learning disabilities. The two research syntheses agree in direction and outcomes, but vary substantially in identifying the moderator variables that underlie effect sizes. Some of the differences between the two syntheses were related to the effects of gender, ethnicity, and type of measurement on effect size. Differences between the two syntheses were attributed to (a) inadequate reporting of coding reliability, (b) failure to include similar articles for analysis, and (c) poor operationalization of the term learning disabilities as the basis for article selection. Because differences existed in coding schemes and effect sizes between the two syntheses, guidelines for enhancing replication for future syntheses are suggested. These guidelines reflect several criteria on which to judge a meta-analysis of the literature. Syntheses of research-especially meta-analyses-on children with learning disabilities (LD) appear quite frequently in the literature. Unfortunately, there have been very few criteria to judge the quality of these reports. In addition, there have been few if any replications, to my knowledge, on the findings of these syntheses. Recently, Ochoa and Olivarez (1995) conducted a replication of Swanson and Malone's (1992) meta-analysis of sociometric research on students with learning disabilities. Earlier, Swanson and Malone (1992) had found that students with learning disabilities had considerably lower sociometric status than their nondisabled counterparts. Ochoa and Olivarez's replication supported Swanson and Malone's findings in terms of the direction and magnitude of the effects. Ochoa and Olivarez argued, however, that their study was more accurate in terms of (a) effect sizes (ES's), (b) conceptual comparability of dependent measures, and (c) internal validity. In sum, they argued that (a) their ES's were smaller and took into consideration sample size and outliers, (b) they correctly classified measures, and (c) they placed parameters on the quality of the studies included in the synthesis. Because of a failure to replicate our findings completely, our natural response was to elaborate and qualify our synthesis. Much to our surprise, however, we found the meta-analysis by Ochoa and Olivarez more thorough and methodologically sophisticated than ours. We have some quibbles with Ochoa and Olivarez (1995), however, such as their failure to (a) report interrater agreement on the various codes, (b) define what is meant by a "methodologically sound study," and (c) compare conclusions on various categories on the same articles (the percentage of overlap in articles

Characteristics of Meta-Analyses Reported in Mental Retardation, Learning Disabilities, and Emotional and Behavioral Disorders

“…For example, Dunn (1990) debated Forness (1987, 1990) over meta-analytic results concerning modality-based instruction, and also Dunn et al (1995) debated Kavale et al (1998) about learning styles. A similar debate can be found between Swanson and Malone (1992), Swanson (1996), andOchoa andOlivarez (1995).…”

“…For example, complete references of studies in the meta-analysis or a detailed listing and description of all coded variables is quite feasible when the pool of primary studies or variables and subvariables is small (e.g., Soto et al, 1994, contained 7 studies, five independent variables, and no subvariables), but it is logistically unmanageable when the pool is large (e.g., Didden et al, 1997, contained 482 studies, two variables, and many subvariables). In contrast, 1 study in this review (O'Shaughnessy & Swanson, 1998) provided all of the criteria in all six domains, whereas 2 other studies (Durlak, Fuhrman, & Lampman, 1991;Swanson et al, 1996) provided most of the criteria (.93). Of the studies reviewed here and in Mostert (1996), these 3 provide the best model for publishing relevant information for meta-analytic findings.…”

“…Sixth, meta-analysts are almost always dependent on the information provided in the primary studies-information that itself may be incomplete or confusing, thereby circumscribing the scope and detail of the meta-analysis. For example, Swanson et al (1996), Swanson and Hoskyn (1998), and Xin and Jitendra (1999) reported excluding primary studies that did not provide enough information for meta-analytic calculations.…”

“…This is an impression that can be challenged on the grounds that (a) meta-analytic results rely heavily on how the independent variables from the primary studies are defined, related, and coded (Abrami, Cohen, & d'Appolonia, 1988;Bangert-Drowns, 1986); (b) the meta-analytic information provided is often too sparse for readers to make reasonable judgments regarding the face validity of the meta-analysis (Cooper & Lindsay, 1998); and (c) some evidence suggests that meta-analyses conducted on the same body of primary studies can yield different results (see Dunn, 1990;Dunn, Griggs, Olson, Beasley, & Gorman, 1995;Kavale & Forness, 1987Kavale, Hirshoren, & Forness, 1998;Ochoa & Olivarez, 1995;Swanson, 1996;Swanson & Malone, 1992). Therefore, to avoid these limitations to the validity of any meta-analysis, the analyses should be carefully reported or risk being "described as a significant step backward in the art of research synthesis" (Slavin, 1986, p. 8).…”

Reciprocal Interaction

“…The least socially competent students with learning disabilities generally are those individuals in the nonverbal learning disability (NLD) category (Dimitrovsky et al, 1998;Foss, 1991;Rourke, 1989;Swanson, 1996;Telzrow & Bonar, 2002). McDonald and tone in themselves and in others (Lavoie, 1994).…”