Evaluating the Quality of Literature Reviews in the Social Sciences: Developing a Measure of Quality with an Illustration

Schumm, Walter R.; Crawford, Duane W.

doi:10.22606/pra.2019.12003

Cited by 5 publications

(4 citation statements)

References 138 publications

(21 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Methodological quality was measured by the sum of scores awarded for selected study characteristics (Schumm & Crawford, 2019a). Studies that involved comparison groups, random samples, or larger sample sizes were awarded two points for each of those characteristics; one point was awarded to each study if the study explained the ages of the children clearly (minimum, average, maximum), explained parent gender clearly, explained child gender clearly, used child reports about their own sexual orientation, explained all demographic variables clearly (i.e., broke down sexual orientation as a function of the combination of child's gender and age as well as parent's gender), or included more than one sample within the same article, for a maximum score of 12 quality points.…”

Section: Methodsmentioning

confidence: 99%

Can Social Desirability Be Detected Statistically at the Macro-Sociological Level?

Schumm¹,

Crawford²

2021

PRA

Self Cite

View full text Add to dashboard Cite

While a few have argued that social science has been subject to progressive biases, others have discounted such ideas. However, no one has yet performed empirical tests over a large range of studies for such possible bias, which we label macro-level social desirability (MLSD). Combining the results from fifty-nine empirical studies that assessed rates of nonheterosexuality among children of same-sex parents, we found that the higher the maximum rates reported, the less likely those reports were to have been cited in Google Scholar by counts or by annual rate, which may reflect MLSD. However, after several statistical controls, the association for counts became non-significant, while the association for rates became stronger, although the effect sizes were in a moderate (d = .28 or higher) to large range (d, up to .68) by either analysis. Generally, research quality acted as a suppressor variable for MLSD but was significantly related to both counts and rates of citations, indicating that higher quality articles were more likely to have been cited, even controlling for the number of years since first publication. Higher quality articles were slightly more likely to report higher rates of nonheterosexuality among children of same-sex parents. We discuss implications of our findings and suggest future directions of research.

show abstract

Section: Methodsmentioning

confidence: 99%

Can Social Desirability Be Detected Statistically at the Macro-Sociological Level?

Schumm¹,

Crawford²

2021

PRA

Self Cite

View full text Add to dashboard Cite

show abstract

“…One detectable pattern is the omission of reported scientific findings, especially in literature reviews, that don’t fit the desired narrative (Regnerus 2020; Schumm and Crawford 2019a, 2019b). For example, most of the time scientists look for statistically significant findings and are eager to report them.…”

Section: Confirmation Bias In Social Sciencementioning

confidence: 99%

Difficulties With Methodology in Social Science Research With Controversial Issues Regarding Human Sexuality

Schumm

Crawford

2023

Linacre Q

Self Cite

View full text Add to dashboard Cite

Social science is commonly used in debates about controversial issues, especially for those concerning human sexuality. However, caution must be exercised in interpreting such social science literature, because of a variety of methodological and theoretical weaknesses that are not uncommon. Families are complex structurally and over time; such data are not easily analyzed. Merely determining the number of, for example, sexual minority families has been a difficult task. While some new theories are popular with social scientists, for example, sexual minority theory, they are often used to the exclusion of other, equally valid theories and often are not well tested empirically. Some types of families remain relatively unexamined. Social scientists can be biased by their own values, which are reflected in weak use of theory and in a variety of methodological problems. Eight studies are presented as examples of probable confirmation bias, in which methods and theory were modified in unusual ways that may have affected the outcomes and conclusions. Suggestions for improving social science include greater attention to effect sizes rather than statistical significance per se, deliberately minimizing the politicization of science, developing a culture of humility with respect to social science, deliberately reducing common biases, and maintaining a deeper curiosity about social science than is often seen. Scientists must be open to seeing their best “sacred cow” ideas or theories disproven or modified with increases in research on such issues. Summary In controversial areas of social science, there can be numerous threats to the validity of science. Here, some of the more common risks for social science research and theory are examined, with several specific illustrations of how bias appears to have crept into social science, often as confirmation bias. Recommendations are made for reducing bias in future research.

show abstract

“…Copy@ Walter R Schumm bias literature reviews, lower their quality [11] and create a false scientific consensus [12]. Publication bias may exacerbate the problem, if positive manuscripts are more likely to be published as journal articles than negative manuscripts.…”

Section: American Journal Of Biomedical Science and Researchmentioning

confidence: 99%

Assessing Citation Bias in Scientific Literature

Schumm¹

2020

AJBSR

View full text Add to dashboard Cite

Our objective was to provide a further test of possible citation bias or confirmation bias in science based on differential outcomes of two articles published between 1979 and 2013. The two articles used here formed a natural experiment because the articles were from the same author and concerned similar topics except one article had more positive results for its participants. Based on Google Scholar citations, citation rates were compared using binomial tests, including normal approximation z scores, and one-sample chi-square tests. Between the two articles, the one that presented more favorable findings was cited far more often than the one with less positive results, providing empirical support for the existence of citation or confirmation bias, which may result in lower quality and less comprehensive literature reviews.

show abstract

Evaluating the Quality of Literature Reviews in the Social Sciences: Developing a Measure of Quality with an Illustration

Cited by 5 publications

References 138 publications

Can Social Desirability Be Detected Statistically at the Macro-Sociological Level?

Can Social Desirability Be Detected Statistically at the Macro-Sociological Level?

Difficulties With Methodology in Social Science Research With Controversial Issues Regarding Human Sexuality

Assessing Citation Bias in Scientific Literature

Contact Info

Product

Resources

About