Transparency and Replication in Cross-national Survey Research: Identification of Problems and Possible Solutions

Abstract

In this article, we examine whether cross-national studies disclose enough information for independent researchers to evaluate the validity and reliability of the findings (evaluation transparency) or to perform a direct replication (replicability transparency). The first contribution is theoretical. We develop a heuristic theoretical model including the actors, factors, and processes that influence the transparency of cross-national studies and provide an overview of the measures currently taken to improve research transparency. The second contribution is empirical, in which we analyze the level of transparency in published cross-national studies. Specifically, using a random sample of 305 comparative studies published in one of 29 peer-reviewed social sciences journals (from 1986 to 2016), we show that, even though all the articles include some methodological information, the great majority lack sufficient information for evaluation and replication. Lastly, we develop and propose a set of transparency guidelines tailored for reporting cross-national survey research.

Has the Credibility of the Social Sciences Been Credibly Destroyed? Reanalyzing the “Many Analysts, One Data Set” Project

“…The CSI has had a huge impact. It seems to be a standard reference when discussing the (low) credibility of social science research with observational data (see e.g., Damian, Meuleman, and van Oorschot 2019; Reed 2019; Young 2018). Many, especially in media articles and Internet discussions, even concluded that the social sciences may not be rigorous enough: Findings depend to a large extent on who did the research.…”

Gender Ideology in Europe: Plotting Normative Types in a Multidimensional Space

“…Obviously, we recommend researchers to include as much information as possible to be able to replicate their article. Damian et al (2019) suggest 12 issues that should be discussed in cross-national comparative social science articles in order to check for replicability transparency. Among these issues are information about the dataset used, but also aspects such as the number and treatment of missing values, weighting, the exact operationalization of variables, and as the most complete information, the syntaxes used.…”

Improving Knowledge Production in Comparative Survey Research: Cross-Using Data from Four International Survey Programmes

“…While many publications in the field of comparative survey research discuss issues of governance, quality control, transparency of documentation and procedures, questionnaire development, harmonisation, translation, and fieldwork management [Jowell 1998;Harkness 1999;Lynn 2003;Lyberg et al 2019;Hadler et al 2015;Pennell et al 2017;Lindstrøm and Kropp 2017], few studies empirically investigate the knowledge production side of comparative programmes [e.g. Damian et al 2019], and we found none that study combined data use. This theme is more commonly discussed in the medical literature, particularly in relation to systematic reviews, based on the entire population of relevant studies [Oliver et al 2005;Petticrew and Roberts 2006;Harden 2010].…”

“…Besides coverage issues, the validation of findings is another vital aspect of comparative survey data use to which the ecosystem notion can be applied. Methodological literature documents an almost endless number of equivalence issues in comparative research, and highlights how designing successful crossnational studies is an elaborate and methodologically complex process in which there is much more potential for biases and errors than there is in single-nation surveys [Lyberg et al 2019;Smith 2019;Lynn et al 2006;Damian et al 2019]. Ensuring measurement equivalence when there are multiple distributed actors and many social and cultural contexts remains a challenge [Briceno-Rosas et al 2020] and may not result in the desired level of quality, as the 'data from different countries were possibly collected using different methods, under different conditions, and in different cultural contexts' [Quandt and Luijkx 2015: 795].…”