Abstract:Scientific advances across a range of disciplines hinge on the ability to make inferences about unobservable theoretical entities on the basis of empirical data patterns. Accurate inferences rely on both discovering valid, replicable data patterns and accurately interpreting those patterns in terms of their implications for theoretical constructs. The replication crisis in science has led to widespread efforts to improve the reliability of research findings, but comparatively little attention has been devoted … Show more
“…They found that no two teams followed the same data preprocessing steps or analysis strategies, resulting in substantial variability in their conclusions. This and other initiatives (9)(10)(11)(12)(13)(14)(15)(16)(17)(18) demonstrate that the multi-analyst approach is able to explore how research findings vary depending on the applied analytic strategy.…”
Section: Introductionmentioning
confidence: 74%
“…Before the expert consensus process, the first three authors and the last author (henceforth: proposers) created an initial multi-analyst guidance draft after brainstorming and reviewing all the previously published multi-analyst-type projects they were aware of (9)(10)(11)(12)(13)(14)(15)(16)(17)(18). This initial document is available here: https://osf.io/kv8jt/…”
Section: Creating the Initial Multi-analyst Guidance Draftmentioning
confidence: 99%
“…The proposers contacted 81 experts to join the project. The contacted experts included all the organisers of the previous known multi-analyst projects at the time (9)(10)(11)(12)(13)(14)(15)(16)(17)(18), as well as experts on research methodology. Of the 81 experts, 50 accepted the invitation and participated in the expert consensus procedure (their names are available at https://osf.io/fwqvp/).…”
Section: Recruiting Expertsmentioning
confidence: 99%
“…There are alternative solutions to prevent researchers from being biased by their data and results. For example, co-analysts could be provided with blinded datasets (14,16,28), simulated datasets (24), or with a subset of the data (e.g., 11).…”
Section: Preregistering the Processmentioning
confidence: 99%
“…The analytical robustness of results and conclusions can be studied by evaluating many distinct analysis options simultaneously (e.g., vibration of effects (7) or multiverse analysis ( 8)) or by involving multiple analysts who independently analyse the same data (9)(10)(11)(12)(13)(14)(15)(16)(17)(18). Rather than exhaustively evaluating all plausible analyses, the multi-analyst method examines analytical choices that are deemed most appropriate by independent analysts.…”
We present consensus-based guidance for conducting and documenting multi-analyst studies. We discuss why broader adoption of the multi-analyst approach will strengthen the robustness of results and conclusions in empirical sciences.
“…They found that no two teams followed the same data preprocessing steps or analysis strategies, resulting in substantial variability in their conclusions. This and other initiatives (9)(10)(11)(12)(13)(14)(15)(16)(17)(18) demonstrate that the multi-analyst approach is able to explore how research findings vary depending on the applied analytic strategy.…”
Section: Introductionmentioning
confidence: 74%
“…Before the expert consensus process, the first three authors and the last author (henceforth: proposers) created an initial multi-analyst guidance draft after brainstorming and reviewing all the previously published multi-analyst-type projects they were aware of (9)(10)(11)(12)(13)(14)(15)(16)(17)(18). This initial document is available here: https://osf.io/kv8jt/…”
Section: Creating the Initial Multi-analyst Guidance Draftmentioning
confidence: 99%
“…The proposers contacted 81 experts to join the project. The contacted experts included all the organisers of the previous known multi-analyst projects at the time (9)(10)(11)(12)(13)(14)(15)(16)(17)(18), as well as experts on research methodology. Of the 81 experts, 50 accepted the invitation and participated in the expert consensus procedure (their names are available at https://osf.io/fwqvp/).…”
Section: Recruiting Expertsmentioning
confidence: 99%
“…There are alternative solutions to prevent researchers from being biased by their data and results. For example, co-analysts could be provided with blinded datasets (14,16,28), simulated datasets (24), or with a subset of the data (e.g., 11).…”
Section: Preregistering the Processmentioning
confidence: 99%
“…The analytical robustness of results and conclusions can be studied by evaluating many distinct analysis options simultaneously (e.g., vibration of effects (7) or multiverse analysis ( 8)) or by involving multiple analysts who independently analyse the same data (9)(10)(11)(12)(13)(14)(15)(16)(17)(18). Rather than exhaustively evaluating all plausible analyses, the multi-analyst method examines analytical choices that are deemed most appropriate by independent analysts.…”
We present consensus-based guidance for conducting and documenting multi-analyst studies. We discuss why broader adoption of the multi-analyst approach will strengthen the robustness of results and conclusions in empirical sciences.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.