This study explores how researchers’ analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers’ expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team’s workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers’ results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.

No abstract

Panel models in structural equation modeling that combine static and dynamic components make it possible to investigate reciprocal relations while controlling for time-invariant unobserved heterogeneity. Recently, the Latent Curve Model with Structured Residuals and the Random-Intercept Cross-Lagged Panel Model were suggested as 'residual-level' versions of the more traditional Autoregressive Latent Trajectory and Dynamic Panel Models, respectively. Their main benefit is that they allow for a more straightforward interpretation of the trajectory factors. It is not widely known, however, that the residuallevel models place potentially strong assumptions on the initial conditions, i.e., the process that was occurring before the observation period began. If the process under investigation is not both stationary and at equilibrium then the residual-level models are not appropriate. They then do not control for all time-invariant unobserved heterogeneity and can result in biased cross-lagged and autoregressive estimates. I demonstrate this using the problem behavior of cigarette smoking amongst adolescents: because the mean and variance of this process changes as a young person's smoking behavior develops, early stages of this process should not be examined using the residual-level models. This issue potentially exists for a wide variety of psychological and sociological topics, essentially whenever the process under investigation is changing over the course of the observation period. This paper discusses strategies to help researchers decide which model to use when, and compares some of their relative advantages and drawbacks. An amendment to the residual-level models is suggested in which the latent individual effects are allowed to covary with the initial residuals. This makes the residual-level models robust to violations of the assumptions surrounding the initial conditions, while retaining their other beneficial aspects.

This article looks at paradata in the form of response latencies to identify socially desirable response behaviour. Response latencies are used as proxies to infer information processing modes. So far, evidence is conflicted as to whether socially desirable responding is indicated by shorter or longer response latencies. Our results show that faster responses are associated with the reporting of desirable attitudes and behaviour while slower responses are linked with those that are undesirable. Trait desirability measures that do not take this difference in direction into account may be responsible for the often contradictory results of various researchers who have employed the method in the past.

ObjectivesThe aim of the study is to investigate the relationship between migration background and COVID-19 vaccine intentions, exploring multiple mediation paths. We argue that the migrational and sociocultural background influences general attitudes toward health and political/public institutions. The effects of these general attitudes on vaccination intentions are mediated by fears of infection. Additionally, we analyze a migrant-only model including acculturation variables (years since migration, foreign and host country media consumption) and region of origin (European vs. Non-European). Design: The data (n = 1027) stem from an online access panel collected between March 15 and March 25, 2021. Quotas for gender and age were set according the online population of Germany. The use of an oversampling framework for first generation migrants resulted in a sample with 50% first generation migrants and 50% native Germans without migration background. Models were calculated using a Structural Equation Modeling approach.ResultsMigration background both increases and decreases antecedents of vaccination intentions. Being a migrant increases positive antecedents like religiosity, which in turn positively influence general attitudes and thus fears of infection and vaccination intentions. But being a migrant has also a significant direct negative association with vaccination intentions, implying missing mediators. Increasing years since migration increase host country (German) media consumption and decrease consumption of media from the country of origin. Both media variables are positively associated with political trust and health consciousness. Additionally, European compared to Non-European migrants have less political trust, fear of personal infection and lower vaccination intentions on the whole.ConclusionsThe study found that vaccination intentions can be understood by applying the proposed hypothetical structure. We found complex associations of the migration and sociocultural background and COVID-19 vaccination intentions, where antecedents of vaccination intentions are both increased and decreased by migration background and migration specific factors.