Questionable and Open Research Practices: Attitudes and Perceptions among Quantitative Communication Researchers

Bakker, Bert N.; Jaidka, Kokil; Dorr, Timothy G.; Fasching, Neil; Lelkes, Yphtach

doi:10.1093/joc/jqab031

Cited by 20 publications

(5 citation statements)

References 51 publications

(90 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Failing to fully disclose the handling of missing variables and the method used for imputation is considered a questionable research practice (B. N. Bakker et al, 2021), and deciding for a favorable imputation method based on statistical significance constitutes a p-hacking strategy (Aguinis et al, 2018;Wicherts et al, 2016). Meta-scientific evidence suggests that authors rarely provide a satisfactory justification for their chosen method for handling missing values (Crede & Harms, 2019;Sterne et al, 2009) We simulate p-hacking through favorable imputation using a total of ten different methods for handling missing variables.…”

Section: (10) Favorable Imputationmentioning

confidence: 99%

Big Little Lies: A Compendium and Simulation of p-Hacking Strategies

Stefan¹,

Schönbrodt²

2022

Preprint

View full text Add to dashboard Cite

In many research fields, the widespread use of questionable research practices has jeopardized the credibility of scientific results. One of the most prominent questionable research practices is p-hacking. Typically, p-hacking is defined as a compound of strategies targeted at rendering non-significant hypothesis testing results significant. However, a comprehensive overview of these p-hacking strategies is missing, and current meta-scientific research often ignores the heterogeneity of strategies. Here, we compile a list of twelve p-hacking strategies based on an extensive literature review, identify factors that control their level of severity, and demonstrate their impact on false-positive rates using simulation studies. We also use our simulation results to evaluate several approaches that have been proposed to mitigate the influence of questionable research practices. Our results show that investigating p-hacking at the level of strategies can provide a better understanding of the process of p-hacking, as well as a broader basis for developing effective countermeasures. By making our analyses available through a Shiny app and R package, we facilitate future meta-scientific research aimed at investigating the ramifications of p-hacking across multiple strategies, and we hope to start a broader discussion about different manifestations of p-hacking in practice.

show abstract

Section: (10) Favorable Imputationmentioning

confidence: 99%

Big Little Lies: A Compendium and Simulation of p-Hacking Strategies

Stefan¹,

Schönbrodt²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Failing to fully disclose the handling of missing variables and the method used for imputation is considered a questionable research practice [ 83 ], and deciding on a favourable imputation method based on statistical significance constitutes a p -hacking strategy [ 26 , 84 ]. Meta-scientific evidence suggests that authors rarely provide a satisfactory justification for their chosen method for handling missing values [ 85 , 86 ].…”

Section: A Compendium Of P -Hacking Strategiesmentioning

confidence: 99%

“…Meta-scientific evidence suggests that authors rarely provide a satisfactory justification for their chosen method for handling missing values [ 85 , 86 ]. Surveys on ethical research conduct among communication scientists and criminologists indicate that slightly less than 10% of researchers have hidden imputation methods from research reports [ 66 , 83 ].…”

Section: A Compendium Of P -Hacking Strategiesmentioning

confidence: 99%

Big little lies: a compendium and simulation ofp-hacking strategies

Stefan

Schönbrodt

2023

R. Soc. open sci.

View full text Add to dashboard Cite

In many research fields, the widespread use of questionable research practices has jeopardized the credibility of scientific results. One of the most prominent questionable research practices is p -hacking. Typically, p -hacking is defined as a compound of strategies targeted at rendering non-significant hypothesis testing results significant. However, a comprehensive overview of these p -hacking strategies is missing, and current meta-scientific research often ignores the heterogeneity of strategies. Here, we compile a list of 12 p -hacking strategies based on an extensive literature review, identify factors that control their level of severity, and demonstrate their impact on false-positive rates using simulation studies. We also use our simulation results to evaluate several approaches that have been proposed to mitigate the influence of questionable research practices. Our results show that investigating p -hacking at the level of strategies can provide a better understanding of the process of p -hacking, as well as a broader basis for developing effective countermeasures. By making our analyses available through a Shiny app and R package, we facilitate future meta-scientific research aimed at investigating the ramifications of p -hacking across multiple strategies, and we hope to start a broader discussion about different manifestations of p -hacking in practice.

show abstract

“…Replication and extensions of prior work are rare in communication research and other social science fields (Bakker et al, 2021;Keating & Totzkay, 2019;Markowitz, Song, et al, 2021), but they remain essential to evaluate the validity and reliability of theory. The current work adopts this approach by considering how the link between processing fluency, the instrumentality heuristic, and language replicates and extends to online real estate advertisements.…”

Section: The Current Papermentioning

confidence: 99%

Words for Sale: Linguistic Complexity Associates with Higher Housing Prices in Online Realty Advertisements

Markowitz

2023

Journal of Language and Social Psychology

View full text Add to dashboard Cite

Prior work suggests when instrumental goals are salient, linguistic complexity associates with positive behavioral outcomes compared to linguistic simplicity. The current work tested this idea using descriptions from over 1,000 online realty advertisements to associate with housing prices. The evidence suggested linguistic complexity (e.g., fewer common words, more analytic writing, and less readable writing) indeed associated with higher housing prices. These data explicated the contingent-on-effort hypothesis: linguistic complexity is favored when people value effort in a particular setting.

show abstract

Questionable and Open Research Practices: Attitudes and Perceptions among Quantitative Communication Researchers

Cited by 20 publications

References 51 publications

Big Little Lies: A Compendium and Simulation of p-Hacking Strategies

Big Little Lies: A Compendium and Simulation of p-Hacking Strategies

Big little lies: a compendium and simulation ofp-hacking strategies

Words for Sale: Linguistic Complexity Associates with Higher Housing Prices in Online Realty Advertisements

Contact Info

Product

Resources

About