Three Roles for Statistical Significance and the Validity Frontier in Theory Testing

Lee, Allen; Mohajeri, Kaveh; Hubona, Geoffrey S.

doi:10.24251/hicss.2017.691

“…For instance, consider a theory operationalized as a statistical equation in a regression analysis or PLS study. Lee and Hubona (2009) and Lee, Mohajeri, and Hubona (2017) explain how the statistical equation can be used to make predictions which are judged to either succeed or fail, where the overall record of successes or failures can then be used to confirm or refute the theory. Until this testing is done, the statistical analysis just engages in theory fitting, that is, the matter of how well the theory fits the data, rather than the matter of whether the data confirm or refute the theory.…”

Section: Discussionmentioning

confidence: 99%

Conflating relevance with practical significance and other issues: Commentary on Sen, Smith, and Van Note’s “statistical significance versus practical importance in information systems research”

Mohajeri

¹

,

Mesgari

²

,

Lee

³

2022

Journal of Information Technology

Self Cite

0

View full text Add to dashboard Cite

Expanding on the current debate on the issues of statistical and practical significance in information systems research, where the article by Sen, Smith, and Van Note is a recent contribution, this commentary cautions against conflating relevance with practical significance. We emphasize that relevance is 1) about the real-world usefulness of research findings rather than their impressiveness for the researcher audience, 2) an essential quality of research spanning beyond its findings and not merely limited to statistical studies, and 3) determined by nonacademics rather than academics. We also comment on other aspects of the article by Sen et al., such as the term “practical importance,” the treatment of effect size measures, and the presentation of “marginal effects.”

show abstract

“…During data analysis and interpretation, we believe there is evidence to suggest that thresholdbased reporting is prevalent in IS scholarship. We also found instances where the usage of statistical significance and p-values confuse statistical and practical significance (Lee, Mohajeri, & Hubona, 2017). For example, if we find that the unstandardized regression coefficient for the effect of perceived ease of use on perceived usefulness is .116, this means that someone scoring one point closer to the 'strongly agree' side of a 7-point Likert-type response scale for perceived ease of use, scores .116 points closer to 7 for perceived usefulness.…”

Section: Report Findingsmentioning

confidence: 91%

“…The "mid-range script" and its typical statistically testable model has been challenged as a mode of knowledge construction; Grover and Lyytinen (2015) call for either more theoretically or practically oriented epistemic scripts. Moreover, there has already been a push for theory testing to go beyond 'effect' and 'prediction' testing, and for equal weight to be given to statistical significance and 'practical significance' (Lee et al, 2017). Others have highlighted the danger of Type I errors ('false positives') when sample sizes are large (Lin et al, 2013) and when reviewing papers (Straub, 2008), and discussed challenges relating to measurement (Bagozzi, 2011;Burton-Jones & Lee, 2017) and generalization (Lee & Baskerville, 2003;Tsang & Williams, 2012).…”

Section: Proposing a Way Forwardmentioning

confidence: 99%

See 1 more Smart Citation

New Guidelines for Null Hypothesis Significance Testing in Hypothetico-Deductive IS Research

Mertens¹,

Recker²

2020

JAIS

View full text Add to dashboard Cite

The objective of this Research Perspectives article is to promote policy change amongst journals, scholars and students with a vested interest in hypothetico-deductive information systems (IS) research. We are concerned about the design, analysis, reporting and reviewing of quantitative IS studies that draw on null hypothesis significance testing (NHST). We observe that debates about misinterpretations, abuse, and issues with NHST, while having persisted for about half a century, remain largely absent in IS. We find this an untenable position for a discipline with a proud quantitative tradition. We discuss traditional and emergent threats associated with the application of NHST and examine how they manifest in recent IS scholarship. To encourage the development of new standards for NHST in hypothetico-deductive IS research, we develop a balanced account of possible actions that are implementable short-term or long-term and that incentivize or penalize specific practices. To promote an immediate push for change, we also develop two sets of guidelines that IS scholars can adopt right away.

show abstract

“…The "mid-range script" and its typical statistically testable model has been challenged as a mode of knowledge construction; Grover and Lyytinen (2015) call for either more theoretically or practically oriented epistemic scripts. Moreover, there has already been a push for theory testing to go beyond 'effect' and 'prediction' testing, and for equal weight to be given to statistical significance and 'practical significance' (Lee et al, 2017). Others have highlighted the danger of Type I errors ('false positives') when sample sizes are large (Lin et al, 2013) and when reviewing papers (Straub, 2008), and discussed challenges relating to measurement (Bagozzi, 2011;Burton-Jones & Lee, 2017) and generalization (Lee & Baskerville, 2003;Tsang & Williams, 2012).…”

Section: Proposing a Way Forwardmentioning

confidence: 99%

New Guidelines for Null Hypothesis Significance Testing in Hypothetico-Deductive IS Research

Mertens,

Recker

2023

Technology, Work and Globalization

0

View full text Add to dashboard Cite

The objective of this Research Perspectives article is to promote policy change amongst journals, scholars and students with a vested interest in hypothetico-deductive information systems (IS) research. We are concerned about the design, analysis, reporting and reviewing of quantitative IS studies that draw on null hypothesis significance testing (NHST). We observe that debates about misinterpretations, abuse, and issues with NHST, while having persisted for about half a century, remain largely absent in IS. We find this an untenable position for a discipline with a proud quantitative tradition. We discuss traditional and emergent threats associated with the application of NHST and examine how they manifest in recent IS scholarship. To encourage the development of new standards for NHST in hypothetico-deductive IS research, we develop a balanced account of possible actions that are implementable short-term or long-term and that incentivize or penalize specific practices. To promote an immediate push for change, we also develop two sets of guidelines that IS scholars can adopt right away.

show abstract

Three Roles for Statistical Significance and the Validity Frontier in Theory Testing

Abstract: Abstract

Cited by 3 publications

References 12 publications

Conflating relevance with practical significance and other issues: Commentary on Sen, Smith, and Van Note’s “statistical significance versus practical importance in information systems research”

Conflating relevance with practical significance and other issues: Commentary on Sen, Smith, and Van Note’s “statistical significance versus practical importance in information systems research”

New Guidelines for Null Hypothesis Significance Testing in Hypothetico-Deductive IS Research

New Guidelines for Null Hypothesis Significance Testing in Hypothetico-Deductive IS Research

Contact Info

Product

Resources

About