Substantive Importance and the Veil of Statistical Significance

McCaskey, Kelly; Rainey, Carlisle

doi:10.1515/spp-2015-0001

Cited by 13 publications

(7 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is statistically different from the influence of business interests at a 0.1 level. As sample sizes as large as the one in this study deflate p-values (McCaskey and Rainey, 2015), this level of significance is weak evidence for an actual difference in influence. We, therefore, assess the substantive meaning of the range of predicted effects that is included in 90% confidence intervals (as suggested by (McCaskey and Rainey, 2015)).…”

Section: Resultscontrasting

confidence: 57%

Shaping EU agencies’ rulemaking

2022

View full text Add to dashboard Cite

EU agencies have become important regulatory venues. Initially established to provide expert advice, many have gained far-reaching decision-making and enforcement powers. This has attracted considerable attention from stakeholders, but the extent of their influence on EU agency conduct has remained a black box. We employ a novel dataset of 203 consultations (2007-2017) containing 26,468 attempts of stakeholders to change proposed regulatory rules by the European Union Aviation Safety Agency, as well as the agency's response. This dataset allows for an original approach to measuring influence by linking influence attempts to rule changes. We found that business interests are far more influential than diffuse public interests. This has important implications for the legitimacy of EU agency stakeholder policies, as they are meant to make EU agencies more broadly accessible. National regulators are also influential in EU agency consultations, pointing to the unacknowledged importance of stakeholder consultations for EUnational regulator interactions.

show abstract

Section: Resultscontrasting

confidence: 57%

Shaping EU agencies’ rulemaking

2022

View full text Add to dashboard Cite

show abstract

“…In so doing, we pay attention to our 95% confidence intervals. This allows us to discuss what effect sizes we are able to rule out; an approach intuitively similar to the equivalence testing approach suggested by Hartman and Hidalgo (2018) and others in the literature on null effects and statistical/substantive significance (Gross 2015; McCaskey and Rainey 2015; Rainey 2014). In our models below, we use Hartman and Hidalgo’s default values for equivalence testing (36% of a standard deviation) and test whether our effects are distinct from that benchmark.…”

Section: Methodsmentioning

confidence: 99%

Noisy Retrospection: The Effect of Party Control on Policy Outcomes

Dynes

Holbein

2019

Am Polit Sci Rev

View full text Add to dashboard Cite

Retrospective voting is vital for democracy. But, are the objective performance metrics widely thought to be relevant for retrospection—such as the performance of the economy, criminal justice system, and schools, to name a few—valid criteria for evaluating government performance? That is, do political coalitions actually have the power to influence the performance metrics used for retrospection on the timeline introduced by elections? Using difference-in-difference and regression discontinuity techniques, we find that US states governed by Democrats and those by Republicans perform equally well on economic, education, crime, family, social, environmental, and health outcomes on the timeline introduced by elections (2–4 years downstream). Our results suggest that voters may struggle to truly hold government coalitions accountable, as objective performance metrics appear to be largely out of the immediate control of political coalitions.

show abstract

“…Therefore, we follow their suggestion to accept uncertainty and to report the original two‐tailed p ‐values in separate columns to avoid the use of arbitrary statistical thresholds. Note that, statistically, p ‐values cannot play the role of t ‐statistics because they can only tell us ‘the maximum probability of obtaining hypothetical data at least as extreme as the observed data if the null hypothesis were true’ (McCaskey and Rainey ).…”

Section: Empirical Analyses and Resultsmentioning

confidence: 99%

How social capital shapes citizen willingness to co‐invest in public service: The case of flood control

Zhang

Liu

Vedlitz

2020

Public Administration

View full text Add to dashboard Cite

Why are citizens in some communities more willing to co‐invest in public services than citizens in other communities? Citizen co‐investment has become an important policy tool for governments to finance public services, yet little research has theorized and empirically examined the impacts of community contexts on citizens' willingness to invest their personal resources in public services. As social capital is often viewed as an important determinant of citizen behaviours, we propose two competing hypotheses explaining the relationship between social capital and citizen co‐investment: the facilitation effect hypothesis and the inhibition effect hypothesis. Based on three data sources, our statistical analyses consistently show that citizens living in counties with more social capital are less willing to co‐invest in local flood control. This finding provides empirical evidence to support the inhibition effect hypothesis. Key implications of our research and suggestions for future research are discussed in the conclusion.

show abstract

Substantive Importance and the Veil of Statistical Significance

Cited by 13 publications

References 15 publications

Shaping EU agencies’ rulemaking

Shaping EU agencies’ rulemaking

Noisy Retrospection: The Effect of Party Control on Policy Outcomes

How social capital shapes citizen willingness to co‐invest in public service: The case of flood control

Contact Info

Product

Resources

About