Inferactive data analysis

Bi, Nan; Markovic, Jelena; Xia, Lucy; Taylor, Jonathan

doi:10.48550/arxiv.1707.06692

Cited by 7 publications

(13 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Informally, the problem above goes by many names such as the "file drawer effect" (Fithian et al, 2014;Taylor and Tibshirani, 2015) or "p-hacking" (Simmons et al, 2011;Gelman and Loken, 2013), where researchers only report the final statistical analysis and ignore prior analysis, such as variable selection, that led up to the final analysis. Some recent work have framed the problem as selective inference and showed that ignoring variable selection when testing a hypothesis can lead to inflated Type I errors and biased confidence intervals (Fithian et al, 2014;Lee and Taylor, 2014;Taylor et al, 2014;Bi et al, 2017). In Section 4, we also show that this phenomena holds in MR, where the typical confidence interval from MR ignoring instrument selection can have coverage much lower than the nominal level in many cases.…”

Section: Prior Workmentioning

confidence: 55%

Inference After Selecting Plausibly Valid Instruments with Application to Mendelian Randomization

Bi,

Kang,

Taylor

2019

Preprint

Self Cite

View full text Add to dashboard Cite

Mendelian randomization (MR) is a popular method in genetic epidemiology to estimate the effect of an exposure on an outcome by using genetic instruments. These instruments are often selected from a combination of prior knowledge from genome wide association studies (GWAS) and data-driven instrument selection procedures or tests. Unfortunately, when testing for the exposure effect, the instrument selection process done a priori is not accounted for. This paper studies and highlights the bias resulting from not accounting for the instrument selection process by focusing on a recent data-driven instrument selection procedure, sisVIVE, as an example. We introduce a conditional inference approach that conditions on the instrument selection done a priori and leverage recent advances in selective inference to derive conditional null distributions of popular test statistics for the exposure effect in MR. The null distributions can be characterized with individual-level or summary-level data in MR. We show that our conditional confidence intervals derived from conditional null distributions attain the desired nominal level while typical confidence intervals computed in MR do not. We conclude by reanalyzing the effect of BMI on diastolic blood pressure using summary-level data from the UKBiobank that accounts for instrument selection.

show abstract

Section: Prior Workmentioning

confidence: 55%

Inference After Selecting Plausibly Valid Instruments with Application to Mendelian Randomization

Bi,

Kang,

Taylor

2019

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…We take a different approach than Moreira (2009) and derive (1) by using recent advances in selective inference (Fithian et al, 2014;Tian et al, 2018Tian et al, , 2016Lee et al, 2016;Bi et al, 2017). Broadly speaking, selective inference derives conditional distributions of test statistics for complex conditioning events such as those resulting from marginal screening (Lee and Taylor, 2014), forward stepwise regression (Tibshirani et al, 2016;Loftus and Taylor, 2014), and the Lasso (Tibshirani, 1996;Lee et al, 2016).…”

Section: Prior Work and Our Contributionsmentioning

confidence: 99%

“…We use the conditional nulls in equations ( 11) and ( 12) to construct tests that control the conditional Type I error at level α and to invert them to achieve conditional 1 − α coverage (Fithian et al, 2014;Bi et al, 2017). This is akin to using the traditional null distribution of tests in (2) to obtain p-values and confidence intervals.…”

Section: Problem Statementmentioning

confidence: 99%

“…First, we initialize t (0) to be the observed TSLS test statistic t TSLS and d (0) to be the solution from the randomized convex program (15). Second, if g is Normal, a simple hit-and-run sampling algorithm can be used without the need for a step size; in this case, the sampling density reduces to a truncated gaussian with quadrant constraint on d ; see Bi et al (2017) for details.…”

Section: Algorithm 1 Gibbs Sampler For the Conditional Inferencementioning

confidence: 99%

See 1 more Smart Citation

Inferring Treatment Effects After Testing Instrument Strength in Linear Models

Bi¹,

Kang²,

Taylor³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

A common practice in IV studies is to check for instrument strength, i.e. its association to the treatment, with an F-test from regression. If the F-statistic is above some threshold, usually 10, the instrument is deemed to satisfy one of the three core IV assumptions and used to test for the treatment effect. However, in many cases, the inference on the treatment effect does not take into account the strength test done a priori. In this paper, we show that not accounting for this pretest can severely distort the distribution of the test statistic and propose a method to correct this distortion, producing valid inference. A key insight in our method is to frame the F-test as a randomized convex optimization problem and to leverage recent methods in selective inference. We prove that our method provides conditional and marginal Type I error control. We also extend our method to weak instrument settings. We conclude with a reanalysis of studies concerning the effect of education on earning where we show that not accounting for pre-testing can dramatically alter the original conclusion about education's effects. | INTRODUCTION | MotivationInstrumental variables (IV) is a commonly used approach in economics, epidemiology, genetics, and health policy to estimate the effect of an exposure, treatment, or policy on an outcome of interest; see Angrist and Krueger (2001), Robins (2006), andBaiocchi et al. (2014) for overviews. IV methods require finding variables, known as 1

show abstract

“…Randomization indeed overcomes the drawbacks of excessively long intervals (Bi et al, 2017;Panigrahi et al, 2018), but the computational cost for the adjustment for selection is much higher than Lee et al (2016). A major roadblock in correcting for randomized selection rules is posed by the lack of a pivot in closed form expression.…”

Section: Introductionmentioning

confidence: 99%

Approximate selective inference via maximum likelihood

Panigrahi¹,

Taylor²

2019

Preprint

Self Cite

View full text Add to dashboard Cite

This paper considers an approximate version of the conditional approach to selective inference via maximum likelihood. There are two important considerations in taking a post-selective approach to inference. While one of them concerns the effective use of information in data during inference, the other aspect deals with the computational cost of adjusting for selection. Our approximate proposal serves both these purposes-exploits the use of randomness for efficient utilization of left-over information from selection and enables us to bypass potentially expensive MCMC sampling from conditional distributions. At the core of our method is the solution to a convex optimization problem, leading to an approximate pivot which yields valid post-selective inference across a wide range of signal regimes. Efficient by orders of magnitude than MCMC sampling, correcting for multiple exploratory queries via our proposal involves solving only a single optimization-which takes a separable form across queries. This allows us to address the problem of tractable and efficient inference in many practical scenarios, where more than one exploratory query is conducted on the data to define and perhaps, redefine models and associated parameters. Through an in-depth simulation analysis, we illustrate the promise of our proposal and provide extensive comparisons with other post-selective schemes in both randomized and non-randomized paradigms of inference.

show abstract

Inferactive data analysis

Cited by 7 publications

References 0 publications

Inference After Selecting Plausibly Valid Instruments with Application to Mendelian Randomization

Inference After Selecting Plausibly Valid Instruments with Application to Mendelian Randomization

Inferring Treatment Effects After Testing Instrument Strength in Linear Models

Approximate selective inference via maximum likelihood

Contact Info

Product

Resources

About