Heterogeneous Subgroup Identification in Observational Studies

Chen, Jianshen; Keller, Bryan

doi:10.1080/19345747.2019.1615159

Cited by 7 publications

(6 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Chen and Keller (2019) and Keller and Tipton (2016) replicated Morgan et al's (2010) propensity matching analysis. Chen and Keller (2019) found similar overall effects to Morgan et al (2010) but noted more beneficial effects of special education on mathematics achievement for female students. Keller and Tipton (2016) provided a technical primer on propensity score matching, finding more negative effects of special education on mathematics achievement than Morgan et al (2010).…”

Section: Methodological Limitations Of Prior Meta-analytic Worksupporting

confidence: 58%

“…For example, studies that use propensity score matching have found mostly negative (Kanaya et al, 2019;Sullivan & Field, 2013) or non-significant (Morgan et al, 2010) effects of special education on academic achievement (Morgan et al, 2010;Sullivan & Field, 2013), high school course completion (Shifrer et al, 2013), and long-term outcomes such as educational attainment, income, welfare and other public assistance, drug use, and criminal convictions (Kanaya et al, 2019). Chen and Keller (2019) and Keller and Tipton (2016) replicated Morgan et al's (2010) propensity matching analysis. Chen and Keller (2019) found similar overall effects to Morgan et al (2010) but noted more beneficial effects of special education on mathematics achievement for female students.…”

Section: Methodological Limitations Of Prior Meta-analytic Workmentioning

confidence: 86%

“…Yet, this adjustment has not been made in much prior work. This is because many studies compare SWD to SWOD when evaluating the causal effects of special education services (Chen & Keller, 2019;Kanaya et al, 2019;Keller & Tipton, 2016;Morgan et al, 2010;Shifrer et al, 2013;Sullivan & Field, 2013).…”

Section: Methodological Limitations Of Prior Meta-analytic Workmentioning

confidence: 99%

See 2 more Smart Citations

A Multivariate Meta-Analysis of the Effects of Business-as-Usual Special Education Services in U.S. Schools

Woods¹,

Gloski²,

Wang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Special education services provided in U.S. schools should be designed to enable students with disabilities (SWD) to make “appropriately ambitious” educational progress (Endrew F., 2017). Yet the field lacks consensus on whether business-as-usual (BAU) special education services effectively support outcomes for SWD. We conducted a moderated multilevel multivariate meta-analysis of 40 studies and 1,721 effect sizes from more than 14 million participants to evaluate the effectiveness of BAU special education services. Effects ranged from null-to-positive (z = -0.07 to 0.12) for achievement and negative-to-null (z = -0.13 to -0.01) for behavior. We observed positive outcomes for educational attainment including graduation outcomes (z = 0.12 to 0.24) and postsecondary outcomes (z = 0.10 to 0.23), as well as for other outcomes (z = 0.21 to 0.38). Studies that used a more rigorous research design or comparison condition reported more positive effects across outcomes. Additional casual work is needed in this field.

show abstract

Section: Methodological Limitations Of Prior Meta-analytic Worksupporting

confidence: 58%

Section: Methodological Limitations Of Prior Meta-analytic Workmentioning

confidence: 86%

See 1 more Smart Citation

A Multivariate Meta-Analysis of the Effects of Business-as-Usual Special Education Services in U.S. Schools

Woods¹,

Gloski²,

Wang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In addition, some have leveraged these permutation procedures to propose a method with the aim of identifying the appropriate complexity parameter for regression trees for the conditional average treatment effect estimated through propensity score matching such that the type I error rate is maintained and the conditional average treatment effect can be modeled if detected. 35 While our proposed method shares the goal of tuning regression trees’ complexity parameters, it is far more general and can support any Step 2 method with a single tuning parameter.…”

Section: Discussionmentioning

confidence: 99%

A permutation procedure to detect heterogeneous treatment effects in randomized clinical trials while controlling the type I error rate

2022

View full text Add to dashboard Cite

Background/Aims Secondary analyses of randomized clinical trials often seek to identify subgroups with differential treatment effects. These discoveries can help guide individual treatment decisions based on patient characteristics and identify populations for which additional treatments are needed. Traditional analyses require researchers to pre-specify potential subgroups to reduce the risk of reporting spurious results. There is a need for methods that can detect such subgroups without a priori specification while allowing researchers to control the probability of falsely detecting heterogeneous subgroups when treatment effects are uniform across the study population. Methods We propose a permutation procedure for tuning parameter selection that allows for type I error control when testing for heterogeneous treatment effects framed within the Virtual Twins procedure for subgroup identification. We verify that the type I error rate can be controlled at the nominal rate and investigate the power for detecting heterogeneous effects when present through extensive simulation studies. We apply our method to a secondary analysis of data from a randomized trial of very low nicotine content cigarettes. Results In the absence of type I error control, the observed type I error rate for Virtual Twins was between 99% and 100%. In contrast, models tuned via the proposed permutation were able to control the type I error rate and detect heterogeneous effects when present. An application of our approach to a recently completed trial of very low nicotine content cigarettes identified several variables with potentially heterogeneous treatment effects. Conclusions The proposed permutation procedure allows researchers to engage in secondary analyses of clinical trials for treatment effect heterogeneity while maintaining the type I error rate without pre-specifying subgroups.

show abstract

“…The purpose was to assess, of those key predictors, what were the driving factors that differentiated students' writing performance. A similar two‐step method was implemented in Chen and Keller (2019). The rpart package was used to conduct single regression tree analyses (Therneau et al, 2019) in R (R Core Team, 2015).…”

Section: Methodsmentioning

confidence: 99%

Examining the Writing Processes in Scenario‐Based Assessment Using Regression Trees

Cao

Chen

Zhang

et al. 2020

ETS Research Report Series

Self Cite

View full text Add to dashboard Cite

Scenario‐based writing assessment has two salient characteristics by design: a lead‐in/essay scaffolding structure and a unified scenario/topic throughout. In this study, we examine whether the scenario‐based assessment design would impact students' essay scores compared to its alternative conditions, which intentionally broke the scaffolding effect and/or topic effect. Furthermore, with the availability of rich keystroke log data, we used tree‐based methods to investigate which writing process features had greater influence in predicting students' essay scores under different assessment conditions. The results revealed significantly lower essay scores when the unified topic effect was removed and no significant essay score change when neither scaffolding nor topic effects existed compared to the original scenario‐based assessment condition. The findings also suggested that different assessment conditions call on different configurations of writing process features. The rate of typo correction in the writing process was consistently among the highest ranking features that predicted the essay scores well, regardless of the presence of scaffolding and/or topic effects. The broken topic effect might affect the extent of word finding and retrieval as well as editing across multiple words by the students during writing. Topics for future study and limitations are also discussed.

show abstract

Heterogeneous Subgroup Identification in Observational Studies

Cited by 7 publications

References 27 publications

A Multivariate Meta-Analysis of the Effects of Business-as-Usual Special Education Services in U.S. Schools

A Multivariate Meta-Analysis of the Effects of Business-as-Usual Special Education Services in U.S. Schools

A permutation procedure to detect heterogeneous treatment effects in randomized clinical trials while controlling the type I error rate

Examining the Writing Processes in Scenario‐Based Assessment Using Regression Trees

Contact Info

Product

Resources

About