This paper summarizes results from 12 empirical evaluations of observational methods in education contexts. We look at the performance of three common covariate-types in observational studies where the outcome is a standardized reading or math test. They are: pretest measures, local geographic matching, and rich covariate sets with a strong theory of treatment selection. Overall, the review demonstrates that although the pretest often reduces bias in observational studies, it does not always eliminate it. Its performance depends on the pretest's correlation with treatment selection and the outcome, and whether pre-intervention trends are present. We also find that although local comparisons are prioritized for matching, its performance depends on whether comparable no-treatment cases are available. Otherwise, local comparisons may produce badly biased results. In cases where researchers have a strong theory of selection and rich covariate sets, observational methods perform well, but additional replication studies are needed. Finally, observational methods that rely on demographic covariates without a theory of selection rarely produce unbiased treatment effects. The paper Downloaded by [University of Nebraska, Lincoln] at 05:40 13 June 2016 ACCEPTED MANUSCRIPT ACCEPTED MANUSCRIPT 2 concludes by offering education researchers empirically-based guidance on covariate selection in observational studies.
This exploratory mixed methods study describes skills required to be an effective peer reviewer as a member of review panels conducted for federal agencies that fund research, and examines how reviewer experience and the use of technology within such panels impacts reviewer skill development. Two specific review panel formats are considered: inperson face-to-face and virtual video conference. Data were collected through interviews with seven program officers and five expert peer review panelists, and surveys from 51 respondents. Results include the skills reviewers' consider necessary for effective review panel participation, their assessment of the relative importance of these skills, how they are learned, and how review format affects skill development and improvement. Results are discussed relative to the peer review literature and with consideration of the importance of professional skills needed by successful scientists and peer reviewers.
Significance: Individuals with type 1 diabetes (T1D) frequently report experiencing diabetes burnout, a feeling of exhaustion, and frustration that may lead to ignoring self-care behaviors. Though the construct was identified in the early 1980s as a primary barrier to optimal diabetes care, scientific understanding of the concept is lacking. Prior literature has conflated diabetes burnout with the other psychosocial concepts of diabetes distress and depression. In an attempt to address the gap, our qualitative findings highlighted that diabetes burnout includes: (1) feelings of mental, emotional, and physical exhaustion, (2) detachment from illness identity, support systems, and self-care, and (3) a sense of powerlessness to re-take the ownership of self-care. Aim: To examine the relationship between the three hypothesized dimensions of diabetes burnout and to determine if the data would be able to identify a relationship between diabetes burnout, distress, depression, and short-term outcomes. Methods: We conducted a cross-sectional study on a sample of 111 adults with T1D who completed an online survey developed by existing valid and reliable measures. Results: All dimensions of diabetes burnout were significantly correlated between .60 and .85 (p<.001). We found that diabetes burnout was significantly associated with both depression (es=0.67; p<.001) and diabetes distress (es=0.83; p<.001). Although overall burnout score explained more of the variation in key diabetes outcomes such as last reported HbA1C or missed appointments than diabetes distress or depression, and those relationships persist and strengthen even after controlling for these other related constructs. Conclusion: A single cross-sectional study is insufficient for any strong claims; however, this study challenges current knowledge on diabetes care by suggesting diabetes burnout as a distinct but interrelated concept with distress or depressions. These findings need to be replicated, ideally with larger and more diverse samples and with longitudinal studies. Disclosure S. Abdoli: None. D.M. Hessler: Consultant; Self; Eli Lilly and Company. K. Miller-Bains: None. A.C. Vora: Speaker’s Bureau; Self; Boehringer Ingelheim Pharmaceuticals, Inc., Dexcom, Inc., Novo Nordisk Inc. B. Smither: None. E.M. Burr: None. H.L. Stuckey: None. Funding Oak Ridge Associated Universities
Modern policies are commonly evaluated not with randomized experiments but with repeated measures designs like difference-in-differences (DID) and the comparative interrupted time series (CITS). The key benefit of these designs is that they control for unobserved confounders that are fixed over time. However, DID and CITS designs only result in unbiased impact estimates when the model assumptions are consistent with the data at hand. In this paper, we empirically test whether the assumptions of repeated measures designs are met in field settings. Using a within-study comparison design, we compare experimental estimates of the impact of patient-directed care on medical expenditures to non-experimental DID and CITS estimates for the same target population and outcome. Our data come from a multi-site experiment that includes participants receiving Medicaid in Arkansas, Florida, and New Jersey. We present summary measures of repeated measures bias across three states, four comparison groups, two model specifications, and two outcomes. We find that, on average, bias resulting from repeated measures designs are very close to zero (less than 0.01 standard deviations; SDs). Further, we find that comparison groups which have pre-treatment trends that are visibly parallel to the treatment group result in less bias than those with visibly divergent trends. However, CITS models that control for baseline trends produced slightly more bias and were less precise than DID models that only control for baseline means. Overall, we offer optimistic evidence in favor of repeated measures designs when randomization is not feasible.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
customersupport@researchsolutions.com
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.