Emergency Medicine Residents and Statistics: What is the Confidence?

Hack, Jason B.; Bakhtiari, P.; O’Brien, Kevin F.

doi:10.1016/j.jemermed.2007.07.021

Cited by 13 publications

(10 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous research suggests that physicians do not understand biostatistical concepts. 28,29 Because few want to be the first or the last to adopt research-based practice changes, busy clinicians face dual impediments to interpreting the medical literature: a continually decreasing signal-to-noise ratio because of an ever-expanding volume of research manuscripts and insufficient time, training, or expertise to critically appraise original scientific reports independently. 30–34 Therefore, one would surmise that even fewer practicing EPs would be able to review a study in its entirety.…”

Section: Discussionmentioning

confidence: 99%

Consensus Conference Follow-up: Inter-rater Reliability Assessment of the Best Evidence in Emergency Medicine (BEEM) Rater Scale, a Medical Literature Rating Tool for Emergency Physicians

Worster

Kulasegaram²,

Carpenter

et al. 2011

Academic Emergency Medicine

View full text Add to dashboard Cite

Background Studies published in general and specialty medical journals have the potential to improve emergency medicine (EM) practice, but there can be delayed awareness of this evidence because emergency physicians (EPs) are unlikely to read most of these journals. Also, not all published studies are intended for or ready for clinical practice application. The authors developed “Best Evidence in Emergency Medicine” (BEEM) to ameliorate these problems by searching for, identifying, appraising, and translating potentially practice-changing studies for EPs. An initial step in the BEEM process is the BEEM rater scale, a novel tool for EPs to collectively evaluate the relative clinical relevance of EM-related studies found in more than 120 journals. The BEEM rater process was designed to serve as a clinical relevance filter to identify those studies with the greatest potential to affect EM practice. Therefore, only those studies identified by BEEM raters as having the highest clinical relevance are selected for the subsequent critical appraisal process and, if found methodologically sound, are promoted as the best evidence in EM. Objectives The primary objective was to measure inter-rater reliability (IRR) of the BEEM rater scale. Secondary objectives were to determine the minimum number of EP raters needed for the BEEM rater scale to achieve acceptable reliability and to compare performance of the scale against a previously published evidence rating system, the McMaster Online Rating of Evidence (MORE), in an EP population. Methods The authors electronically distributed the title, conclusion, and a PubMed link for 23 recently published studies related to EM to a volunteer group of 134 EPs. The volunteers answered two demographic questions and rated the articles using one of two randomly assigned seven-point Likert scales, the BEEM rater scale (n = 68) or the MORE scale (n = 66), over two separate administrations. The IRR of each scale was measured using generalizability theory. Results The IRR of the BEEM rater scale ranged between 0.90 (95% confidence interval [CI] = 0.86 to 0.93) to 0.92 (95% CI = 0.89 to 0.94) across administrations. Decision studies showed a minimum of 12 raters is required for acceptable reliability of the BEEM rater scale. The IRR of the MORE scale was 0.82 to 0.84. Conclusions The BEEM rater scale is a highly reliable, single-question tool for a small number of EPs to collectively rate the relative clinical relevance within the specialty of EM of recently published studies from a variety of medical journals. It compares favorably with the MORE system because it achieves a high IRR despite simply requiring raters to read each article’s title and conclusion.

show abstract

Section: Discussionmentioning

confidence: 99%

Consensus Conference Follow-up: Inter-rater Reliability Assessment of the Best Evidence in Emergency Medicine (BEEM) Rater Scale, a Medical Literature Rating Tool for Emergency Physicians

Worster

Kulasegaram²,

Carpenter

et al. 2011

Academic Emergency Medicine

View full text Add to dashboard Cite

show abstract

“…Understanding the descriptive and analytic statistics that underlie EM research is an essential part of the practice of EM 1,4,5 . Emergency department length of stay has been identified as a key process metric of ED function, and numerous studies have shown an association between ED length of stay and patient satisfaction, 21,22 quality of care, 19,20 and departmental revenue 23 .…”

Section: Discussionmentioning

confidence: 99%

“…Despite this recent increased emphasis on statistical literacy, many physicians cannot demonstrate competence with basic biostatistical concepts 1,4,5 . This leads to mistakes in statistical analysis 6 and reluctance to conduct research 7,8 .…”

mentioning

confidence: 99%

Parametric Versus Nonparametric Statistical Tests: The Length of Stay Example

Qualls

Pallin

Schuur

2010

Academic Emergency Medicine

View full text Add to dashboard Cite

Objectives: This study examined selected effects of the proper use of nonparametric inferential statistical methods for analysis of nonnormally distributed data, as exemplified by emergency department length of stay (ED LOS). The hypothesis was that parametric methods have been used inappropriately for evaluation of ED LOS in most recent studies in leading emergency medicine (EM) journals. To illustrate why such a methodologic flaw should be avoided, a demonstration, using data from the National Hospital Ambulatory Medical Care Survey (NHAMCS), is presented. The demonstration shows how inappropriate analysis of ED LOS increases the probability of type II errors.Methods: Five major EM journals were reviewed, January 1, 2004, through December 31, 2007, and all studies with ED LOS as one of the reported outcomes were reviewed. The authors determined whether ED LOS was analyzed correctly by ascertaining whether nonparametric tests were used when indicated. An illustrative analysis of ED LOS was constructed using 2006 NHAMCS data, to demonstrate how inferential testing for statistical significance can deliver differing conclusions, depending on whether nonparametric methods are used when indicated.Results: Forty-nine articles were identified that studied ED LOS; 80% did not perform a test of normality on the ED LOS data. Data were not normally distributed in all 10 of the studies that did perform such tests. Overall, 43% failed to use appropriate nonparametric methods. Analysis of NHAMCS data confirmed that failure to use nonparametric bivariate tests results in type II statistical error and in multivariate models with less explanatory power (a smaller R 2 value).Conclusions: ED LOS, a key ED operational metric, is frequently analyzed incorrectly in the EM literature. Applying parametric statistical tests to such nonnormally distributed data reduces power and increases the probability of a type II error, which is the failure to find true associations. Appropriate use of nonparametric statistics should be a core component of statistical literacy because such use increases the validity of ED research and quality improvement projects.

show abstract

“…To further complicate the issue, we were teaching non‐statisticians with little research experience. Previous research has shown not only an inadequate level of resident competence in research methods and statistics, but also a general anxiety or resistance to learning about these topics (Hack, Bakhtiari, & O'Brien, ; Novack, Jotkowitz, Knyazer, & Novack, ; Windish, Huot, & Green, ). This lecture series was also not associated with a grade, which made accountability and attendance a voluntary endeavor.…”

Section: The Five Strategiesmentioning

confidence: 99%

Five Strategies for Successful Assessment in Graduate Medical Education

Barlow¹,

Smith²,

Skolits³

2014

World Med & Health Policy

View full text Add to dashboard Cite

This discussion provides faculty, or anyone conducting their own assessment, with five key strategies for useful assessment in graduate medical education (GME) and the broader higher education environment with or without the benefit of pre‐existing learning objectives. Higher education faculty, including GME, are not trained to develop their own assessment strategies (Hutchings, 2010). Consequently, developing assessment “from the ground up” can be an arduous process, particularly without previous strategies in place. We introduce the strategies using a free‐standing evidence‐based medicine curriculum as a case. The five strategies are: (1) know your situation; (2) clarify your purpose; (3) use what you have; (4) fit the instrument to your purpose, not the other way around; and (5) get consistent and critical feedback. These five strategies should prove beneficial in making sure effective assessment data is gathered efficiently as medical education moves towards a competency‐based model.

show abstract

Emergency Medicine Residents and Statistics: What is the Confidence?

Cited by 13 publications

References 27 publications

Consensus Conference Follow-up: Inter-rater Reliability Assessment of the Best Evidence in Emergency Medicine (BEEM) Rater Scale, a Medical Literature Rating Tool for Emergency Physicians

Consensus Conference Follow-up: Inter-rater Reliability Assessment of the Best Evidence in Emergency Medicine (BEEM) Rater Scale, a Medical Literature Rating Tool for Emergency Physicians

Parametric Versus Nonparametric Statistical Tests: The Length of Stay Example

Five Strategies for Successful Assessment in Graduate Medical Education

Contact Info

Product

Resources

About