Heckroccurve: ROC Curves for Selected Samples

Cook, Jonathan; Rajbhandari, Ashish

doi:10.1177/1536867x1801800110

Cited by 8 publications

(8 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…23 We corrected receiver operating characteristic curve estimates using the selection model, as developed by Cook and Rajbhandari. 24 in the Supplement). Of the remaining lesions used for the analysis, 77 (16.7%) were identified as melanoma.…”

Section: Discussionmentioning

confidence: 98%

Assessment of Accuracy of an Artificial Intelligence Algorithm to Detect Melanoma in Images of Skin Lesions

Marsden²,

et al. 2019

View full text Add to dashboard Cite

IMPORTANCE A high proportion of suspicious pigmented skin lesions referred for investigation are benign. Techniques to improve the accuracy of melanoma diagnoses throughout the patient pathway are needed to reduce the pressure on secondary care and pathology services. OBJECTIVE To determine the accuracy of an artificial intelligence algorithm in identifying melanoma in dermoscopic images of lesions taken with smartphone and digital single-lens reflex (DSLR) cameras. DESIGN, SETTING, AND PARTICIPANTS This prospective, multicenter, single-arm, masked diagnostic trial took place in dermatology and plastic surgery clinics in 7 UK hospitals. Dermoscopic images of suspicious and control skin lesions from 514 patients with at least 1 suspicious pigmented skin lesion scheduled for biopsy were captured on 3 different cameras. Data were collected from January 2017 to July 2018. Clinicians and the Deep Ensemble for Recognition of Malignancy, a deterministic artificial intelligence algorithm trained to identify melanoma in dermoscopic images of pigmented skin lesions using deep learning techniques, assessed the likelihood of melanoma. Initial data analysis was conducted in September 2018; further analysis was conducted from February 2019 to August 2019. INTERVENTIONS Clinician and algorithmic assessment of melanoma.MAIN OUTCOMES AND MEASURES Area under the receiver operating characteristic curve (AUROC), sensitivity, and specificity of the algorithmic and specialist assessment, determined using histopathology diagnosis as the criterion standard. RESULTSThe study population of 514 patients included 279 women (55.7%) and 484 white patients (96.8%), with a mean (SD) age of 52.1 (18.6) years. A total of 1550 images of skin lesions were included in the analysis (551 [35.6%] biopsied lesions; 999 [64.4%] control lesions); 286 images (18.6%) were used to train the algorithm, and a further 849 (54.8%) images were missing or unsuitable for analysis. Of the biopsied lesions that were assessed by the algorithm and specialists, 125 (22.7%) were diagnosed as melanoma. Of these, 77 (16.7%) were used for the primary analysis.The algorithm achieved an AUROC of 90.1% (95% CI, 86.3%-94.0%) for biopsied lesions and 95.8% (95% CI, 94.1%-97.6%) for all lesions using iPhone 6s images; an AUROC of 85.8% (95% CI, 81.0%-90.7%) for biopsied lesions and 93.8% (95% CI, 91.4%-96.2%) for all lesions using Galaxy S6 images; and an AUROC of 86.9% (95% CI, 80.8%-93.0%) for biopsied lesions and 91.8% (95% CI, 87.5%-96.1%) for all lesions using DSLR camera images. At 100% sensitivity, the algorithm achieved (continued) Key Points Question How accurate is an artificial intelligence-based melanoma detection algorithm, which analyzes dermoscopic images taken by smartphone and digital single-lens reflex cameras, compared with clinical assessment and histopathological diagnosis? Findings In this diagnostic study, 1550 images of suspicious and benign skin lesions were analyzed by an artificial intelligence algorithm. When compared with histopathological diagnosis, ...

show abstract

Section: Discussionmentioning

confidence: 98%

Assessment of Accuracy of an Artificial Intelligence Algorithm to Detect Melanoma in Images of Skin Lesions

Marsden²,

et al. 2019

View full text Add to dashboard Cite

show abstract

“…The higher ROC areas indicated a better performance of the models. In ROC curve, lower P-value conclude that the model actually does discriminate between two categories and area under curve is higher than the 0.50 [ 22 ]. Calibration and Hasmer–Lemesho goodness of the fit test measures how good the model-estimated probabilities is similar with the observed outcomes and it is usually evaluated through a goodness-of-fit test.…”

Section: Methodsmentioning

confidence: 99%

Knowledge of HIV/AIDS among married women in Bangladesh: analysis of three consecutive multiple indicator cluster surveys (MICS)

et al. 2022

View full text Add to dashboard Cite

Married women have a higher risk of contracting human immunodeficiency virus (HIV) or develop acquired immune deficiency syndrome (AIDS) than men. Knowledge of HIV/AIDS contributes significantly to describing the prevalence and consequences of such virus/disease. The study aimed to investigate the level of HIV/AIDS knowledge and the socio-demographic variables that influence HIV/AIDS knowledge among married women in Bangladesh. We used three waves of Multiple Indicator Cluster Survey (MICS), which included 33,843, 20,727, and 29,724 married women from 2006, 2012, and 2019 MICS. A score was prepared through their interrogation to determine the level of knowledge and logistic regression models were used for analyzing the data. This study found that the prevalence of knowledge level of HIV/AIDS in different questions increased from 55.20% in 2006 to 58.69% in 2019. In our study, respondents having highest education had 4.03 (95% CI 3.50–4.64) times more chance to obtain “High Score” in 2019 MICS which is 5.30 times in 2012 MICS (95% CI 4.41–6.37) and 2.58 times in 2006 MICS (95% CI 2.28–2.93) compared to illiterate married women. Moreover, respondents from urban area were 1.13 times more likely to obtain “High Score” in 2019 MICS which is 1.14 times in 2012 MICS and 1.16 times in 2006 MICS, respectively than the rural married women. This study also found respondent’s age, division, mass media access, and wealth status have played an important role in HIV/AIDS knowledge. Although a significant proportion of women had adequate knowledge of HIV/AIDS, more knowledge is still required to protect against such viruses/diseases. Thus, we advocate for the implementation of educational program in the curriculum, counselling, particularly in rural areas, and mass media access to ensure quality knowledge throughout the country.

show abstract

“…Both of these scores are based on regressions of the Securities and Exchange Commission's Accounting and Auditing Enforcement Releases (AAERs) on characteristics of firms that did and did not receive an AAER. Details about these scores are described in Beneish (1999) and Dechow et al (2011).…”

Section: Example: Fraud Detection Scoresmentioning

confidence: 99%

“…Our new command, , plots PR curves. There are many commands for plotting ROC curves, including the built-in commands and and recent community-contributed commands by Cattaneo, Malighetti, and Spinelli (2017) and Cook and Rajbhandari (2018). There do not appear to be any available commands for creating PR plots.…”

Section: Introductionmentioning

confidence: 99%

“…Second, we compare two commonly used corporate fraud predictors. While both Beneish (1999) and Dechow et al (2011) discuss their out-of-sample predictions, there appears to be little subsequent work that compares these two predictors. Exceptions are Price, Sharp, and Wood (2011) and Cecchini et al (2010), who compare the ROC curves for the M and F scores over the periods 1995-2008 and 1999-2006 (which overlap with our period of 2006-2011), respectively.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

When to consult precision-recall curves

Cook¹,

Ramadas²

2020

The Stata Journal

View full text Add to dashboard Cite

Receiver operating characteristic (ROC) curves are commonly used to evaluate predictions of binary outcomes. When there is a small percentage of items of interest (as would be the case with fraud detection, for example), ROC curves can provide an inflated view of performance. This can cause challenges in determining which set of predictions is better. In this article, we discuss the conditions under which precision-recall curves may be preferable to ROC curves. As an illustrative example, we compare two commonly used fraud predictors (Beneish’s [1999, Financial Analysts Journal 55: 24–36] M score and Dechow et al.’s [2011, Contemporary Accounting Research 28: 17–82] F score) using both ROC and precision-recall curves. To aid the reader with using precision-recall curves, we also introduce the command prcurve to plot them.

show abstract

Heckroccurve: ROC Curves for Selected Samples

Cited by 8 publications

References 6 publications

Assessment of Accuracy of an Artificial Intelligence Algorithm to Detect Melanoma in Images of Skin Lesions

Assessment of Accuracy of an Artificial Intelligence Algorithm to Detect Melanoma in Images of Skin Lesions

Knowledge of HIV/AIDS among married women in Bangladesh: analysis of three consecutive multiple indicator cluster surveys (MICS)

When to consult precision-recall curves

Contact Info

Product

Resources

About