Meta-analysis of the strength of exploratory suicide prediction models; from clinicians to computers

Abstract

Background Suicide prediction models have been formulated in a variety of ways and are heterogeneous in the strength of their predictions. Machine learning has been a proposed as a way of improving suicide predictions by incorporating more suicide risk factors. Aims To determine whether machine learning and the number of suicide risk factors included in suicide prediction models are associated with the strength of the resulting predictions. Method Random-effect meta-analysis of exploratory suicide prediction models constructed by combining two or more suicide risk factors or using clinical judgement (Prospero Registration CRD42017059665). Studies were located by searching for papers indexed in PubMed before 15 August 2020 with the term suicid* in the title. Results In total, 86 papers reported 102 suicide prediction models and included 20 210 411 people and 106 902 suicides. The pooled odds ratio was 7.7 (95% CI 6.7–8.8) with high between-study heterogeneity (I2 = 99.5). Machine learning was associated with a non-significantly higher odds ratio of 11.6 (95% CI 6.0–22.3) and clinical judgement with a non-significantly lower odds ratio of 4.7 (95% CI 2.1–10.9). Models including a larger number of suicide risk factors had a higher odds ratio when machine-learning studies were included (P = 0.02). Among non-machine-learning studies, suicide prediction models including fewer risk factors performed just as well as those including more risk factors. Conclusions Machine learning might have the potential to improve the performance of suicide prediction models by increasing the number of included suicide risk factors but its superiority over other methods is unproven.

The ‘double whammy’ of low prevalence in clinical risk prediction

“…One review suggested, albeit with uncertainty, that the best performing machine learning methods in suicide prediction might bring performance equivalent to an odds ratio of 12, much less than what would be required under the second condition outlined above. [19] Across a range of epidemiological research areas, relative risks and likelihood ratios of the required magnitude are rarely observed and cannot be realistically expected. [20] While these relative measures may help in developing conceptual understanding of the general issues relating to low prevalence, we recommend that they be presented alongside absolute measures of risk when they are reported in particular studies.…”

Use of Artificial Intelligence-Based Strategies for Assessing Suicidal Behavior and Mental Illness: A Literature Review

“…A combination of NLP and ML was used in four studies [11][12][13][14]. ML as the exclusive AI technique was used in four studies [15][16][17][18], artificial neural network (ANN) was used in three studies [19][20][21], and NLP in two studies [9,22].…”

“…A study by Corke et al (2021) mentioned that by increasing the number of suicide risk factors, ML can improve the performance of suicide risk prediction. However, its superiority over other methods has yet to be proven [17].…”

“…A study by Corke et al (2021) mentioned that by increasing the number of suicide risk factors, ML can improve the performance of suicide risk prediction. However, its superiority over other methods has yet to be proven [17]. AI learning models had an overall agreement of 83.5% with the traditional models with ML displaying significantly better results in identifying self-harm visits [18].…”

“…The risk factors for determining suicidal ideation include parameters such as the history of continuous depressed mood, stress awareness, alcohol abuse, educational status, unmet medical needs, socio-economic background, and severity of depression. Data has demonstrated that ML models (such as LogitBoost, and ANN) were more effective than the traditional LR model (0.867) in assessing the risk of suicide [17].…”