The NILS Study Protocol: A Retrospective Validation Study of an Artificial Neural Network Based Preoperative Decision-Making Tool for Noninvasive Lymph Node Staging in Women with Primary Breast Cancer (ISRCTN14341750)

Skarping, Ida; Dihge, Looket; Bendahl, Pär‐Ola; Huss, Linnea; Ellbrant, Julia; Ohlsson, Mattias; Rydén, Lisa

doi:10.3390/diagnostics12030582

Cited by 8 publications

(8 citation statements)

References 40 publications

(59 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although the NILS prediction tool has been developed for clinical use following the framework of the Tripod checklist (53) to increase the methodological consistency and quality of the prediction models, the true validity of the model should be assessed in a fully independent dataset. To address this, a validation study is currently being conducted based on two external cohorts that assess the geographic and temporal validity of the NILS prediction tool (ISRCTN 14341750) (29).…”

Section: Discussionmentioning

confidence: 99%

“…To ensure quality assurance for future clinical preoperative applicability, the NILS predictive tool is now applied in a clinical trial for validation purposes without interfering with the current breast cancer workflow (ISRCTN registry, study ID 14341750) (29). The prediction tool or "calculator" is currently accessible within the research program by entering login credentials.…”

Section: The Interactive Web-based Nils Interfacementioning

confidence: 99%

See 1 more Smart Citation

The implementation of NILS: A web-based artificial neural network decision support tool for noninvasive lymph node staging in breast cancer

et al. 2023

Self Cite

View full text Add to dashboard Cite

ObjectiveTo implement artificial neural network (ANN) algorithms for noninvasive lymph node staging (NILS) to a decision support tool and facilitate the option to omit surgical axillary staging in breast cancer patients with low-risk of nodal metastasis.MethodsThe NILS tool is a further development of an ANN prototype for the prediction of nodal status. Training and internal validation of the original algorithm included 15 clinical and tumor-related variables from a consecutive cohort of 800 breast cancer cases. The updated NILS tool included 10 top-ranked input variables from the original prototype. A workflow with four ANN pathways was additionally developed to allow different combinations of missing preoperative input values. Predictive performances were assessed by area under the receiver operating characteristics curves (AUC) and sensitivity/specificity values at defined cut-points. Clinical utility was presented by estimating possible sentinel lymph node biopsy (SLNB) reduction rates. The principles of user-centered design were applied to develop an interactive web-interface to predict the patient’s probability of healthy lymph nodes. A technical validation of the interface was performed using data from 100 test patients selected to cover all combinations of missing histopathological input values.ResultsANN algorithms for the prediction of nodal status have been implemented into the web-based NILS tool for personalized, noninvasive nodal staging in breast cancer. The estimated probability of healthy lymph nodes using the interface showed a complete concordance with estimations from the reference algorithm except in two cases that had been wrongly included (ineligible for the technical validation). NILS predictive performance to distinguish node-negative from node-positive disease, also with missing values, displayed AUC ranged from 0.718 (95% CI, 0.687-0.748) to 0.735 (95% CI, 0.704-0.764), with good calibration. Sensitivity 90% and specificity 34% were demonstrated. The potential to abstain from axillary surgery was observed in 26% of patients using the NILS tool, acknowledging a false negative rate of 10%, which is clinically accepted for the standard SLNB technique.ConclusionsThe implementation of NILS into a web-interface are expected to provide the health care with decision support and facilitate preoperative identification of patients who could be good candidates to avoid unnecessary surgical axillary staging.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: The Interactive Web-based Nils Interfacementioning

confidence: 99%

The implementation of NILS: A web-based artificial neural network decision support tool for noninvasive lymph node staging in breast cancer

et al. 2023

Self Cite

View full text Add to dashboard Cite

show abstract

“…First, the models were developed using a combination of variables available before and after surgery to externally validate the original NILS model [7] which is based on preoperative and postoperative variables. Further development of the NILS concept is an ongoing validation of the NILS model, using exclusively preoperative variables [29]. Second, the generalizability of the LVI and N models developed in Cohort I can be affected by the smaller size of the development cohorts which can be considered a weakness of the study.…”

Section: Discussionmentioning

confidence: 99%

“…The final cohort size was 525 patients (Figure S2, Appendix ). The data extraction for Cohort III was validated and monitored by an independent researcher according to a specific quality assurance protocol [29]. The sample size calculation for validating the NILS concept has been published previously [29].…”

Section: Methodsmentioning

confidence: 99%

Noninvasive Staging of Lymph Node Status (NILS) in Breast Cancer using Machine Learning: External Validation and Further Model Development

Hjärtström

Dihge

Bendahl

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Background:Most patients diagnosed with breast cancer present with node–negative disease. Sentinel lymph node biopsy (SLNB) is routinely used to stage the axilla, leaving patients with healthy axillary lymph nodes without therapeutic effects but at risk of morbidities from the intervention. Numerous studies have developed nodal (N) status prediction models for noninvasive axillary staging using postoperative data or imaging features that are not part of the diagnostic workup. Lymphovascular invasion (LVI) is a top-ranked predictor of N metastasis; however, it is challenging to assess preoperatively.Objective:To externally validate a multilayer perceptron (MLP) model for noninvasive lymph node staging (NILS) in a large population-based register cohort (n=18 633) while developing a new N MLP in the same cohort. Data were extracted from the Swedish National Quality Register for Breast Cancer (NKBC), 2014–2017, comprising only routinely and preoperatively available documented clinicopathological variables. Furthermore, we aimed to develop and validate an LVI MLP to predict missing values of LVI status to increase the preoperative feasibility of the original NILS model.Methods:Three non-overlapping cohorts were used for model development and validation. Four N MLPs and one LVI MLP were developed using 11–12 routinely available predictors. Three N models were used to account for the different availabilities of LVI status in the cohorts and external validation in NKBC. The fourth N model was developed for 80% of NKBC cases (n=14 906) and validated in the remaining 20% (n=3727). Three alternatives for imputing missing values of the LVI status were compared using the LVI model. The discriminatory capacity was evaluated by validation area under the receiver operating characteristics curve (AUC) in three of the N models. The clinical feasibility of the models was evaluated using calibration and decision curve analyses.Results:External validation of the original NILS model was performed in NKBC (AUC 0.699, 95% CI: 0.690–0.708) with good calibration and the potential of sparing 16% of patients with node-negative disease from SLNB. The LVI model was externally validated (AUC 0.747, 95% CI: 0.694–0.799) with good calibration but did not improve the discriminatory performance of the N models. The new N model was developed in NKBC without information on LVI (AUC 0.709, 95% CI: 0.688–0.729) with excellent calibration in the hold-out internal validation cohort, resulting in the potential omission of 24% of patients from unnecessary SLNB.Conclusions:The NILS model was externally validated in NKBC, where the imputation of LVI status did not improve the model′s discriminatory performance. A new N model demonstrated the feasibility of using register data comprising only the variables available in the preoperative setting for NILS using machine learning. Future steps include ongoing preoperative validation of the NILS model and extending the model with, for example, mammography images.

show abstract

“…We developed an artificial neural network (ANN) model, the noninvasive lymph node staging (NILS) prediction model, producing a probability score of a patient being node negative [ 12 ]. A validation study is currently being conducted (ISRCTN14341750), and the study protocol has been published [ 13 ]. Before broadly implementing new health care technologies and decision tools such as the NILS model, careful considerations of costs and health consequences are needed in addition to information on efficacy and safety [ 14 , 15 ].…”

Section: Introductionmentioning

confidence: 99%

The implementation of a noninvasive lymph node staging (NILS) preoperative prediction model is cost effective in primary breast cancer

Skarping

Nilsson

Dihge

et al. 2022

Breast Cancer Res Treat

Self Cite

View full text Add to dashboard Cite

Purpose The need for sentinel lymph node biopsy (SLNB) in clinically node-negative (cN0) patients is currently questioned. Our objective was to investigate the cost-effectiveness of a preoperative noninvasive lymph node staging (NILS) model (an artificial neural network model) for predicting pathological nodal status in patients with cN0 breast cancer (BC). Methods A health-economic decision-analytic model was developed to evaluate the utility of the NILS model in reducing the proportion of cN0 patients with low predicted risk undergoing SLNB. The model used information from a national registry and published studies, and three sensitivity/specificity scenarios of the NILS model were evaluated. Subgroup analysis explored the outcomes of breast-conserving surgery (BCS) or mastectomy. The results are presented as cost (€) and quality-adjusted life years (QALYs) per 1000 patients. Results All three scenarios of the NILS model reduced total costs (–€93,244 to –€398,941 per 1000 patients). The overall health benefit allowing for the impact of SLNB complications was a net health gain (7.0–26.9 QALYs per 1000 patients). Sensitivity analyses disregarding reduced quality of life from lymphedema showed a small loss in total health benefits (0.4–4.0 QALYs per 1000 patients) because of the reduction in total life years (0.6–6.5 life years per 1000 patients) after reduced adjuvant treatment. Subgroup analyses showed greater cost reductions and QALY gains in patients undergoing BCS. Conclusion Implementing the NILS model to identify patients with low risk for nodal metastases was associated with substantial cost reductions and likely overall health gains, especially in patients undergoing BCS.

show abstract

The NILS Study Protocol: A Retrospective Validation Study of an Artificial Neural Network Based Preoperative Decision-Making Tool for Noninvasive Lymph Node Staging in Women with Primary Breast Cancer (ISRCTN14341750)

Cited by 8 publications

References 40 publications

The implementation of NILS: A web-based artificial neural network decision support tool for noninvasive lymph node staging in breast cancer

The implementation of NILS: A web-based artificial neural network decision support tool for noninvasive lymph node staging in breast cancer

Noninvasive Staging of Lymph Node Status (NILS) in Breast Cancer using Machine Learning: External Validation and Further Model Development

The implementation of a noninvasive lymph node staging (NILS) preoperative prediction model is cost effective in primary breast cancer

Contact Info

Product

Resources

About