A comprehensive EHR timeseries pre-training benchmark

McDermott, Matthew B. A.; Nestor, Bret; Kim, Evan; Zhang, Wancong; Goldenberg, Anna; Szolovits, Peter; Ghassemi, Marzyeh

doi:10.1145/3450439.3451877

Cited by 21 publications

(39 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Instead of a 2-step process, we use a single-step process in which all the tasks are jointly trained under a multitask framework. The improved label efficiency of SeqSNR corresponds with the few-shot learning experiments conducted by McDermott et al, 23 which found that MTL pretraining preserved performance at subsampling rates as low as 0.1% of training data.…”

Section: Discussionsupporting

confidence: 54%

“…Besides the low-label scenario, there is also the issue of negative transfer across EHR prediction tasks, which was reported by McDermott et al 23 for most common MIMIC-III endpoints. Our results corroborate these findings, demonstrating that SB MTL tends to show a performance drop relative to ST learning.…”

Section: Discussionmentioning

confidence: 94%

“…We find that the degree of negative transfer varies depending on the index task and is more common when using the full feature set. McDermott et al 23 propose a solution involving multistage training (MTL pretraining followed by ST fine tuning); however, this carries the risk of catastrophic forgetting. 46 We propose SeqSNR as an alternative approach for mitigating negative transfer via soft parameter sharing, which allows the network to optimize for cross-learning between related tasks.…”

Section: Discussionmentioning

confidence: 99%

“…They demonstrated that MTL provided consistent, though modest, improvements over ST discriminative performance for 3 of 4 tasks, concluding that it serves as an important regularizer. More recently, McDermott et al 23 used the same ICU dataset to show that only highly related tasks result in effective cross-learning, with a high risk for negative transfer (reduced performance with MTL) when certain task combinations were used. Negative transfer happens when dissimilar tasks introduce conflicting inductive biases in the shared layers thereby hurting performance.…”

Section: Introductionmentioning

confidence: 99%

“…Negative transfer happens when dissimilar tasks introduce conflicting inductive biases in the shared layers thereby hurting performance. 28 Furthermore, McDermott et al 23 found that MTL pretraining with finetuning on a new task significantly outperforms ST in few-shot learning (scarce training data) scenarios, especially on continuous (rolling) outcome prediction tasks.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Multitask prediction of organ dysfunction in the intensive care unit using sequential subnetwork routing

Roy

Mincu

Loreaux

et al. 2021

Journal of the American Medical Informatics Association

View full text Add to dashboard Cite

Objective Multitask learning (MTL) using electronic health records allows concurrent prediction of multiple endpoints. MTL has shown promise in improving model performance and training efficiency; however, it often suffers from negative transfer – impaired learning if tasks are not appropriately selected. We introduce a sequential subnetwork routing (SeqSNR) architecture that uses soft parameter sharing to find related tasks and encourage cross-learning between them. Materials and Methods Using the MIMIC-III (Medical Information Mart for Intensive Care-III) dataset, we train deep neural network models to predict the onset of 6 endpoints including specific organ dysfunctions and general clinical outcomes: acute kidney injury, continuous renal replacement therapy, mechanical ventilation, vasoactive medications, mortality, and length of stay. We compare single-task (ST) models with naive multitask and SeqSNR in terms of discriminative performance and label efficiency. Results SeqSNR showed a modest yet statistically significant performance boost across 4 of 6 tasks compared with ST and naive multitasking. When the size of the training dataset was reduced for a given task (label efficiency), SeqSNR outperformed ST for all cases showing an average area under the precision-recall curve boost of 2.1%, 2.9%, and 2.1% for tasks using 1%, 5%, and 10% of labels, respectively. Conclusions The SeqSNR architecture shows superior label efficiency compared with ST and naive multitasking, suggesting utility in scenarios in which endpoint labels are difficult to ascertain.

show abstract

Section: Discussionsupporting

confidence: 54%

Section: Discussionmentioning

confidence: 94%

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Multitask prediction of organ dysfunction in the intensive care unit using sequential subnetwork routing

Roy

Mincu

Loreaux

et al. 2021

Journal of the American Medical Informatics Association

View full text Add to dashboard Cite

show abstract

Pre-training in Medical Data: A Survey

et al. 2023

View full text Add to dashboard Cite

Medical data refers to health-related information associated with regular patient care or as part of a clinical trial program. There are many categories of such data, such as clinical imaging data, bio-signal data, electronic health records (EHR), and multi-modality medical data. With the development of deep neural networks in the last decade, the emerging pre-training paradigm has become dominant in that it has significantly improved machine learning methods’ performance in a data-limited scenario. In recent years, studies of pre-training in the medical domain have achieved significant progress. To summarize these technology advancements, this work provides a comprehensive survey of recent advances for pre-training on several major types of medical data. In this survey, we summarize a large number of related publications and the existing benchmarking in the medical domain. Especially, the survey briefly describes how some pre-training methods are applied to or developed for medical data. From a data-driven perspective, we examine the extensive use of pre-training in many medical scenarios. Moreover, based on the summary of recent pre-training studies, we identify several challenges in this field to provide insights for future studies.

show abstract