Pretrained transformer framework on pediatric claims data for population specific tasks

Abstract

The adoption of electronic health records (EHR) has become universal during the past decade, which has afforded in-depth data-based research. By learning from the large amount of healthcare data, various data-driven models have been built to predict future events for different medical tasks, such as auto diagnosis and heart-attack prediction. Although EHR is abundant, the population that satisfies specific criteria for learning population-specific tasks is scarce, making it challenging to train data-hungry deep learning models. This study presents the Claim Pre-Training (Claim-PT) framework, a generic pre-training model that first trains on the entire pediatric claims dataset, followed by a discriminative fine-tuning on each population-specific task. The semantic meaning of medical events can be captured in the pre-training stage, and the effective knowledge transfer is completed through the task-aware fine-tuning stage. The fine-tuning process requires minimal parameter modification without changing the model architecture, which mitigates the data scarcity issue and helps train the deep learning model adequately on small patient cohorts. We conducted experiments on a real-world pediatric dataset with more than one million patient records. Experimental results on two downstream tasks demonstrated the effectiveness of our method: our general task-agnostic pre-training framework outperformed tailored task-specific models, achieving more than 10% higher in model performance as compared to baselines. In addition, our framework showed a potential to transfer learned knowledge from one institution to another, which may pave the way for future healthcare model pre-training across institutions.

The shaky foundations of large language models and foundation models for electronic health records

“…Several FEMRs have been trained on insurance claims, which are typically larger in size and more diverse than EMR data but contain less granular information 63 . Examples of claims datasets include Truven Health MarketScan (170 million patients) 64 and Partners For Kids (1.8 million pediatric patients) 65 . In terms of data modalities, most FEMRs are unimodal as they only consider structured codes (e.g., LOINC, SNOMED, etc.).…”

Evaluating the Impact of Health Care Data Completeness for Deep Generative Models

“…Transformers provide a computationally efficient method for learning temporal relationships between data points. Transformers have been applied to solve a variety of predictive health care tasks, including opioid use, 35 coronavirus disease 2019, 36 suicide risk, 37 and asthma exacerbation prediction, 38 as well as an increasing number of generative tasks, such as clinical text generation. 39…”

Self-Supervised Time-to-Event Modeling with Structured Medical Records

“…CLMBR [Steinberg et al, 2021] proposed an autoregressive, "next day" code prediction task to train RNN models. Many authors have presented work using masked language modeling objectives to predict masked or "corrupted" tokens in an input stream [Li et al, 2019, Rasmy et al, 2021b, Pang et al, 2021, Zeng et al, 2022.…”