PLM-ICD: Automatic ICD Coding with Pretrained Language Models

Abstract

Automatically classifying electronic health records (EHRs) into diagnostic codes has been challenging to the NLP community. State-ofthe-art methods treated this problem as a multilabel classification problem and proposed various architectures to model this problem. However, these systems did not leverage the superb performance of pretrained language models, which achieved superb performance on natural language understanding tasks. Prior work has shown that pretrained language models underperformed on this task with the regular finetuning scheme. Therefore, this paper aims at analyzing the causes of the underperformance and developing a framework for automatic ICD coding with pretrained language models. We spotted three main issues through the experiments: 1) large label space, 2) long input sequences, and 3) domain mismatch between pretraining and fine-tuning. We propose PLM-ICD, a framework that tackles the challenges with various strategies. The experimental results show that our proposed framework can overcome the challenges and achieves state-ofthe-art performance in terms of multiple metrics on the benchmark MIMIC data. 1

Aiding ICD-10 Encoding of Clinical Health Records Using Improved Text Cosine Similarity and PLM-ICD

“…Huang, et al (2022) [12] proposed a PLM to predict ICD codes (PLM-ICD) that tackles the challenges of previous pretrained models, namely the use of domain-specific pretraining, segment pooling for the long input sequence problem, and level attention for the large label set problem. Experiments were conducted over MIMIC-II and MIMIC-III datasets.…”

Kohonen neural network and symbiotic-organism search algorithm for intrusion detection of network viruses

“…RPGNet is trained using Reinforcement Learning (RL) in an adversarial fashion. [17] PLM-ICD use Pretrained Language Models to extract the contextualuzed representations, it spotted three main issues through the experiments: 1) large label space, 2) long input sequences, and 3) domain mismatch between pretraining and fine-tuning. It's the sota method for ICD classification.…”

“…We report a variety of metrics that are commonly used by previous studies on this task [16,9,17], including micro-averaged and macro-averaged metrics. Micro-averaged metrics are calculated by treating each sample as a separate prediction, while macro-averaged metrics are calculated by averaging metrics computed per label.…”

Transforming clinical trials: the emerging roles of large language models

“…LLMs possess the ability to label and classify free form text, an attribute that can be integrated into the coding process itself. For instance, studies have demonstrated the feasibility of employing LLMs to automatically classify electronic health records into International Classification of Diseases codes [ 17 ]. Furthermore, LLMs have been successfully employed to classify free text in regulatory documents into specific predefined sections [ 18 ], and to code text data that require deductive analysis [ 19 ], addressing a long-standing challenge in clinical research.…”