Three-level Hierarchical Transformer Networks for Long-sequence and Multiple Clinical Documents Classification

Si, Yuqi; Roberts, Kirk

doi:10.48550/arxiv.2104.08444

Cited by 2 publications

(3 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Bidirectional Encoder Representations from Transformers (BERT), is a state-of-the-art large language model that was pre-trained on millions of English documents and has been shown to achieve high performances in facilitating many downstream NLP tasks (termed ne-turning) including named entity recognition 16,17 and document classi cation 18,19 . By taking labelled referrals as input, we developed an end-to-end model by ne-tuning the pre-trained BERT-base-uncased model 3 for text classi cation.…”

Section: Bert End-to-end Modelmentioning

confidence: 99%

Towards automated dermatology triage: deep learning and knowledge-driven approaches

Wang,

Kloczko,

Altayeb

et al. 2023

Preprint

View full text Add to dashboard Cite

Background The current triage process in the National Health Service (NHS) requires secondary care clinicians to manually read every General Practitioner’s (GP) referral letter, which makes the process time-consuming with associated high costs. Artificial Intelligence (AI) algorithms can be adopted to accelerate this process and reduce the required resources. Objectives To design AI models that can automatically stratify GP referrals to routine and non-routine categories, and to evaluate different AI algorithms against the current manual triage process. Methods We developed and evaluated multiple AI models to triage dermatology referrals into binary outcomes, i.e., routine or non-routine. The models ranged from a totally data-driven (deep learning) approach to different levels of knowledge-enriched approaches: 1) a transfer learning approach using a pre-trained large language model; 2) a deep learning model using Long Short-Term Memory architecture, enriched with key concepts from referral guidelines; and 3) a knowledge-driven model utilising the semantics of key concepts from clinical guidelines and customised clinicians’ dictionaries. Random oversampling and data augmentation were used for dealing with highly imbalanced triage classes. All referrals were individually triaged by two dermatologists and then compared against the results generated from AI-assisted triage models. Performances were evaluated using Precision-Recall Area Under Curve (PR-AUC) and Receiver Operating Characteristic Area Under Curve (ROC-AUC). Results 268 GP referrals to adult dermatology services were included. The knowledge-driven approach achieved the best performance (micro average PR-AUC of 0.907±0.006, ROC-AUC of 0.720 ± 0.010) compared to the baseline end-to-end deep learning model (micro average PR-AUC of 0.823±0.038, ROC-AUC of 0.616 ± 0.096) and the Long Short-Term Memory model (0.867±0.013, 0.600 ± 0.071). Imbalance preprocessing methods improved the model performance in some cases but not to a significant level. Combining all types of domain knowledge in AI models outperformed any subsets of these knowledge inputs. Conclusions The knowledge-enhanced AI approach showed promising results in achieving triage outcomes comparable to manual outcomes despite the limited data input from the referrals. AI-assisted triage has the potential to make the triaging process less time-consuming and more cost-effective, whilst retaining accuracy.

show abstract

Section: Bert End-to-end Modelmentioning

confidence: 99%

Towards automated dermatology triage: deep learning and knowledge-driven approaches

Wang,

Kloczko,

Altayeb

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Gao et al (2021) [11] presents evidence showing BERT-based models under-perform in clinical text classification tasks with long input data, such as MIMIC-III [15], when compared to a CNN trained on word embeddings that can process the complete input sequences. Si and Roberts (2021) [24] presents an alternative system to overcome the issue of long documents, where transformer-based encoders are used to learn from words to sentences, sentences to notes and notes to patients progressively. This transformer-based hierarchical attention networks system presents SOTA methods for in-hospital mortality prediction and phenotype predictions using MIMIC-III.…”

Section: Related Workmentioning

confidence: 99%

“…This transformer-based hierarchical attention networks system presents SOTA methods for in-hospital mortality prediction and phenotype predictions using MIMIC-III. However, it requires considerable computational resources [24]. Chalkidis et al (2020) [6] proposes a similar hierarchical version using SCI-BERT to deal with long documents for predicting medical codes from MIMIC III.…”

Section: Related Workmentioning

confidence: 99%

Improving Predictions of Tail-end Labels using Concatenated BioMed-Transformers for Long Medical Documents

Yogarajan¹,

Pfahringer²,

Smith³

et al. 2021

Preprint

View full text Add to dashboard Cite

Multi-label learning predicts a subset of labels from a given label set for an unseen instance while considering label correlations. A known challenge with multi-label classification is the long-tailed distribution of labels. Many studies focus on improving the overall predictions of the model and thus do not prioritise tail-end labels. Improving the tail-end label predictions in multi-label classifications of medical text enables the potential to understand patients better and improve care. The knowledge gained by one or more infrequent labels can impact the cause of medical decisions and treatment plans. This research presents variations of concatenated domain-specific language models, including multi-BioMed-Transformers, to achieve two primary goals. First, to improve F1 scores of infrequent labels across multi-label problems, especially with long-tail labels; second, to handle long medical text and multi-sourced electronic health records (EHRs), a challenging task for standard transformers designed to work on short input sequences. A vital contribution of this research is new state-of-the-art (SOTA) results obtained using TransformerXL for predicting medical codes. A variety of experiments are performed on the Medical Information Mart for Intensive Care (MIMIC-III) database. Results show that concatenated BioMed-Transformers outperform standard transformers in terms of overall micro and macro F1 scores and individual F1 scores of tail-end labels, while incurring lower training times than existing transformer-based solutions for long input sequences.

show abstract

Three-level Hierarchical Transformer Networks for Long-sequence and Multiple Clinical Documents Classification

Cited by 2 publications

References 25 publications

Towards automated dermatology triage: deep learning and knowledge-driven approaches

Towards automated dermatology triage: deep learning and knowledge-driven approaches

Improving Predictions of Tail-end Labels using Concatenated BioMed-Transformers for Long Medical Documents

Contact Info

Product

Resources

About