Data Mining in Clinical Trial Text: Transformers for Classification and Question Answering Tasks

Abstract

This research on data extraction methods applies recent advances in natural language processing to evidence synthesis based on medical texts. Texts of interest include abstracts of clinical trials in English and in multilingual contexts. The main focus is on information characterized via the Population, Intervention, Comparator, and Outcome (PICO) framework, but data extraction is not limited to these fields. Recent neural network architectures based on transformers show capacities for transfer learning and increased performance on downstream natural language processing tasks such as universal reading comprehension, brought forward by this architecture's use of contextualized word embeddings and self-attention mechanisms. This paper contributes to solving problems related to ambiguity in PICO sentence prediction tasks, as well as highlighting how annotations for training named entity recognition systems are used to train a high-performing, but nevertheless flexible architecture for question answering in systematic review automation. Additionally, it demonstrates how the problem of insufficient amounts of training annotations for PICO entity extraction is tackled by augmentation. All models in this paper were created with the aim to support systematic review (semi)automation. They achieve high F1 scores, and demonstrate the feasibility of applying transformer-based classification methods to support data mining in the biomedical literature.

Data extraction methods for systematic review (semi)automation: A living systematic review

“…39 In another example, the authors reported using Google Colab GPUs, along with estimates of computing time for different training settings. 68 3.4.2.4 Is the source code available?…”

“…A small number of publications did a real-life assessment, where the data extraction algorithm was applied to different, unlabelled, and often much larger datasets or tested while conducting actual systematic reviews. 30,33,39,42,68,71,72 3.4.3.2 Are basic metrics reported (true/false positives and negatives)?…”

“…27 It is challenging to find the best set of hyperparameters 74 and to adjust precision and recall trade-offs to maximise the utility of a system while being transparent about the number of data points that might be missed when increasing system precision to save work for a human reviewer. 39,68,71 4.4 Explainability and interpretability of data extraction systems The neural networks or machine-learning models from publications included in this review learn to classify and extract data by adjusting numerical weights and by applying mathematical functions to these sets of weights. The decisionmaking process behind the classification of a sentence or an entity is therefore comparable with a black box, because it is very hard to comprehend how, or why the model made its predictions.…”

Scaling Systematic Literature Reviews with Machine Learning Pipelines

“…A sentence-based classifier can be used to label sentences as containing target information, and has been the method of extraction used in previous work Kiritchenko et al, 2010;Schmidt et al, 2020). But this does not fit in well with the desired tabular output for all target fields: the same information can appear in multiple sentences and the same sentence can contain multiple targets.…”

“…Kiritchenko et al (2010) create an extraction system that identifies sentences and then post-processes them to extract data, but operate only on structured HTML & XML. Schmidt et al (2020) apply fine-tuned BERT-based Transformers to the task of to sentence classification for semi-automated systematic review. Goswami et al (2019) build a PDF retrieval system for systematic reviews for psychology and use a random forest classifier to identify sentences for extraction.…”