Concept Recognition as a Machine Translation Problem

Abstract

BackgroundAutomated assignment of specific ontology concepts to mentions in text is a critical task in biomedical natural language processing, and the subject of many open shared tasks. Although the current state of the art involves the use of neural network language models as a post-processing step, the very large number of ontology classes to be recognized and the limited amount of gold-standard training data has impeded the creation of end-to-end systems based entirely on machine learning. Recently, Hailu et al. recast the concept recognition problem as a type of machine translation and demonstrated that sequence-to-sequence machine learning models had the potential to outperform multi-class classification approaches. Here we systematically characterize the factors that contribute to the accuracy and efficiency of several approaches to sequence-to-sequence machine learning.ResultsWe report on our extensive studies of alternative methods and hyperparameter selections. The results not only identify the best-performing systems and parameters across a wide variety of ontologies but also illuminate about the widely varying resource requirements and hyperparameter robustness of alternative approaches. Analysis of the strengths and weaknesses of such systems suggest promising avenues for future improvements as well as design choices that can increase computational efficiency with small costs in performance. Bidirectional Encoder Representations from Transformers for Biomedical Text Mining (BioBERT) for span detection (as previously found) along with the Open-source Toolkit for Neural Machine Translation (OpenNMT) for concept normalization achieve state-of-the-art performance for most ontologies in CRAFT Corpus. This approach uses substantially fewer computational resources, including hardware, memory, and time than several alternative approaches.ConclusionsMachine translation is a promising avenue for fully machine-learning-based concept recognition that achieves state-of-the-art results on the CRAFT Corpus, evaluated via a direct comparison to previous results from the 2019 CRAFT Shared Task. Experiments illuminating the reasons for the surprisingly good performance of sequence-to-sequence methods targeting ontology identifiers suggest that further progress may be possible by mapping to alternative target concept representations. All code and models can be found at: https://github.com/UCDenver-ccp/Concept-Recognition-as-Translation.

Identifying and classifying goals for scientific knowledge

“…Identifying these subjects will then allow for future work to identify the specific biomedical concepts that these statements of ignorance include using existing automatic concept recognition tools (e.g. Boguslav et al , 2021 ). In the example above, the brackets < > signify the subject of the statement of ignorance, i.e.…”

“…The word-level classification tasks are to determine which words in a sentence indicate a lexical cue (binary task), and specifically for which taxonomy category (multi-classification). As our taxonomy is very similar to an ontology, we can use prior methods for concept recognition ( Boguslav et al , 2021 ) that explores different algorithms over many different ontologies. In particular, we make use of the best-performing span detection algorithms, namely CRF ( Lafferty et al , 2001 ) and BioBERT ( Lee et al , 2020 ), to determine the words in all lexical cues given an article.…”

“…To then determine what the lexical cue is, we re-assemble the BIO- tags by finding the B tags (single word lexical cues), combining the words with B and I in a row (multi-word lexical cues), ignoring the O labels (not lexical cues), and by combining B, I, and O- (discontinuous lexical cues). A more thorough discussion of BIO- tags can be found in Boguslav et al (2021 ).…”

“…For the word-level multi-classification task, we explore two different methods. First, we created a model (CRF and BioBERT) for each ignorance taxonomy category following the previous work ( Boguslav et al , 2021 ). At the same time, however, BIO- tags have the capability to encode which category the word is from (i.e.…”