Sequence-to-Set Semantic Tagging for Complex Query Reformulation and Automated Text Categorization in Biomedical IR using Self-Attention

Abstract

Novel contexts, comprising a set of terms referring to one or more concepts, may often arise in complex querying scenarios such as in evidence-based medicine (EBM) involving biomedical literature. These may not explicitly refer to entities or canonical concept forms occurring in a fact-based knowledge source, e.g. the UMLS ontology. Moreover, hidden associations between related concepts meaningful in the current context, may not exist within a single document, but across documents in the collection. Predicting semantic concept tags of documents can therefore serve to associate documents related in unseen contexts, or categorize them, in information filtering or retrieval scenarios. Thus, inspired by the success of sequence-to-sequence neural models, we develop a novel sequence-to-set framework with attention, for learning document representations in a unique unsupervised setting, using no human-annotated document labels or external knowledge resources and only corpus-derived term statistics to drive the training. This can effect term transfer within a corpus for semantically tagging a large collection of documents. Our sequence-to-set modeling approach to predict semantic tags , gives to the best of our knowledge, the state-of-theart for both, an unsupervised query expansion (QE) task for the TREC CDS 2016 challenge dataset when evaluated on an Okapi BM25based document retrieval system; and also over the MLTM system baseline (Soleimani and Miller, 2016), for both supervised and semi-supervised multi-label prediction tasks with del.icio.us and Ohsumed datasets. We make our code and data publicly available 1 .

Literature-Augmented Clinical Outcome Prediction

“…For more details, see Appendix B. Our bi-encoder achieves mean precision@10 score of 45.67 on TREC 2016 data in 5-fold cross-validation, comparable to stateof-the-art results (Das et al, 2020).…”

“…This embedding is run through a linear layer to produce a relevance score, trained using crossentropy loss with respect to document relevance labels from the TREC 2016 dataset. Our crossencoder achieves a mean precision@10 score of 48.33 on TREC 2016 in 5-fold cross-validation, which is also comparable to state-of-the-art performance on TREC CDS 2016 (Das et al, 2020).…”

Comparing Human Text Classification Performance and Explainability with Large Language and Machine Learning Models Using Eye-Tracking

“…Another reason for limited prediction accuracy in traditional ML models is the lack of usage of semantic and syntactic information of the text. Recent advancements in natural language processing (NLP) such as word embeddings, transformers, and large language models (LLMs) have better capabilities in terms of understanding and utilization of syntactic and contextual semantic information of the text [1,2,3], which is likely to improve the classi cation performance.…”