Lowry, “Tiger” Ted

A Novel Question-Answering Framework for Automated Abstract Screening Using Large Language Models

“…We utilize cosine similarity to gauge the congruence between each abstract and the posed questions, thus augmenting the relevance assessment for each selection criterion. For each Yes/No query, the cosine similarity between the query and the abstract is computed based on their GPT-3.5 embeddings [61], and averaged with the original answer score, producing K re-ranked scores. The re-ranked score for each answer is defined by where cos ( d i , Q k ) is the cosine similarity between (the abstract of) the candidate study d i and one selection criterion question Q k .…”

“…Several methods are available for embedding selection criteria and abstracts. Given the emphasis on the capabilities of LLMs in this paper, GPT Embeddings [63] are chosen. Two approaches to re-ranking are defined: Abstract-level re-ranking and answer-level re-ranking.…”

“…To enhance screening further, one significant contribution of the current paper involves re-ranking candidate studies based on how well abstracts are semantically aligned with the selection criteria. For each Yes/No question, the cosine similarity between the question (selection criterion) and the abstract is computed based on their text embeddings encoded by an LLM [63], and averaged with the original answer score, producing K re-ranked scores, defined by where cos ( d i , Q k ) is the cosine similarity between (the abstract of) a candidate study d i and one selection criterion question Q k .…”

“…To comprehensively assess performance, we implemented two IR baselines of our own. One is cosine similarity between selection criteria and abstract based on GPT embeddings [63], named GPT Cos Sim Criteria. The other is the classical IR approach BM25 [66], using selection criteria as a query to rank candidate studies.…”