Online Health Search Via Multidimensional Information Quality Assessment Based on Deep Language Models: Algorithm Development and Validation (Preprint)

Abstract

BACKGROUND The presence of widespread misinformation in Web resources and the limited quality control provided by search engines can lead to serious implications for individuals seeking health advice. OBJECTIVE We aimed to investigate a multi-dimensional information quality assessment model based on deep learning to enhance the reliability of online healthcare information search results. METHODS In this retrospective study, we simulated online health information search scenarios with a topic set of 35 different health-related inquiries and a corpus containing one billion Web documents from the April 2019 snapshot of Common Crawl. Using state-of-the-art pre-trained language models, we inferred the usefulness, supportiveness, and credibility quality dimensions of the retrieved documents for a given search query. RESULTS The usefulness model provided the largest distinction between help and harm compatibility documents with a difference of 0.053. The supportiveness model achieved the best harm compatibility (0.024), while the combination of usefulness, supportiveness, and credibility models achieved the best help and harm compatibility on helpful topics. CONCLUSIONS Our results suggest that integrating automatic ranking models created for specific information quality dimensions can increase the effectiveness of health-related information retrieval for decision-making.

Online health search via multi-dimensional information quality assessment based on deep language models

“…Comparing the compatibilities among default, automatic, and manual retrieval strategies proposed in the literature, results show that the default BM25 model [36] model with -0.022 help-harm compatibility is the lower bound for automatic models on all topics with a mixture of helpful and harmful documents when we put no effort in detecting health misinformation. On the other hand, the help-harm compatibility for the three best automatic models found in the literature [29,31,58] range from 0.040 to 0.043. Automatic models prioritize helpful documents, resulting thus in positive help-harm compatibility.…”

“…In this study, we aim to investigate the impact of usefulness, supportiveness, and credibility dimensions on improving the quality of the retrieved health-related information. We propose an unsupervised multi-dimensional ranking model by utilizing deep learning-based pre-trained language models [31] through transfer learning, which categorizes the quality of the retrieved Web resources by adhering to different information quality dimensions. Specialized quality-oriented ranks obtained by re-ranking components are fused [32] to provide the final ranked list.…”

“…Due to its coverage of the main information quality dimensions used in the scientific literature, we follow the empirical approach proposed in the TREC 2021 challenge, which considers usefulness, supportiveness, and credibility metrics, to propose a multi-dimensional ranking model. Using deep learning-based pre-trained language models [32] through transfer learning and domain adaption approaches, we categorize the retrieved Web resources according to different information quality dimensions. Specialized quality-oriented ranks obtained by re-ranking components are then fused [33] to provide the final ranked list.…”