SciLens: Evaluating the Quality of Scientific News Articles Using Social Media and Scientific Literature Indicators

Abstract

This paper describes, develops, and validates SciLens, a method to evaluate the quality of scientific news articles. The starting point for our work are structured methodologies that define a series of quality aspects for manually evaluating news. Based on these aspects, we describe a series of indicators of news quality. According to our experiments, these indicators help non-experts evaluate more accurately the quality of a scientific news article, compared to nonexperts that do not have access to these indicators. Furthermore, SciLens can also be used to produce a completely automated quality score for an article, which agrees more with expert evaluators than manual evaluations done by non-experts. One of the main elements of SciLens is the focus on both content and context of articles, where context is provided by (1) explicit and implicit references on the article to scientific literature, and (2) reactions in social media referencing the article. We show that both contextual elements can be valuable sources of information for determining article quality. The validation of SciLens, done through a combination of expert and non-expert annotation, demonstrates its effectiveness for both semi-automatic and automatic quality evaluation of scientific news.

Open science saves lives: lessons from the COVID-19 pandemic

“…During the COVID-19 crisis, the United Nations Educational, Scientific and Cultural Organisation (UNESCO) has organized a series of 'webinars' tackling the importance of journalists' scientific literacy [117]. However, this process is still ongoing and -since many examples of problematic dissemination of scientific news are still seen -some researchers have proposed indicators to evaluate the scientific accuracy of news articles [118]. Some scientists are also grasping these issues by writings blogs directly addressing scientific integrity and exposing examples of bad science in the scientific literature (see e.g., the work of Elisabeth Bik [119]), directly motivating a more responsible science communication.…”

Open Science Saves Lives: Lessons from the COVID-19 Pandemic

“…During the COVID-19 crisis, the United Nations Educational, Scientific and Cultural Organisation (UNESCO) has organized a series of 'webinars' tackling the importance of journalists' scientific literacy [89]. However, this process is still ongoing and -since many examples of problematic dissemination of scientific news are still seen -some researchers have proposed indicators to evaluate the scientific accuracy of news articles [90].…”

“…During the COVID-19 crisis, the United Nations Educational, Scientific and Cultural Organisation (UNESCO) has organized a series of ‘webinars’ tackling the importance of journalists’ scientific literacy [121]. However, this process is still ongoing and – since many examples of problematic dissemination of scientific news are still seen – some researchers have proposed indicators to evaluate the scientific accuracy of news articles [122]. Some scientists are also grasping these issues by writings blogs directly addressing scientific integrity and exposing examples of bad science in the scientific literature (see e.g., the work of Elisabeth Bik [123]), directly motivating a more responsible science communication.…”

Scilens news platform

“…According to a thorough experimental evaluation which is presented by Smeros et al [9], the aforementioned indicators help nonexpert users evaluate more accurately the quality of news articles, compared to non-experts that do not have access to these indicators.…”