SciLens (Evaluating the Quality of Scientific News Articles Using Social Media and Scientific Literature Indicators) is a method to assess the quality of scientific news items that is described, developed and validated in a communication published in the Proceedings of the Web Conference 2019, held in San Francisco (USA) from 13 to 17 May.

A communication presented by Carlos Castillo, director of the Web Science and Social Computing (WSSC) research group at the Department of Information and Communication Technologies (DTIC) at UPF, together with professor Karl Aberer, both tutors of the doctoral thesis by Panayotis Smeros of École Polytechnique Fédérale de Lausanne (EPFL) (Switzerland), and first author of the work.

“The starting point of our work consists of structured methodologies that define a series of quality-related aspects to evaluate news items manually. Based on these aspects, we describe a series of quality indicators of news items. According to our experiments, these indicators help non-experts to assess more accurately the quality of a science news article, compared with non-experts who do not have access to these indicators”, explains Carlos Castillo.

A method for compiling contextual data that captures both the contents of the article, its relationship with the scientific literature, and the reactions it generates in social media

One of the main attributes of SciLens is its focus on the content and also on the context of scientific papers. The context is obtained, first, through explicit and implicit references of the article in the scientific literature, and second, through its impact in social media. “We showed that both contextual elements can be valuable sources of information to determine the quality of the article”, say the authors.

The work contributes a method for compiling contextual data that captures both the contents of the article, its relationship with the scientific literature, and the reactions it generates in social media

The validation of SciLens, performed via a combination of annotation by experts in communication and science, and by non-experts, demonstrates its effectiveness for both the semi-automatic and the automatic assessment of the quality of scientific news items, as shown graphically in the work method diagram.

This methodology enables identifying low quality journalistic articles that do not adequately represent or sometimes even contradict the scientific articles  they claim to cite

SciLens presents a series of quality indicators of scientific papers automatically from the citations generated by the article, the relation of a journalistic article with the scientific literature, where the authors introduce content similarity and graphic methods, and the reactions by social media to the article, where the authors propose a method to interpret the stance taken in the networks on the reception of the article (comments, support, reviews, etc.) as signals of quality.

Finally, the authors make an experimental assessment of their methods with experts and non-experts. SciLens can generate a quality score of an article in a totally automated manner, which is more in line with the assessments made by experts than by manual assessments carried out by non-experts. This enables identifying low quality journalistic articles that do not adequately represent or sometimes even contradict the scientific articles  they claim to cite.

Reference work:

Panayiotis Smeros, Carlos Castillo, Karl Aberer (2019), “SciLens: Evaluating the Quality of Scientific News Articles Using Social Media and Scientific Literature Indicators”, Proceedings WWW’2019, 13-17 May, San Francisco (USA), pp. 1747-1758. DOI: https://doi.org/10.1145/3308558.3313657