Paula Fortuna, supervised by Leo Wanner and Joan Soler Company, obtained the Open Science Award at the 7th PhD workshop organised at the Department. The jury was formed by winners of the award in previous editions: Mónica Domínguez, Xavier Favory, Alp Öktem.

Are Hate Speech Classification Results Reproducible? An Approach Using Deep Learning

Abstract

Our approach is based on replicating one of the most relevant works on the state-of-the-art literature for hate speech automatic detection, using word embeddings and LSTM. After circumventing some of the problems of the original code, an initial experiment proved to have worse results than the original study. Additionally, we applied the same classification algorithm to two datasets more. We found poor results when applying it to other hate speech data. However, for a dataset on offensive language, the classifier performed well, proving to have a better performance for offense detection than for hate speech.

Additional information provided by Paula as part of her candidature to the award

The work described in this paper corresponds to a simpler and reduced version of a paper soon to be published in the scope of the participation of the Stop PropagHate team in the SemEval 2019’s conference shared tasks. As the name of the work points out (“Are hate speech classification results reproducible?”), reproducibility is and was central in this work. We did it by trying to replicate and use as the base for our work a well recognized work in the state of the art. Morevoer, we conducted different steps promoting reproducibility. We assure the availability of:

• software and source code (https://github.com/paulafortuna/SemEval_2019_public ).
• the implementations of algorithms developed within the paper are provided (e.g. https://github.com/paulafortuna/SemEval_2019_public/blob/master/scripts_pipeline/classification_models/Classifier_LSTMFromEmbedding.py).
• we used public and available datasets.

Additionally in the readme of the project it is possible to find links for:

• scripts to run with the exact parameter settings used in the paper (e.g. https://github.com/paulafortuna/SemEval_2019_public/blob/master/main_replication.py)
• available documentation for installing and using software (https://github.com/paulafortuna/SemEval_2019_public/tree/master/configurations )
• documentation generated from the code (Doxygen https://paulafortuna.github.io/SemEval2019_docs/docs/hierarchy.html )
• comments within the code (e.g https://github.com/paulafortuna/SemEval_2019_public/blob/master/scripts_pipeline/Experiment.py )
• class diagrams (e.g https://docs.google.com/presentation/d/1t64DdCrN2avDvKocUBp-2kDHYl5dkT_2M8aRPLiw3u8/edit?usp=sharing )

Also this project was developed considering the openness of the license used for the software and data, by using MIT.

This research is being developed in the scope of the Stop PropagHate project (round 4 from Google DNI fundings). This project has the general goal of reducing online hate speech by acting and impacting in online journalism. For this we are developing hate speech detection automatic tools. The project has a website (http://stop-propaghate.inesctec.pt) that has different goals:

• provides a GUI for the developed algorithms and can be tested by the researchers, but also by curious people on the topic.
• promotes the visibility of the topic.
• presents the project team.

As you can see in the team members, this project counts with the participation of one journalist (Vanessa Cortez) bringing a multidisciplinary team for the project and making it more connected with the reality of the newspapers and journalists. We expect that this multidisciplinary collaboration can shorten the distance between academic research and news industry.

In the scope of the Stop PropagHate project we aim at building tools for online hate speech detection and news evaluation on the probability for contributing for hate speech dissemination. The final goal of this project is to have a real impact by helping to solve a problem for the news and media ecossystem. Due to the high volume of data in this fields and the increasing of hate speech online. We are giving steps to facilitate hate speech fighting by providing an API’s for our machine learning models.

I would like to point out previous work developed towards reproducibility and open data. As result of the work developed in my master thesis, there was a Portuguese Dataset for hate speech classification public available (https://rdm.inesctec.pt/pt_PT/dataset/cs-2017-008).