Blogs

The reproducibility of research data, a challenge for science

(Original content: UPF news)

Low-cost devices allow obtaining the results of tests on patients and research subjects quickly. For example, in the study of cognitive processes underlying learning, these devices can provide multimodal data that complement behavioural data. However, the accurate collection and combination of multimodal and behavioural data in a manner that enables reproducing the experimenty is challenging and often requires researchers to refine their approaches.

Davinia Hernández Leo, coordinator of the Education Technologies research group (TIDE) together with Marc Yoshimi Beardsley, a member of her team, with the collaboration of Rafael Ramírez Meléndez, coordinator of the Music and Machine Learning Lab of the Music Technology Group (MTG) at UPF’s Department of Information and Communication Technologies (DTIC), have published a paper in Journal of Computer Assisted Learning, which deals with this problem, one of the strategic points of the DTIC’s Maria de Maeztu Unit of Excellence programme.

As Hernández-Leo says, “our study has attempted to replicate a standard cognitive experiment of memorizing a list of words, in an environment other than the laboratory and with a more accessible approach using an affordable device and with open source software”. The purpose of this study was to validate a procedure by means of a low-cost, open source installation able to replicate a multimodal experiment, in this case the experiment of Pastötter et al. (2011); apply the appropriate methods for data acquisition, and facilitate reproducibility through transparent documentation.

So, the researchers carried out recall tests of the contents learned and physiological tests of brain activity patterns, based on alpha wave oscillations measured through electroencephalogram (EEG) recordings. The brain alpha waves are related to attention and the working memory.

The experiment consisted of the direct replication of the multimodal wordlist of Pastötter et al. (2011) using the low-cost Emotiv EPOC® device, a non-invasive brain-computer interface that acquires electrophysiological measures of brain activity during learning, in this case a list of words (20 words in Spanish), in order to study if during learning retrieval or recovery facilitates the encoding of the subsequent learning of new words.

Fifty-seven Spanish-speaking participants took part in this experiment. The results were evaluated through performance in the recall tests and also through changes in alpha-wave oscillations. In this study it was possible to reproduce the behaviour of the participants but not of the physiological results obtained through the device.

It is concluded that, with regard to the use of multimodal data about learning, especially in interdisciplinary investigations, researchers seeking to incorporate multimodal data in innovative ways in their studies should strive to do so in a sustainable manner in accordance with the requirements of reproducibility. The scientific community is aware of the problem of the reproducibility of the scientific proof reported in the literature as some papers show that the reliability of some studies may be compromised due to bias, insufficient sample size, etc. In response to this problem, recently, many journals and organizations have adhered to the TOP guidelines (Transparency and Openness Promotion), promoted by the Center of Open Science since April 2017. 

Reference work:

Marc Yoshimi Beardsley, Davinia Hernández-Leo, Rafael Ramírez-Meléndez (2018), “Seeking reproducibility: Assessing a multimodal study of the testing effect“, Journal of Computer Assisted Learning, 8 May, DOI: 10.1111 / jcal.12265.