A line of research being pursued by Narcís Parés, head of the Full-Body Interaction Lab at the UPF Cognitive Media Technologies Group at the DTIC, seeks to demonstrate the potential of a mixed reality system to promote social initiation behaviours in children with autism spectrum disorder. The latest results of this research have just been presented at the 2020 International Conference on Multimodal Interaction along with Batuhan Sayis, a doctoral student at the DTIC-UPF, and Hatice Gunes, head of the Affective Intelligence & Robotics Lab at the University of Cambridge (UK).

“Our main aim in this study was to demonstrate that a mixed reality platform can be used to alter non-verbal body behaviour during social interaction between children with and without autism spectrum disorder (ASD) as well as in a traditional therapy setting using LEGO building tools”, Parés explains.  And he adds: “a second goal was to show how these body cues differ between ASD and non-ASD children during social initiation using these two platforms. 

They present the first analysis of the body cues generated under two conditions in a repeated-measures design

“We used machine learning techniques to analyse the visual data of eighteen trials of ASC and non-ASC dyads”, Parés points out. In their work, the authors present the first analysis of the body cues generated under two conditions in a repeated-measures design. The authors compared a mixed reality system with a typical social intervention strategy based on LEGO building tools, where both mediated a face-to-face dyadic play session between an ASC child and a non-ASC child. During the children’s interaction, the researchers obtained body cue indicators generated in both conditions (mixed reality and construction games) of both subjects individually (e.g., distances between joints and their movements) and interpersonally (e.g., proximity and visual focus of attention). 

The results of the study revealed that during social interaction there were differences between the body expressions of ASC and non-ASC children, both at individual and interpersonal level, for LEGO and for the MR system. They also showed that the characteristics that indicate differences between ASD and non-ASD in terms of non-verbal behaviour during initiation were higher for the mixed reality system compared to traditional therapy using LEGO.

Finally, the computational models evaluated by combining these characteristics that are indicative of differences between children with and without ASD allowed the researchers to recognize the kinds of social initiation (ASD or non-ASD) on the basis of bodily attitudes in LEGO and in mixed reality settings. These results demonstrate the potential benefits of full body interaction and MR settings for children with ASD. 

Related work:

Batuhan Sayis , Narcís Parés, Hatice Gunes (2020), "Bodily Expression of Social Initiation Behaviors in ASC and non-ASC children: Mixed Reality vs. LEGO Game Play". Companion Publication of the 2020 International Conference on Multimodal Interaction https://doi.org/10.17863/CAM.58023.