A European research consortium, of which UPF is a member, has begun work on the VISTA-SL research project, the aim of which to develop a new online model for teaching sign languages (SL) based on advanced technologies. The researchers from UPF involved in the project are responsible for designing 3D avatars, which will serve as the students’ virtual teachers. 

In addition to the avatars, the platform will also integrate automatic learning tools and AI, to help adapt the system to the specific needs of each student, and gamification, to make learning SL more entertaining through play. The platform is expected to include chatbots, which will diagnose the learning needs of each student and tailor the educational model, as well as monitor their progress.

The VISTA-SL (Visual Interactive System for Teaching and Assessment of Sign Languages) project will be conducted over the next three years by a European consortium headed by the University of Patras (Greece). In addition to UPF, the project features four other partners: Hellenic Mediterranean University (Greece), the Royal Dutch Kentalis International Foundation (Netherlands), the University of Hamburg (Germany) and Technological University Dublin (Ireland). The project is financed by the EU’s Erasmus+ programme, with a total of €400,000. 

Active in this initiative on behalf of Pompeu Fabra University is the Interactive Technologies Research Group (GTI) from the Department of Engineering. As experts in computational design and animation, its members will be responsible for developing the new e-learning platform’s 3D avatars. Nonetheless, the work team is interdisciplinary and will include experts in linguistics and education from the other universities. The new e-learning system will initially serve to learn Greek, German, Dutch and Irish sign languages. However, it has the potential to integrate other SLs in the future. In fact, the GTI hopes to add Catalan Sign Language (CSL) to the e-learning platform, based on material from the CSL MOOC developed by the Department of Translation and Language Sciences at UPF.

The avatars will act as virtual sign language teachers

As members of the VISTA-SL consortium, the researchers from the GTI at UPF will design virtual avatars with several educational functions. These include teaching users how to correctly express messages using sign language, thus serving as a reference model, and providing the students visual feedback on their exercises. The e-learning platform may also offer students the option to upload their own exercises, showing themselves expressing messages in sign language, and have them be corrected by these online teachers or other automatic tools.

At UPF, the driving force behind the project is Josep Blat, while the principal investigator is currently Ricardo Marques, both of whom are linked to the GTI. According to Marques, “We will use 3D avatars to show the correct sign language movements, mainly manual, and then expand the use for a better representation of the facial expressions, which are much more complex. We will implement a function through which students can see their attempts alongside the avatar’s demonstration, providing a direct comparison and enhancing both their awareness and performance. We will harness the avatars’ custom capabilities, such as expansion, background adaptation....”

Previous sign language research at UPF 

To develop the VISTA-SL project, the GTI from the Department of Engineering at UPF has drawn on previous experience in another project linked to sign language. This research project, SignON (2021-2024), involved developing an open, mobile-based solution to automatically translate between sign and oral languages (written and spoken language), through funding from the EU’s Horizon 2020 programme.

VISTA-SL and SignON are just two of the research projects devoted to sign languages that UPF has developed, and not just in the field of engineering, but also in linguistics. In this regard, it should be noted that the Department and Faculty of Translation and Language Sciences at UPF have been leaders in the teaching, research and dissemination of Catalan Sign Language (CSL) for over two decades.

In fact, the departments of Engineering and Translation and Language Sciences have teamed up to pursue several research projects on this subject. Such is the case of the projects “Natural Language Processing for Sign Languages”, the purpose of which is to design systems that automatically process sign languages, thus far much less developed than those relating to oral languages; and “Visual Signals for Language Technologies”, which also involves the SignLab at the University of Amsterdam and focuses more on the generation of visual signals.