Visual Signals in Language Technologies - Progress

PIs: Josep Blat, Coloma Ballester, Gloria Haro

External Collaborators: Josep Quer, Gemma Barberà, Delfina Aliaga (LSC-Lab, Dept. of Language Sciences) and Pilar Prieto (Dept. of Language Sciences)

MdM-supported staff: Marcel Granero Moya, Laia Tarrés

Other collaborators: Eva Valls, Victor Ubieto, Jaume Pozo (UPF), Floris Roelofsen, Marloes Oomen, Lyke Esselink, Gomèr Otterspeer, Jari Andersen (Univ. Amsterdam)

Goals of the project

The project focuses on developing technologies based on Artificial Intelligence (AI) for the automatic translation and production of sign language (SL), aiming to enhance access to digital content for deaf individuals and to improve communication within and with the deaf community.

The first goal has been to create a dataset in Catalan Sign Language (LSC) that includes isolated signs, hand configurations and continuous signing samples. This data will be used for linguistic research and the development of AI-based technologies applied to the LSC.

First Achievements

We have recorded over 1,600 isolated signs and the most common hand configurations in LSC. We have also recorded example sentences and other natural signing situations. This material has been recorded using two techniques: i) using several cameras positioned from different angles and ii) using motion capture technology to obtain high-precision 3D data, which will be useful for training artificial intelligence models.

Activities

• Recordings at the UPF with multiple cameras.

• LSC 3D motion capture recordings at the University of Amsterdam's SignLab.

Next steps

• Process and organize data for analysis and use in AI.

• Start the first experiments with machine learning models.

• Compare the two recording techniques to obtain accurate 3D representations of the signature in LSC.