Títol: "Multimodal Interaction in First Acquaintance Conversations: Effects of Information Structure and Multimodal Prosody", a càrrec de Patrizia Paggio (University of Copenhagen and University of Malta).

Dia: dilluns 17 de d’abril de 2023
Hora: 12.30 h
Lloc: sala de reunions 52.939 - 9a planta edifici 52. Roc Boronat - Campus del Poblenou - UPF

Resum:
In this presentation I will introduce the First Acquaintance corpora for Danish and for Maltese conversational data. The two corpora have been created under my coordination by two groups of researchers at the Centre for Language Technology of the University of Copenhagen and at the Institute of Linguistics and Language Technology of the University of Malta. The development of the two corpora follows the same design principles and have the same goal, which is providing resources for the study of speech and gesture (not only of the hands) in face-to-face conversations. Each consists of twelve videorecorded, transcribed and annotated first encounter dyadic conversations. After describing the corpora in more detail, and explaining the principles that have guided their development, I will discuss three studies based on data from one of the other of the two corpora. Common to these studies is that they have looked at different aspects of the interaction between gestural behaviour and information structure on the one hand, and gestural behaviour and prosody on the other. The first study deals with the relation between head movements and focus by analysing the temporal coordination of head movements and focused words in the Danish multimodal corpus. The results point at an effect of information structure on the occurrence of head movements by showing that head movements tend to co-occur with focused words about twice as often as it would be expected had the distribution been random. The second study uses the Maltese data to study complement fronting, a linguistic mechanism whereby a verbal complement is placed initially in a sentence, as in second year għad-ni (in second year, I am), and discusses the ways in which suprasegmental features, both in terms of prosody and gestures, underpin the different discourse functions of the resulting constructions in Maltese. Finally, in the third study, the Maltese corpus was used to extract stimuli for an analysis of the way hand gestures interact with prosody in the perception of prominence by Maltese listeners. The study found that although seeing the speaker produce the gesture increases the perceived prominence of the co-occurring word, it does not so in a statistically significant way. The conclusion is that gestures may provide an additional but not necessary cue to prominence for the listener.

Referències:
Paggio, P., & Navarretta, C. (2017). The Danish NOMCO corpus: multimodal interaction in first acquaintance conversations. Language Resources and Evaluation, 51, 463-494.
Paggio, P., Galea, L. and S. Vella (2018) Prosodic and gestural marking of complement fronting in Maltese. In Paggio and Gatt (2018) Languages of Malta. Studies in Diversity Linguistics 18, Language Science Press, pp. 81-116.
Paggio, P., Vella, A., Mitterer, H, and G. Attard (under review). Do Hand Gestures Increase Prominence in Naturally Produced Utterances? Submitted to Special Issue of the Journal Language and Cognition on Multimodal Prosody.