Sukno FM, Domínguez M, Ruiz A, Schiller D, Lingenfelser F, Pragst L, Kamateri E, Vrochidis S. A Multimodal Annotation Schema for Non-Verbal Affective Analysis in the Health-Care Domain. MARMI'16: 1st International Workshop on Multimedia Analysis and Retrieval for Multimodal Interaction Proceedings.

The development of conversational agents with human interaction capabilities requires advanced affective state recognition integrating non-verbal cues from the different modalities constituting what in human communication we perceive as an overall affective state. Each of the modalities is often handled by a different subsystem that conveys only a partial interpretation of the whole and, as such, is evaluated only in terms of its partial view. To tackle this shortcoming, we investigate the generation of a unified multimodal annotation schema of non-verbal cues from the perspective of an inter-disciplinary group of experts. We aim at obtaining a common ground-truth with a unique representation using the Valence and Arousal space and a discrete non-linear scale of values. The proposed annotation schema is demonstrated on a corpus in the health-care domain but is scalable to other purposes. Preliminary results on inter-rater variability show a positive correlation of consensus level with high (absolute) values of Valence and Arousal as well as with the number of annotators labeling a given video sequence.

Additional material:

Postprint in UPF e-repository