Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Publisher
Association for Computational Linguistics
Publication year
2018
Pages
2616-2626
ISBN
9781948087841
Abstract
We address the task of visual semantic role labeling (vSRL), the identification of the participants of a situation or event in a visual scene, and their labeling with their semantic relations to the event or situation. We render candidate participants as image regions of objects, and train a model which learns to ground roles in the regions which depict the corresponding participant. Experimental results demonstrate that we can train a vSRL model without reliance on prohibitive image-based role annotations, by utilizing noisy data which we extract automatically from image captions using a linguistic SRL system. Furthermore, our model induces frame¿semantic visual representations, and their comparison to previous work on supervised visual verb sense disambiguation yields overall better results.
Complete citation
Silberer, Carina; Pinkal, Manfred. Grounding semantic roles in images. In: -. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 1 ed. East Stroudsburg PA: Association for Computational Linguistics; 2018. p. 2616-2626.
Articles and book chapters