Articles and book chapters Articles and book chapters

Return to Full Page

Grounding semantic roles in images

  • Authors
  • Silberer, Carina; Pinkal, Manfred
  • UPF authors
  • SILBERER ., CARINA HELGA;
  • Authors of the book
  • -
  • Book title
  • Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
  • Publisher
  • Association for Computational Linguistics
  • Publication year
  • 2018
  • Pages
  • 2616-2626
  • ISBN
  • 9781948087841
  • Abstract
  • We address the task of visual semantic role labeling (vSRL), the identification of the participants of a situation or event in a visual scene, and their labeling with their semantic relations to the event or situation. We render candidate participants as image regions of objects, and train a model which learns to ground roles in the regions which depict the corresponding participant. Experimental results demonstrate that we can train a vSRL model without reliance on prohibitive image-based role annotations, by utilizing noisy data which we extract automatically from image captions using a linguistic SRL system. Furthermore, our model induces frame¿semantic visual representations, and their comparison to previous work on supervised visual verb sense disambiguation yields overall better results.
  • Complete citation
  • Silberer, Carina; Pinkal, Manfred. Grounding semantic roles in images. In: -. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 1 ed. East Stroudsburg PA: Association for Computational Linguistics; 2018. p. 2616-2626.