Articles and book chapters Articles and book chapters

Return to Full Page

How to represent a word and predict it, too: improving tied architectures for language modelling

  • Authors
  • Gulordava K, Aina L, Boleda G
  • UPF authors
  • AINA ., LAURA; GULORDAVA ., KRISTINA; BOLEDA TORRENT, GEMMA;
  • Authors of the book
  • -
  • Book title
  • Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
  • Publisher
  • Association for Computational Linguistics
  • Publication year
  • 2018
  • Pages
  • 2936-2941
  • ISBN
  • 9781948087841
  • Abstract
  • Recent state-of-the-art neural language models share the representations of words given by the input and output mappings. We propose a simple modification to these architectures that decouples the hidden state from the word embedding prediction. Our architecture leads to comparable or better results compared to previous tied models and models without tying, with a much smaller number of parameters. We also extend our proposal to word2vec models, showing that tying is appropriate for general word prediction tasks.
  • Complete citation
  • Gulordava K, Aina L, Boleda G. How to represent a word and predict it, too: improving tied architectures for language modelling. In: -. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 1 ed. East Stroudsburg PA: Association for Computational Linguistics; 2018. p. 2936-2941.