An echo state network with working memories for probabilistic language modeling

Yukinori Homma, Masafumi Hagiwara

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper, we propose an ESN having multiple timescale layer and working memories as a probabilistic language model. The reservoir of the proposed model is composed of three neuron groups each with an associated time constant, which enables the model to learn the hierarchical structure of language. We add working memories to enhance the effect of multiple timescale layers. As shown by the experiments, the proposed model can be trained efficiently and accurately to predict the next word from given words. In addition, we found that use of working memories is especially effective in learning grammatical structure.

Original languageEnglish
Title of host publicationArtificial Neural Networks and Machine Learning, ICANN 2013 - 23rd International Conference on Artificial Neural Networks, Proceedings
Pages595-602
Number of pages8
DOIs
Publication statusPublished - 2013 Oct 8
Event23rd International Conference on Artificial Neural Networks, ICANN 2013 - Sofia, Bulgaria
Duration: 2013 Sep 102013 Sep 13

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume8131 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other23rd International Conference on Artificial Neural Networks, ICANN 2013
CountryBulgaria
CitySofia
Period13/9/1013/9/13

Keywords

  • ESNs
  • Probabilistic language model
  • working memory

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint Dive into the research topics of 'An echo state network with working memories for probabilistic language modeling'. Together they form a unique fingerprint.

  • Cite this

    Homma, Y., & Hagiwara, M. (2013). An echo state network with working memories for probabilistic language modeling. In Artificial Neural Networks and Machine Learning, ICANN 2013 - 23rd International Conference on Artificial Neural Networks, Proceedings (pp. 595-602). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8131 LNCS). https://doi.org/10.1007/978-3-642-40728-4_74