An echo state network with working memories for probabilistic language modeling

Yukinori Homma, Masafumi Hagiwara

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper, we propose an ESN having multiple timescale layer and working memories as a probabilistic language model. The reservoir of the proposed model is composed of three neuron groups each with an associated time constant, which enables the model to learn the hierarchical structure of language. We add working memories to enhance the effect of multiple timescale layers. As shown by the experiments, the proposed model can be trained efficiently and accurately to predict the next word from given words. In addition, we found that use of working memories is especially effective in learning grammatical structure.

Original languageEnglish
Title of host publicationLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Pages595-602
Number of pages8
Volume8131 LNCS
DOIs
Publication statusPublished - 2013
Event23rd International Conference on Artificial Neural Networks, ICANN 2013 - Sofia, Bulgaria
Duration: 2013 Sep 102013 Sep 13

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume8131 LNCS
ISSN (Print)03029743
ISSN (Electronic)16113349

Other

Other23rd International Conference on Artificial Neural Networks, ICANN 2013
CountryBulgaria
CitySofia
Period13/9/1013/9/13

Fingerprint

Echo State Network
Probabilistic Modeling
Working Memory
Language Modeling
Multiple Time Scales
Data storage equipment
Language Model
Hierarchical Structure
Time Constant
Probabilistic Model
Neuron
Model
Neurons
Predict
Experiment
Experiments

Keywords

  • ESNs
  • Probabilistic language model
  • working memory

ASJC Scopus subject areas

  • Computer Science(all)
  • Theoretical Computer Science

Cite this

Homma, Y., & Hagiwara, M. (2013). An echo state network with working memories for probabilistic language modeling. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 8131 LNCS, pp. 595-602). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8131 LNCS). https://doi.org/10.1007/978-3-642-40728-4_74

An echo state network with working memories for probabilistic language modeling. / Homma, Yukinori; Hagiwara, Masafumi.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 8131 LNCS 2013. p. 595-602 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8131 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Homma, Y & Hagiwara, M 2013, An echo state network with working memories for probabilistic language modeling. in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 8131 LNCS, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 8131 LNCS, pp. 595-602, 23rd International Conference on Artificial Neural Networks, ICANN 2013, Sofia, Bulgaria, 13/9/10. https://doi.org/10.1007/978-3-642-40728-4_74
Homma Y, Hagiwara M. An echo state network with working memories for probabilistic language modeling. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 8131 LNCS. 2013. p. 595-602. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-642-40728-4_74
Homma, Yukinori ; Hagiwara, Masafumi. / An echo state network with working memories for probabilistic language modeling. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 8131 LNCS 2013. pp. 595-602 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{da209b23de8c4947b65a5ce855cf9bee,
title = "An echo state network with working memories for probabilistic language modeling",
abstract = "In this paper, we propose an ESN having multiple timescale layer and working memories as a probabilistic language model. The reservoir of the proposed model is composed of three neuron groups each with an associated time constant, which enables the model to learn the hierarchical structure of language. We add working memories to enhance the effect of multiple timescale layers. As shown by the experiments, the proposed model can be trained efficiently and accurately to predict the next word from given words. In addition, we found that use of working memories is especially effective in learning grammatical structure.",
keywords = "ESNs, Probabilistic language model, working memory",
author = "Yukinori Homma and Masafumi Hagiwara",
year = "2013",
doi = "10.1007/978-3-642-40728-4_74",
language = "English",
isbn = "9783642407277",
volume = "8131 LNCS",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
pages = "595--602",
booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

}

TY - GEN

T1 - An echo state network with working memories for probabilistic language modeling

AU - Homma, Yukinori

AU - Hagiwara, Masafumi

PY - 2013

Y1 - 2013

N2 - In this paper, we propose an ESN having multiple timescale layer and working memories as a probabilistic language model. The reservoir of the proposed model is composed of three neuron groups each with an associated time constant, which enables the model to learn the hierarchical structure of language. We add working memories to enhance the effect of multiple timescale layers. As shown by the experiments, the proposed model can be trained efficiently and accurately to predict the next word from given words. In addition, we found that use of working memories is especially effective in learning grammatical structure.

AB - In this paper, we propose an ESN having multiple timescale layer and working memories as a probabilistic language model. The reservoir of the proposed model is composed of three neuron groups each with an associated time constant, which enables the model to learn the hierarchical structure of language. We add working memories to enhance the effect of multiple timescale layers. As shown by the experiments, the proposed model can be trained efficiently and accurately to predict the next word from given words. In addition, we found that use of working memories is especially effective in learning grammatical structure.

KW - ESNs

KW - Probabilistic language model

KW - working memory

UR - http://www.scopus.com/inward/record.url?scp=84884938773&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84884938773&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-40728-4_74

DO - 10.1007/978-3-642-40728-4_74

M3 - Conference contribution

SN - 9783642407277

VL - 8131 LNCS

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 595

EP - 602

BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

ER -