Language modeling using augmented echo state networks

Arnaud Rachez, Masafumi Hagiwara

研究成果: Article査読

抄録

Interest in natural language modeling using neural networks has been growing in the past decade. The objective of this paper is to investigate the predictive capabilities of echo state networks (ESNs) at the task of modeling English sentences. Based on the finding that ESNs exhibit a Markovian organization of their state space that makes them, close to the widely used n-gram models, we describe two modifications of the conventional architecture that allow significant improvement by leveraging the kind of representation developed in the reservoir. Firstly, the addition of pre-recurrent features is shown to capture syntactic similarities between words and can be trained efficiently by using the contracting property of the reservoir to truncate the gradient descent. Secondly, the addition of multiple linear readouts using the mixture of experts framework is also shown to greatly improve accuracy while being trainable in parallel using Expectation- Maximization. Furthermore it can easily be transformed into a supervised mixture of expert model with several variations allowing reducing the training time and can take into account handmade features.

本文言語English
ページ(範囲)1969-1981
ページ数13
ジャーナルInternational Journal of Innovative Computing, Information and Control
10
6
出版ステータスPublished - 2014 12 1

ASJC Scopus subject areas

  • ソフトウェア
  • 理論的コンピュータサイエンス
  • 情報システム
  • 計算理論と計算数学

フィンガープリント

「Language modeling using augmented echo state networks」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル