Augmented Echo State Networks with a feature layer and a nonlinear readout

Arnaud Rachez, Masafumi Hagiwara

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

Echo State Networks (ESNs) are an alternative to fully trained Recurrent Neural Networks (RNNs) showing State of the Art performance when applied to time series prediction. However, they have seldom been applied to abstract tasks and in the case of language modeling they require a number of units far superior to traditional RNNs in order to achieve similar performance. In this paper we propose a novel architecture by extending a conventional Echo State Network with a pre-recurrent feature layer and a nonlinear readout. The features are learned in a supervised way using a computationally cheap version of gradient descent and automatically capture grammatical similarity between words. They modifiy the dynamic of the network in a way that allows it to significantly outperform an ESN alone. The addition of a nonlinear readout is also investigated making the global system similar to a feed forward network with a memory layer.

Original languageEnglish
Title of host publication2012 International Joint Conference on Neural Networks, IJCNN 2012
DOIs
Publication statusPublished - 2012
Event2012 Annual International Joint Conference on Neural Networks, IJCNN 2012, Part of the 2012 IEEE World Congress on Computational Intelligence, WCCI 2012 - Brisbane, QLD, Australia
Duration: 2012 Jun 102012 Jun 15

Publication series

NameProceedings of the International Joint Conference on Neural Networks

Other

Other2012 Annual International Joint Conference on Neural Networks, IJCNN 2012, Part of the 2012 IEEE World Congress on Computational Intelligence, WCCI 2012
CountryAustralia
CityBrisbane, QLD
Period12/6/1012/6/15

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence

Fingerprint Dive into the research topics of 'Augmented Echo State Networks with a feature layer and a nonlinear readout'. Together they form a unique fingerprint.

Cite this