### Abstract

The back-propagation algorithm has been applied to many fields, and has shown large capability of neural networks. Many people use the back-propagation algorithm together with a momentum term to accelerate its convergence. However, in spite of the importance for theoretical studies, theoretical background of a momentum term has been unknown so far. First, this paper explains clearly the theoretical origin of a momentum term in the back-propagation algorithm for both a batch mode learning and a pattern-by-pattern learning. We will prove that the back-propagation algorithm having a momentum term can be derived through the following two assumptions: 1) The cost function is E^{n} = n/Σ/μ α^{n-μ} E_{μ}, where E_{μ} is the summation of squared error at the output layer at the μth learning time and a is the momentum coefficient. 2) The latest weights are assumed in calculating the cost function E^{n}. Next, we derive a simple relationship between momentum, learning rate, and learning speed and then further discussion is made with computer simulation.

Original language | English |
---|---|

Pages (from-to) | 1080-1086 |

Number of pages | 7 |

Journal | IEICE Transactions on Information and Systems |

Volume | E78-D |

Issue number | 8 |

Publication status | Published - 1995 Aug |

### Fingerprint

### ASJC Scopus subject areas

- Computer Graphics and Computer-Aided Design
- Information Systems
- Software

### Cite this

*IEICE Transactions on Information and Systems*,

*E78-D*(8), 1080-1086.

**Analysis of momentum term in back-propagation.** / Hagiwara, Masafumi; Sato, Akira.

Research output: Contribution to journal › Article

*IEICE Transactions on Information and Systems*, vol. E78-D, no. 8, pp. 1080-1086.

}

TY - JOUR

T1 - Analysis of momentum term in back-propagation

AU - Hagiwara, Masafumi

AU - Sato, Akira

PY - 1995/8

Y1 - 1995/8

N2 - The back-propagation algorithm has been applied to many fields, and has shown large capability of neural networks. Many people use the back-propagation algorithm together with a momentum term to accelerate its convergence. However, in spite of the importance for theoretical studies, theoretical background of a momentum term has been unknown so far. First, this paper explains clearly the theoretical origin of a momentum term in the back-propagation algorithm for both a batch mode learning and a pattern-by-pattern learning. We will prove that the back-propagation algorithm having a momentum term can be derived through the following two assumptions: 1) The cost function is En = n/Σ/μ αn-μ Eμ, where Eμ is the summation of squared error at the output layer at the μth learning time and a is the momentum coefficient. 2) The latest weights are assumed in calculating the cost function En. Next, we derive a simple relationship between momentum, learning rate, and learning speed and then further discussion is made with computer simulation.

AB - The back-propagation algorithm has been applied to many fields, and has shown large capability of neural networks. Many people use the back-propagation algorithm together with a momentum term to accelerate its convergence. However, in spite of the importance for theoretical studies, theoretical background of a momentum term has been unknown so far. First, this paper explains clearly the theoretical origin of a momentum term in the back-propagation algorithm for both a batch mode learning and a pattern-by-pattern learning. We will prove that the back-propagation algorithm having a momentum term can be derived through the following two assumptions: 1) The cost function is En = n/Σ/μ αn-μ Eμ, where Eμ is the summation of squared error at the output layer at the μth learning time and a is the momentum coefficient. 2) The latest weights are assumed in calculating the cost function En. Next, we derive a simple relationship between momentum, learning rate, and learning speed and then further discussion is made with computer simulation.

UR - http://www.scopus.com/inward/record.url?scp=0029354921&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0029354921&partnerID=8YFLogxK

M3 - Article

AN - SCOPUS:0029354921

VL - E78-D

SP - 1080

EP - 1086

JO - IEICE Transactions on Information and Systems

JF - IEICE Transactions on Information and Systems

SN - 0916-8532

IS - 8

ER -