Alleviating overfitting for polysemous words for word representation estimation using lexicons

Yuanzhi Ke, Masafumi Hagiwara

研究成果: Conference contribution

2 被引用数 (Scopus)

抄録

Though there are some works on improving distributed word representations using lexicons, the improper over-fitting of the words that have multiple meanings is a remaining issue deteriorating the learning when lexicons are used, which needs to be solved. An alternative method is to allocate a vector per sense instead of a vector per word. However, the word representations estimated in the former way are not as easy to use as the latter one. Our previous work uses a probabilistic method to alleviate the overfitting, but it is not robust with a small corpus. In this paper, we propose a new neural network to estimate distributed word representations using a lexicon and a corpus. We add a lexicon layer in the continuous bag-of-words model and a threshold node after the output of the lexicon layer. The threshold rejects the unreliable outputs of the lexicon layer that are less likely to be the same with their inputs. In this way, it alleviates the overfitting of the polysemous words. The proposed neural network can be trained using negative sampling, which maximizing the log probabilities of target words given the context words, by distinguishing the target words from random noises. We compare the proposed neural network with the continuous bag-of-words model, the other works improving it, and the previous works estimating distributed word representations using both a lexicon and a corpus. The experimental results show that the proposed neural network is more efficient and balanced for both semantic tasks and syntactic tasks than the previous works, and robust to the size of the corpus.

本文言語English
ホスト出版物のタイトル2017 International Joint Conference on Neural Networks, IJCNN 2017 - Proceedings
出版社Institute of Electrical and Electronics Engineers Inc.
ページ2164-2170
ページ数7
2017-May
ISBN(電子版)9781509061815
DOI
出版ステータスPublished - 2017 6 30
イベント2017 International Joint Conference on Neural Networks, IJCNN 2017 - Anchorage, United States
継続期間: 2017 5 142017 5 19

Other

Other2017 International Joint Conference on Neural Networks, IJCNN 2017
国/地域United States
CityAnchorage
Period17/5/1417/5/19

ASJC Scopus subject areas

  • ソフトウェア
  • 人工知能

フィンガープリント

「Alleviating overfitting for polysemous words for word representation estimation using lexicons」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル