Empirical comparison of word similarity measures based on co-occurrence, context, and a vector space model

Natsuki Kadowaki, Kazuaki Kishida

研究成果: Article査読

抄録

Word similarity is often measured to enhance system performance in the information retrieval field and other related areas. This paper reports on an experimental comparison of values for word similarity measures that were computed based on 50 intentionally selected words from a Reuters corpus. There were three targets, including (1) co-occurrence-based similarity measures (for which a co-occurrence frequency is counted as the number of documents or sentences), (2) context-based distributional similarity measures obtained from a latent Dirichlet allocation (LDA), nonnegative matrix factorization (NMF), and Word2Vec algorithm, and (3) similarity measures computed from the tf-idf weights of each word according to a vector space model (VSM). Here, a Pearson correlation coefficient for a pair of VSM-based similarity measures and co-occurrence-based similarity measures according to the number of documents was highest. Group-average agglomerative hierarchical clustering was also applied to similarity matrices computed by individual measures. An evaluation of the cluster sets according to an answer set revealed that VSM-and LDA-based similarity measures performed best.

本文言語English
ページ(範囲)6-17
ページ数12
ジャーナルJournal of Information Science Theory and Practice
8
2
DOI
出版ステータスPublished - 2020 6 1

ASJC Scopus subject areas

  • 情報システム
  • 情報システムおよび情報管理
  • 図書館情報学

フィンガープリント

「Empirical comparison of word similarity measures based on co-occurrence, context, and a vector space model」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル