Property of average precision and its generalization: An examination of evaluation indicator for information retrieval experiments

Research output: Contribution to journalReview articlepeer-review

23 Citations (Scopus)

Abstract

In information retrieval experiments, indicators for measuring effectiveness of the systems or methods are important. Average precision is often used as an indicator for evaluating ranked output of documents in standard retrieval experiments. This report examines some properties of this indicator. First, we clarify mathematically that relevant documents at a higher position in the ranked list contribute much more to increasing the score of average precision. Second, influence of detecting unknown relevant documents on the score is discussed. Third, we examine statistical variation of average precision scores caused by fluctuations of results from relevance judgments. Another issue of this report is to explore evaluation indicators using data of multi-grade relevance. After reviewing indicators proposed by other researchers, i.e., modified sliding ratio, normalized discounted cumulative gain (nDCG), Q-measure and so on, a new indicator, generalized average precision is developed. We compare these indicators empirically using a simple and artificial example.

Original languageEnglish
Pages (from-to)1-19
Number of pages19
JournalNII Technical Reports
Volume2005
Issue number14
Publication statusPublished - 2005 Sep 29
Externally publishedYes

ASJC Scopus subject areas

  • Information Systems
  • Computer Science Applications

Fingerprint Dive into the research topics of 'Property of average precision and its generalization: An examination of evaluation indicator for information retrieval experiments'. Together they form a unique fingerprint.

Cite this