Affective Image Captioning for Visual Artworks Using Emotion-Based Cross-Attention Mechanisms

Shintaro Ishikawa, Komei Sugiura

Research output: Contribution to journalArticlepeer-review


Within the museum community, the automatic generation of artwork description is expected to accelerate the improvement of accessibility for visually impaired visitors. Captions that describe artworks should be based on emotions because art is inseparable from viewers' emotional reactions. By contrast, artworks typically do not have unique interpretations; thus, it is difficult for systems to reflect the specified emotions in captions precisely. Most existing methods attempt to leverage predicted emotion labels from images to generate emotion-oriented captions; however, they do not allow users to specify arbitrary emotions. In this paper, we aim to build a model that generates emotion-conditioned captions that describe visual art. We propose an affective visual encoder, which integrates emotion attributes and cross-modal joint features of images into visual information over all encoder blocks. Moreover, we introduce affective tokens that fuse grid- and region-based image features to cover both contextual and object-level information. We validated our method on the ArtEmis dataset, and the results demonstrated that our method outperformed baseline methods on all metrics in the emotion-conditioned task.

Original languageEnglish
Pages (from-to)24527-24534
Number of pages8
JournalIEEE Access
Publication statusPublished - 2023


  • affective image captioning
  • Emotion
  • visual art

ASJC Scopus subject areas

  • Computer Science(all)
  • Materials Science(all)
  • Engineering(all)
  • Electrical and Electronic Engineering


Dive into the research topics of 'Affective Image Captioning for Visual Artworks Using Emotion-Based Cross-Attention Mechanisms'. Together they form a unique fingerprint.

Cite this