Retrieving and Highlighting Action with Spatiotemporal Reference

Seito Kasai, Yuchi Ishikawa, Masaki Hayashi, Yoshimitsu Aoki, Kensho Hara, Hirokatsu Kataoka

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper, we present a framework thatjointly retrieves and spatiotemporally highlights actions in videos by enhancing current deep cross-modal retrieval methods. Our work takes on the novel task of action highlighting, which visualizes where and when actions occur in an untrimmed video setting. Action highlighting is a fine-grained task, compared to conventional action recognition tasks which focus on classification or window-based localization. Leveraging weak supervision from annotated captions, our framework acquires spatiotemporal relevance maps and generates local embeddings which relate to the nouns and verbs in captions. Through experiments, we show that our model generates various maps conditioned on different actions, in which conventional visual reasoning methods only go as far as to show a single deterministic saliency map. Also, our model improves retrieval recall over our baseline without alignment by 2-3% on the MSR-VTT dataset.

Original languageEnglish
Title of host publication2020 IEEE International Conference on Image Processing, ICIP 2020 - Proceedings
PublisherIEEE Computer Society
Pages1401-1405
Number of pages5
ISBN (Electronic)9781728163956
DOIs
Publication statusPublished - 2020 Oct
Event2020 IEEE International Conference on Image Processing, ICIP 2020 - Virtual, Abu Dhabi, United Arab Emirates
Duration: 2020 Sep 252020 Sep 28

Publication series

NameProceedings - International Conference on Image Processing, ICIP
Volume2020-October
ISSN (Print)1522-4880

Conference

Conference2020 IEEE International Conference on Image Processing, ICIP 2020
CountryUnited Arab Emirates
CityVirtual, Abu Dhabi
Period20/9/2520/9/28

Keywords

  • 3d cnn
  • action recognition
  • information retrieval
  • interpretability
  • vision language

ASJC Scopus subject areas

  • Software
  • Computer Vision and Pattern Recognition
  • Signal Processing

Fingerprint Dive into the research topics of 'Retrieving and Highlighting Action with Spatiotemporal Reference'. Together they form a unique fingerprint.

Cite this