Building a Video-and-Language Dataset with Human Actions for Multimodal Logical Inference

Riko Suzuki; Hitomi Yanaka; Koji Mineshima; Daisuke Bekki

Building a Video-and-Language Dataset with Human Actions for Multimodal Logical Inference

Riko Suzuki, Hitomi Yanaka, Koji Mineshima, Daisuke Bekki

Department of Philosophy

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

This paper introduces a new video-and-language dataset with human actions for multimodal logical inference, which focuses on intentional and aspectual expressions that describe dynamic human actions. The dataset consists of 200 videos, 5,554 action labels, and 1,942 action triplets of the form (subject, predicate, object) that can be translated into logical semantic representations. The dataset is expected to be useful for evaluating multimodal inference systems between videos and semantically complicated sentences including negation and quantification.

Original language	English
Title of host publication	MMSR 2021 - Multimodal Semantic Representations, Proceedings of the 1st Workshop
Editors	Lucia Donatelli, Nikhil Krishnaswamy, Kenneth Lai, James Pustejovsky
Publisher	Association for Computational Linguistics (ACL)
Pages	102-107
Number of pages	6
ISBN (Electronic)	9781954085213
Publication status	Published - 2021
Event	1st Workshop on Multimodal Semantic Representations, MMSR 2021 - Virtual, Groningen, Netherlands Duration: 2021 Jun 16 → …

Publication series

Name	MMSR 2021 - Multimodal Semantic Representations, Proceedings of the 1st Workshop

Conference

Conference	1st Workshop on Multimodal Semantic Representations, MMSR 2021
Country/Territory	Netherlands
City	Virtual, Groningen
Period	21/6/16 → …

ASJC Scopus subject areas

Computer Networks and Communications
Software

Cite this

Suzuki, R., Yanaka, H., Mineshima, K., & Bekki, D. (2021). Building a Video-and-Language Dataset with Human Actions for Multimodal Logical Inference. In L. Donatelli, N. Krishnaswamy, K. Lai, & J. Pustejovsky (Eds.), MMSR 2021 - Multimodal Semantic Representations, Proceedings of the 1st Workshop (pp. 102-107). (MMSR 2021 - Multimodal Semantic Representations, Proceedings of the 1st Workshop). Association for Computational Linguistics (ACL).

Building a Video-and-Language Dataset with Human Actions for Multimodal Logical Inference. / Suzuki, Riko; Yanaka, Hitomi; Mineshima, Koji et al.

MMSR 2021 - Multimodal Semantic Representations, Proceedings of the 1st Workshop. ed. / Lucia Donatelli; Nikhil Krishnaswamy; Kenneth Lai; James Pustejovsky. Association for Computational Linguistics (ACL), 2021. p. 102-107 (MMSR 2021 - Multimodal Semantic Representations, Proceedings of the 1st Workshop).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Suzuki, R, Yanaka, H, Mineshima, K & Bekki, D 2021, Building a Video-and-Language Dataset with Human Actions for Multimodal Logical Inference. in L Donatelli, N Krishnaswamy, K Lai & J Pustejovsky (eds), MMSR 2021 - Multimodal Semantic Representations, Proceedings of the 1st Workshop. MMSR 2021 - Multimodal Semantic Representations, Proceedings of the 1st Workshop, Association for Computational Linguistics (ACL), pp. 102-107, 1st Workshop on Multimodal Semantic Representations, MMSR 2021, Virtual, Groningen, Netherlands, 21/6/16.

Suzuki R, Yanaka H, Mineshima K, Bekki D. Building a Video-and-Language Dataset with Human Actions for Multimodal Logical Inference. In Donatelli L, Krishnaswamy N, Lai K, Pustejovsky J, editors, MMSR 2021 - Multimodal Semantic Representations, Proceedings of the 1st Workshop. Association for Computational Linguistics (ACL). 2021. p. 102-107. (MMSR 2021 - Multimodal Semantic Representations, Proceedings of the 1st Workshop).

Suzuki, Riko ; Yanaka, Hitomi ; Mineshima, Koji et al. / Building a Video-and-Language Dataset with Human Actions for Multimodal Logical Inference. MMSR 2021 - Multimodal Semantic Representations, Proceedings of the 1st Workshop. editor / Lucia Donatelli ; Nikhil Krishnaswamy ; Kenneth Lai ; James Pustejovsky. Association for Computational Linguistics (ACL), 2021. pp. 102-107 (MMSR 2021 - Multimodal Semantic Representations, Proceedings of the 1st Workshop).

@inproceedings{72e9a05dd90649c4afb00641d7a4787f,

title = "Building a Video-and-Language Dataset with Human Actions for Multimodal Logical Inference",

abstract = "This paper introduces a new video-and-language dataset with human actions for multimodal logical inference, which focuses on intentional and aspectual expressions that describe dynamic human actions. The dataset consists of 200 videos, 5,554 action labels, and 1,942 action triplets of the form (subject, predicate, object) that can be translated into logical semantic representations. The dataset is expected to be useful for evaluating multimodal inference systems between videos and semantically complicated sentences including negation and quantification.",

author = "Riko Suzuki and Hitomi Yanaka and Koji Mineshima and Daisuke Bekki",

note = "Funding Information: This work was partially supported by JST CREST Grant Number JPMJCR20D2, Japan. Thanks to the anonymous reviewers for helpful comments. We would also like to thank Mai Yokozeki and Natsuki Murakami for their contributions. Publisher Copyright: {\textcopyright} 2021 Association for Computational Linguistics; 1st Workshop on Multimodal Semantic Representations, MMSR 2021 ; Conference date: 16-06-2021",

year = "2021",

language = "English",

series = "MMSR 2021 - Multimodal Semantic Representations, Proceedings of the 1st Workshop",

publisher = "Association for Computational Linguistics (ACL)",

pages = "102--107",

editor = "Lucia Donatelli and Nikhil Krishnaswamy and Kenneth Lai and James Pustejovsky",

booktitle = "MMSR 2021 - Multimodal Semantic Representations, Proceedings of the 1st Workshop",

address = "United States",

}

TY - GEN

T1 - Building a Video-and-Language Dataset with Human Actions for Multimodal Logical Inference

AU - Suzuki, Riko

AU - Yanaka, Hitomi

AU - Mineshima, Koji

AU - Bekki, Daisuke

N1 - Funding Information: This work was partially supported by JST CREST Grant Number JPMJCR20D2, Japan. Thanks to the anonymous reviewers for helpful comments. We would also like to thank Mai Yokozeki and Natsuki Murakami for their contributions. Publisher Copyright: © 2021 Association for Computational Linguistics

PY - 2021

Y1 - 2021

N2 - This paper introduces a new video-and-language dataset with human actions for multimodal logical inference, which focuses on intentional and aspectual expressions that describe dynamic human actions. The dataset consists of 200 videos, 5,554 action labels, and 1,942 action triplets of the form (subject, predicate, object) that can be translated into logical semantic representations. The dataset is expected to be useful for evaluating multimodal inference systems between videos and semantically complicated sentences including negation and quantification.

AB - This paper introduces a new video-and-language dataset with human actions for multimodal logical inference, which focuses on intentional and aspectual expressions that describe dynamic human actions. The dataset consists of 200 videos, 5,554 action labels, and 1,942 action triplets of the form (subject, predicate, object) that can be translated into logical semantic representations. The dataset is expected to be useful for evaluating multimodal inference systems between videos and semantically complicated sentences including negation and quantification.

UR - http://www.scopus.com/inward/record.url?scp=85138285564&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85138285564&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85138285564

T3 - MMSR 2021 - Multimodal Semantic Representations, Proceedings of the 1st Workshop

SP - 102

EP - 107

BT - MMSR 2021 - Multimodal Semantic Representations, Proceedings of the 1st Workshop

A2 - Donatelli, Lucia

A2 - Krishnaswamy, Nikhil

A2 - Lai, Kenneth

A2 - Pustejovsky, James

PB - Association for Computational Linguistics (ACL)

T2 - 1st Workshop on Multimodal Semantic Representations, MMSR 2021

Y2 - 16 June 2021

ER -

Building a Video-and-Language Dataset with Human Actions for Multimodal Logical Inference

Abstract

Publication series

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this