Multimodal literacy: Storytelling across senses

Suana Sanchez, Heng Gu, Kai Steven Kunze, Masahiko Inami

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This paper describes a novel framework for literature that presents the reader with a multimodal experience (visual, haptic and auditory). We present an initial prototype, ex- tending an iPad with a surface vibration transducer for hap- tic feedback to augment the reading of a short story. Text provides the reader with verbal information, whereas other senses perceive the context through non-verbal cues. We use our own framework to create this storytelling across senses, integrating text, sound, animation and tactile sen- sation. We believe with a multi-modal reading experience, users will enjoy reading more, will feel more immersed and remember the narrative better.

Original languageEnglish
Title of host publicationUbiComp and ISWC 2015 - Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the Proceedings of the 2015 ACM International Symposium on Wearable Computers
PublisherAssociation for Computing Machinery, Inc
Pages1257-1260
Number of pages4
ISBN (Print)9781450335751
DOIs
Publication statusPublished - 2015 Sep 7
EventACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2015 ACM International Symposium on Wearable Computers, UbiComp and ISWC 2015 - Osaka, Japan
Duration: 2015 Sep 72015 Sep 11

Other

OtherACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2015 ACM International Symposium on Wearable Computers, UbiComp and ISWC 2015
CountryJapan
CityOsaka
Period15/9/715/9/11

    Fingerprint

Keywords

  • Dual-coding
  • Multimodal
  • Reading
  • Valid co-occurrences

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Hardware and Architecture
  • Software

Cite this

Sanchez, S., Gu, H., Kunze, K. S., & Inami, M. (2015). Multimodal literacy: Storytelling across senses. In UbiComp and ISWC 2015 - Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the Proceedings of the 2015 ACM International Symposium on Wearable Computers (pp. 1257-1260). Association for Computing Machinery, Inc. https://doi.org/10.1145/2800835.2807940