Digital Full-Face Mask Display with Expression Recognition using Embedded Photo Reflective Sensor Arrays

Yoshinari Takegawa, Yutaka Tokuda, Akino Umezawa, Katsuhiro Suzuki, Katsutoshi Masai, Yuta Sugiura, Maki Sugimoto, Diego Martinez Plasencia, Sriram Subramanian, Keiji Hirata

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This paper presents a thin digital full-face mask display that can reflect an entire facial expression of a user onto an avatar to support augmented face-to-face communication in real environments. Although camera-based facial expression recognition technology has enabled people to augment their faces with avatars, application was limited to face-to-face communication in virtual environments. To enable digital facial augmentation with an avatar in a real space, we propose a digital face mask display system that integrates a lightweight flexible display with a thin facial expression recognition system. The thin wearable facial expression recognition system was implemented with photo reflective sensor arrays which can measure facial expressions at 40 feature points distributed across an entire face. We investigated a ten-class facial expression identification model based on an SVM training algorithm. The trained model achieved an average accuracy of 79% when identifying the facial expressions of multiple users. User experiments indicated that the proposed thin digital full-face mask display allows the wearer to control the facial expression of the avatar with a fast response rate and create a positive sense of self-agency and self-ownership toward the augmented avatar face.

Original languageEnglish
Title of host publicationProceedings - 2020 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2020
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages101-108
Number of pages8
ISBN (Electronic)9781728185088
DOIs
Publication statusPublished - 2020 Nov
Externally publishedYes
Event19th IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2020 - Virtual, Recife/Porto de Galinhas, Brazil
Duration: 2020 Nov 92020 Nov 13

Publication series

NameProceedings - 2020 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2020

Conference

Conference19th IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2020
CountryBrazil
CityVirtual, Recife/Porto de Galinhas
Period20/11/920/11/13

Keywords

  • Human-centered computing
  • Human-centered computing
  • Treemaps
  • Visualization
  • Visualization
  • Visualization design and evaluation methods
  • Visualization techniques

ASJC Scopus subject areas

  • Computer Vision and Pattern Recognition
  • Computer Science Applications
  • Software

Fingerprint Dive into the research topics of 'Digital Full-Face Mask Display with Expression Recognition using Embedded Photo Reflective Sensor Arrays'. Together they form a unique fingerprint.

Cite this