AffectiveHMD: Facial expression recognition in head mounted display using embedded photo reflective sensors

Masaaki Murakami, Kosuke Kikui, Katsuhiro Suzuki, Fumihiko Nakamura, Masaaki Fukuoka, Katsutoshi Masai, Yuta Sugiura, Maki Sugimoto

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We propose a facial expression mapping technology between virtual avatars and Head-Mounted Display (HMD) users. HMDs allow people to enjoy an immersive Virtual Reality (VR) experience. A virtual avatar can be a representative of the user in the virtual environment. However, the synchronization of the virtual avatar’s expressions with those of the HMD user is limited. The major problem of wearing an HMD is that a large portion of the user’s face is occluded, making facial recognition difficult in an HMD-based virtual environment. To overcome this problem, we propose a facial expression mapping technology using photo-reflective sensors. The sensors attached inside the HMD measure the reflection intensity between the sensors and the user’s face. The intensity values of five basic facial expressions (Neutral, Happy, Angry, Surprised, and Sad) are used for training a classifier to estimate the facial expression of a user. In Siggraph 2019, the user can enjoy two application, the facial expression synchronization with the avatar, and simple manipulation experience for a virtual environment by facial expressions.

Original languageEnglish
Title of host publicationACM SIGGRAPH 2019 Emerging Technologies, SIGGRAPH 2019
PublisherAssociation for Computing Machinery, Inc
ISBN (Electronic)9781450363082
DOIs
Publication statusPublished - 2019 Jul 28
EventACM SIGGRAPH 2019 Emerging Technologies - International Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2019 - Los Angeles, United States
Duration: 2019 Jul 28 → …

Publication series

NameACM SIGGRAPH 2019 Emerging Technologies, SIGGRAPH 2019

Conference

ConferenceACM SIGGRAPH 2019 Emerging Technologies - International Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2019
CountryUnited States
CityLos Angeles
Period19/7/28 → …

Fingerprint

Display devices
Virtual reality
Sensors
Synchronization
Helmet mounted displays
Classifiers

Keywords

  • Avatar
  • Classification
  • Facial expression recognition

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Human-Computer Interaction

Cite this

Murakami, M., Kikui, K., Suzuki, K., Nakamura, F., Fukuoka, M., Masai, K., ... Sugimoto, M. (2019). AffectiveHMD: Facial expression recognition in head mounted display using embedded photo reflective sensors. In ACM SIGGRAPH 2019 Emerging Technologies, SIGGRAPH 2019 (ACM SIGGRAPH 2019 Emerging Technologies, SIGGRAPH 2019). Association for Computing Machinery, Inc. https://doi.org/10.1145/3305367.3335039

AffectiveHMD : Facial expression recognition in head mounted display using embedded photo reflective sensors. / Murakami, Masaaki; Kikui, Kosuke; Suzuki, Katsuhiro; Nakamura, Fumihiko; Fukuoka, Masaaki; Masai, Katsutoshi; Sugiura, Yuta; Sugimoto, Maki.

ACM SIGGRAPH 2019 Emerging Technologies, SIGGRAPH 2019. Association for Computing Machinery, Inc, 2019. (ACM SIGGRAPH 2019 Emerging Technologies, SIGGRAPH 2019).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Murakami, M, Kikui, K, Suzuki, K, Nakamura, F, Fukuoka, M, Masai, K, Sugiura, Y & Sugimoto, M 2019, AffectiveHMD: Facial expression recognition in head mounted display using embedded photo reflective sensors. in ACM SIGGRAPH 2019 Emerging Technologies, SIGGRAPH 2019. ACM SIGGRAPH 2019 Emerging Technologies, SIGGRAPH 2019, Association for Computing Machinery, Inc, ACM SIGGRAPH 2019 Emerging Technologies - International Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2019, Los Angeles, United States, 19/7/28. https://doi.org/10.1145/3305367.3335039
Murakami M, Kikui K, Suzuki K, Nakamura F, Fukuoka M, Masai K et al. AffectiveHMD: Facial expression recognition in head mounted display using embedded photo reflective sensors. In ACM SIGGRAPH 2019 Emerging Technologies, SIGGRAPH 2019. Association for Computing Machinery, Inc. 2019. (ACM SIGGRAPH 2019 Emerging Technologies, SIGGRAPH 2019). https://doi.org/10.1145/3305367.3335039
Murakami, Masaaki ; Kikui, Kosuke ; Suzuki, Katsuhiro ; Nakamura, Fumihiko ; Fukuoka, Masaaki ; Masai, Katsutoshi ; Sugiura, Yuta ; Sugimoto, Maki. / AffectiveHMD : Facial expression recognition in head mounted display using embedded photo reflective sensors. ACM SIGGRAPH 2019 Emerging Technologies, SIGGRAPH 2019. Association for Computing Machinery, Inc, 2019. (ACM SIGGRAPH 2019 Emerging Technologies, SIGGRAPH 2019).
@inproceedings{b636a81f486049108a1b0d4865b326e0,
title = "AffectiveHMD: Facial expression recognition in head mounted display using embedded photo reflective sensors",
abstract = "We propose a facial expression mapping technology between virtual avatars and Head-Mounted Display (HMD) users. HMDs allow people to enjoy an immersive Virtual Reality (VR) experience. A virtual avatar can be a representative of the user in the virtual environment. However, the synchronization of the virtual avatar’s expressions with those of the HMD user is limited. The major problem of wearing an HMD is that a large portion of the user’s face is occluded, making facial recognition difficult in an HMD-based virtual environment. To overcome this problem, we propose a facial expression mapping technology using photo-reflective sensors. The sensors attached inside the HMD measure the reflection intensity between the sensors and the user’s face. The intensity values of five basic facial expressions (Neutral, Happy, Angry, Surprised, and Sad) are used for training a classifier to estimate the facial expression of a user. In Siggraph 2019, the user can enjoy two application, the facial expression synchronization with the avatar, and simple manipulation experience for a virtual environment by facial expressions.",
keywords = "Avatar, Classification, Facial expression recognition",
author = "Masaaki Murakami and Kosuke Kikui and Katsuhiro Suzuki and Fumihiko Nakamura and Masaaki Fukuoka and Katsutoshi Masai and Yuta Sugiura and Maki Sugimoto",
year = "2019",
month = "7",
day = "28",
doi = "10.1145/3305367.3335039",
language = "English",
series = "ACM SIGGRAPH 2019 Emerging Technologies, SIGGRAPH 2019",
publisher = "Association for Computing Machinery, Inc",
booktitle = "ACM SIGGRAPH 2019 Emerging Technologies, SIGGRAPH 2019",

}

TY - GEN

T1 - AffectiveHMD

T2 - Facial expression recognition in head mounted display using embedded photo reflective sensors

AU - Murakami, Masaaki

AU - Kikui, Kosuke

AU - Suzuki, Katsuhiro

AU - Nakamura, Fumihiko

AU - Fukuoka, Masaaki

AU - Masai, Katsutoshi

AU - Sugiura, Yuta

AU - Sugimoto, Maki

PY - 2019/7/28

Y1 - 2019/7/28

N2 - We propose a facial expression mapping technology between virtual avatars and Head-Mounted Display (HMD) users. HMDs allow people to enjoy an immersive Virtual Reality (VR) experience. A virtual avatar can be a representative of the user in the virtual environment. However, the synchronization of the virtual avatar’s expressions with those of the HMD user is limited. The major problem of wearing an HMD is that a large portion of the user’s face is occluded, making facial recognition difficult in an HMD-based virtual environment. To overcome this problem, we propose a facial expression mapping technology using photo-reflective sensors. The sensors attached inside the HMD measure the reflection intensity between the sensors and the user’s face. The intensity values of five basic facial expressions (Neutral, Happy, Angry, Surprised, and Sad) are used for training a classifier to estimate the facial expression of a user. In Siggraph 2019, the user can enjoy two application, the facial expression synchronization with the avatar, and simple manipulation experience for a virtual environment by facial expressions.

AB - We propose a facial expression mapping technology between virtual avatars and Head-Mounted Display (HMD) users. HMDs allow people to enjoy an immersive Virtual Reality (VR) experience. A virtual avatar can be a representative of the user in the virtual environment. However, the synchronization of the virtual avatar’s expressions with those of the HMD user is limited. The major problem of wearing an HMD is that a large portion of the user’s face is occluded, making facial recognition difficult in an HMD-based virtual environment. To overcome this problem, we propose a facial expression mapping technology using photo-reflective sensors. The sensors attached inside the HMD measure the reflection intensity between the sensors and the user’s face. The intensity values of five basic facial expressions (Neutral, Happy, Angry, Surprised, and Sad) are used for training a classifier to estimate the facial expression of a user. In Siggraph 2019, the user can enjoy two application, the facial expression synchronization with the avatar, and simple manipulation experience for a virtual environment by facial expressions.

KW - Avatar

KW - Classification

KW - Facial expression recognition

UR - http://www.scopus.com/inward/record.url?scp=85071251073&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85071251073&partnerID=8YFLogxK

U2 - 10.1145/3305367.3335039

DO - 10.1145/3305367.3335039

M3 - Conference contribution

AN - SCOPUS:85071251073

T3 - ACM SIGGRAPH 2019 Emerging Technologies, SIGGRAPH 2019

BT - ACM SIGGRAPH 2019 Emerging Technologies, SIGGRAPH 2019

PB - Association for Computing Machinery, Inc

ER -