Sentiment pen: recognizing emotional context based on handwriting features

Jiawen Han, George Chernyshov, Dingding Zheng, Peizhong Gao, Takuji Narumi, Katrin Wolf, Kai Steven Kunze

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper, we discuss the assessment of the emotional state of the user from digitized handwriting for implicit human-computer interaction. The proposed concept exemplifies how a digital system could recognize the emotional context of the interaction.We discuss our approach to emotion recognition and the underlying neurophysiological mechanisms. To verify the viability of our approach, we have conducted a series of tests where participants were asked to perform simple writing tasks after being exposed to a series of emotionally-stimulating video clips from EMDB[6], one set of four clips per each quadrant on the circumplex model of emotion[28]. The user-independent Support Vector Classifier (SVC) built using the recorded data shows up to 66% accuracy for certain types of writing tasks for 1 in 4 classification (1. High Valence, High Arousal; 2. High Valence, Low Arousal; 3. Low Valence, High Arousal; 4. Low Valence, Low Arousal). In the same conditions, a user-dependent classifier reaches an average of 70% accuracy across all 12 study participants. While future work is required to improve the classification rate, this work should be seen as proof-of-concept for emotion assessment of users while handwriting aiming to motivate research on implicit interaction while writing to enable emotion-sensitivity in mobile and ubiquitous computing.

Original languageEnglish
Title of host publicationProceedings of the 10th Augmented Human International Conference, AH 2019
PublisherAssociation for Computing Machinery
ISBN (Electronic)9781450365475
DOIs
Publication statusPublished - 2019 Mar 11
Externally publishedYes
Event10th Augmented Human International Conference, AH 2019 - Reims, France
Duration: 2019 Mar 112019 Mar 12

Publication series

NameACM International Conference Proceeding Series

Conference

Conference10th Augmented Human International Conference, AH 2019
CountryFrance
CityReims
Period19/3/1119/3/12

Fingerprint

Classifiers
Mobile computing
Ubiquitous computing
Human computer interaction

Keywords

  • Affective computing
  • Emotional recognition
  • Handwriting analysis

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Computer Networks and Communications
  • Computer Vision and Pattern Recognition
  • Software

Cite this

Han, J., Chernyshov, G., Zheng, D., Gao, P., Narumi, T., Wolf, K., & Kunze, K. S. (2019). Sentiment pen: recognizing emotional context based on handwriting features. In Proceedings of the 10th Augmented Human International Conference, AH 2019 [a24] (ACM International Conference Proceeding Series). Association for Computing Machinery. https://doi.org/10.1145/3311823.3311868

Sentiment pen : recognizing emotional context based on handwriting features. / Han, Jiawen; Chernyshov, George; Zheng, Dingding; Gao, Peizhong; Narumi, Takuji; Wolf, Katrin; Kunze, Kai Steven.

Proceedings of the 10th Augmented Human International Conference, AH 2019. Association for Computing Machinery, 2019. a24 (ACM International Conference Proceeding Series).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Han, J, Chernyshov, G, Zheng, D, Gao, P, Narumi, T, Wolf, K & Kunze, KS 2019, Sentiment pen: recognizing emotional context based on handwriting features. in Proceedings of the 10th Augmented Human International Conference, AH 2019., a24, ACM International Conference Proceeding Series, Association for Computing Machinery, 10th Augmented Human International Conference, AH 2019, Reims, France, 19/3/11. https://doi.org/10.1145/3311823.3311868
Han J, Chernyshov G, Zheng D, Gao P, Narumi T, Wolf K et al. Sentiment pen: recognizing emotional context based on handwriting features. In Proceedings of the 10th Augmented Human International Conference, AH 2019. Association for Computing Machinery. 2019. a24. (ACM International Conference Proceeding Series). https://doi.org/10.1145/3311823.3311868
Han, Jiawen ; Chernyshov, George ; Zheng, Dingding ; Gao, Peizhong ; Narumi, Takuji ; Wolf, Katrin ; Kunze, Kai Steven. / Sentiment pen : recognizing emotional context based on handwriting features. Proceedings of the 10th Augmented Human International Conference, AH 2019. Association for Computing Machinery, 2019. (ACM International Conference Proceeding Series).
@inproceedings{baf11ee2051e4beb80ae257083ed9e8e,
title = "Sentiment pen: recognizing emotional context based on handwriting features",
abstract = "In this paper, we discuss the assessment of the emotional state of the user from digitized handwriting for implicit human-computer interaction. The proposed concept exemplifies how a digital system could recognize the emotional context of the interaction.We discuss our approach to emotion recognition and the underlying neurophysiological mechanisms. To verify the viability of our approach, we have conducted a series of tests where participants were asked to perform simple writing tasks after being exposed to a series of emotionally-stimulating video clips from EMDB[6], one set of four clips per each quadrant on the circumplex model of emotion[28]. The user-independent Support Vector Classifier (SVC) built using the recorded data shows up to 66{\%} accuracy for certain types of writing tasks for 1 in 4 classification (1. High Valence, High Arousal; 2. High Valence, Low Arousal; 3. Low Valence, High Arousal; 4. Low Valence, Low Arousal). In the same conditions, a user-dependent classifier reaches an average of 70{\%} accuracy across all 12 study participants. While future work is required to improve the classification rate, this work should be seen as proof-of-concept for emotion assessment of users while handwriting aiming to motivate research on implicit interaction while writing to enable emotion-sensitivity in mobile and ubiquitous computing.",
keywords = "Affective computing, Emotional recognition, Handwriting analysis",
author = "Jiawen Han and George Chernyshov and Dingding Zheng and Peizhong Gao and Takuji Narumi and Katrin Wolf and Kunze, {Kai Steven}",
year = "2019",
month = "3",
day = "11",
doi = "10.1145/3311823.3311868",
language = "English",
series = "ACM International Conference Proceeding Series",
publisher = "Association for Computing Machinery",
booktitle = "Proceedings of the 10th Augmented Human International Conference, AH 2019",

}

TY - GEN

T1 - Sentiment pen

T2 - recognizing emotional context based on handwriting features

AU - Han, Jiawen

AU - Chernyshov, George

AU - Zheng, Dingding

AU - Gao, Peizhong

AU - Narumi, Takuji

AU - Wolf, Katrin

AU - Kunze, Kai Steven

PY - 2019/3/11

Y1 - 2019/3/11

N2 - In this paper, we discuss the assessment of the emotional state of the user from digitized handwriting for implicit human-computer interaction. The proposed concept exemplifies how a digital system could recognize the emotional context of the interaction.We discuss our approach to emotion recognition and the underlying neurophysiological mechanisms. To verify the viability of our approach, we have conducted a series of tests where participants were asked to perform simple writing tasks after being exposed to a series of emotionally-stimulating video clips from EMDB[6], one set of four clips per each quadrant on the circumplex model of emotion[28]. The user-independent Support Vector Classifier (SVC) built using the recorded data shows up to 66% accuracy for certain types of writing tasks for 1 in 4 classification (1. High Valence, High Arousal; 2. High Valence, Low Arousal; 3. Low Valence, High Arousal; 4. Low Valence, Low Arousal). In the same conditions, a user-dependent classifier reaches an average of 70% accuracy across all 12 study participants. While future work is required to improve the classification rate, this work should be seen as proof-of-concept for emotion assessment of users while handwriting aiming to motivate research on implicit interaction while writing to enable emotion-sensitivity in mobile and ubiquitous computing.

AB - In this paper, we discuss the assessment of the emotional state of the user from digitized handwriting for implicit human-computer interaction. The proposed concept exemplifies how a digital system could recognize the emotional context of the interaction.We discuss our approach to emotion recognition and the underlying neurophysiological mechanisms. To verify the viability of our approach, we have conducted a series of tests where participants were asked to perform simple writing tasks after being exposed to a series of emotionally-stimulating video clips from EMDB[6], one set of four clips per each quadrant on the circumplex model of emotion[28]. The user-independent Support Vector Classifier (SVC) built using the recorded data shows up to 66% accuracy for certain types of writing tasks for 1 in 4 classification (1. High Valence, High Arousal; 2. High Valence, Low Arousal; 3. Low Valence, High Arousal; 4. Low Valence, Low Arousal). In the same conditions, a user-dependent classifier reaches an average of 70% accuracy across all 12 study participants. While future work is required to improve the classification rate, this work should be seen as proof-of-concept for emotion assessment of users while handwriting aiming to motivate research on implicit interaction while writing to enable emotion-sensitivity in mobile and ubiquitous computing.

KW - Affective computing

KW - Emotional recognition

KW - Handwriting analysis

UR - http://www.scopus.com/inward/record.url?scp=85062980704&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85062980704&partnerID=8YFLogxK

U2 - 10.1145/3311823.3311868

DO - 10.1145/3311823.3311868

M3 - Conference contribution

AN - SCOPUS:85062980704

T3 - ACM International Conference Proceeding Series

BT - Proceedings of the 10th Augmented Human International Conference, AH 2019

PB - Association for Computing Machinery

ER -