Fine-grained action recognition in assembly work scenes by drawing attention to the hands

Takuya Kobayashi, Yoshimitsu Aoki, Shogo Shimizu, Katsuhiro Kusano, Seiji Okumura

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The analysis of assembly working scenes is an important tool for improving work efficiency and detecting worker error. Many factories choose to analyze videos of employees working by human observation. It is more efficient, however, to approach this as an action segmentation task, but this is difficult due to the fine-grained actions. In this paper, we focus on featuring the workers hands in action segmentation to describe the detailed moves in assembly work scenes and utilize the tool or product information that the worker is using. We propose two methods to draw attention to the hand from each image in the video: the first is by cutting out the workers hand image and extracting features, and the second is by using an attention module specialized to extract hand features by the training of a regression network. For both methods we combine different types of features - image features and pose features - which are both important information in fine-grained actions. Due to the lack of action datasets that focus on assembly work scenes, we create a new assembly work dataset for our experiments, and the results of those experiments show that our method is effective for analyzing fine-grained action segmentation tasks.

Original languageEnglish
Title of host publicationProceedings - 15th International Conference on Signal Image Technology and Internet Based Systems, SISITS 2019
EditorsKokou Yetongnon, Albert Dipanda, Gabriella Sanniti di Baja, Luigi Gallo, Richard Chbeir
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages440-446
Number of pages7
ISBN (Electronic)9781728156866
DOIs
Publication statusPublished - 2019 Nov
Event15th International Conference on Signal Image Technology and Internet Based Systems, SISITS 2019 - Sorrento, Italy
Duration: 2019 Nov 262019 Nov 29

Publication series

NameProceedings - 15th International Conference on Signal Image Technology and Internet Based Systems, SISITS 2019

Conference

Conference15th International Conference on Signal Image Technology and Internet Based Systems, SISITS 2019
CountryItaly
CitySorrento
Period19/11/2619/11/29

Keywords

  • Action segmentation
  • Attention
  • Fine grained action
  • Pose regression

ASJC Scopus subject areas

  • Computer Science Applications
  • Signal Processing
  • Media Technology
  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'Fine-grained action recognition in assembly work scenes by drawing attention to the hands'. Together they form a unique fingerprint.

  • Cite this

    Kobayashi, T., Aoki, Y., Shimizu, S., Kusano, K., & Okumura, S. (2019). Fine-grained action recognition in assembly work scenes by drawing attention to the hands. In K. Yetongnon, A. Dipanda, G. Sanniti di Baja, L. Gallo, & R. Chbeir (Eds.), Proceedings - 15th International Conference on Signal Image Technology and Internet Based Systems, SISITS 2019 (pp. 440-446). [9067967] (Proceedings - 15th International Conference on Signal Image Technology and Internet Based Systems, SISITS 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/SITIS.2019.00077