Semi-automation of gesture annotation by machine learning and human collaboration

Naoto Ienaga, Alice Cravotta, Kei Terayama, Bryan W. Scotney, Hideo Saito, M. Grazia Busà

研究成果: Article査読


Gesture and multimodal communication researchers typically annotate video data manually, even though this can be a very time-consuming task. In the present work, a method to detect gestures is proposed as a fundamental step towards a semi-automatic gesture annotation tool. The proposed method can be applied to RGB videos and requires annotations of part of a video as input. The technique deploys a pose estimation method and active learning. In the experiment, it is shown that if about 27% of the video is annotated, the remaining parts of the video can be annotated automatically with an F-score of at least 0.85. Users can run this tool with a small number of annotations first. If the predicted annotations for the remainder of the video are not satisfactory, users can add further annotations and run the tool again. The code has been released so that other researchers and practitioners can use the results of this research. This tool has been confirmed to work in conjunction with ELAN.

ジャーナルLanguage Resources and Evaluation
出版ステータスPublished - 2022 9月

ASJC Scopus subject areas

  • 言語および言語学
  • 教育
  • 言語学および言語
  • 図書館情報学


「Semi-automation of gesture annotation by machine learning and human collaboration」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。