Alleviating over-segmentation errors by detecting action boundaries

Yuchi Ishikawa, Seito Kasai, Yoshimitsu Aoki, Hirokatsu Kataoka

研究成果: Conference contribution

29 被引用数 (Scopus)

抄録

We propose an effective framework for the temporal action segmentation task, namely an Action Segment Refinement Framework (ASRF). Our model architecture consists of a long-term feature extractor and two branches: the Action Segmentation Branch (ASB) and the Boundary Regression Branch (BRB). The long-term feature extractor provides shared features for the two branches with a wide temporal receptive field. The ASB classifies video frames with action classes, while the BRB regresses the action boundary probabilities. The action boundaries predicted by the BRB refine the output from the ASB, which results in a significant performance improvement. Our contributions are three-fold: (i) We propose a framework for temporal action segmentation, the ASRF, which divides temporal action segmentation into frame-wise action classification and action boundary regression. Our framework refines frame-level hypotheses of action classes using predicted action boundaries. (ii) We propose a loss function for smoothing the transition of action probabilities, and analyze combinations of various loss functions for temporal action segmentation. (iii) Our framework outperforms state-of-the-art methods on three challenging datasets, offering an improvement of up to 13.7% in terms of segmental edit distance and up to 16.1% in terms of segmental F1 score. Our code is publicly available1.

本文言語English
ホスト出版物のタイトルProceedings - 2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021
出版社Institute of Electrical and Electronics Engineers Inc.
ページ2321-2330
ページ数10
ISBN(電子版)9780738142661
DOI
出版ステータスPublished - 2021 1月
イベント2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021 - Virtual, Online, United States
継続期間: 2021 1月 52021 1月 9

出版物シリーズ

名前Proceedings - 2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021

Conference

Conference2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021
国/地域United States
CityVirtual, Online
Period21/1/521/1/9

ASJC Scopus subject areas

  • コンピュータ ビジョンおよびパターン認識
  • コンピュータ サイエンスの応用

引用スタイル