Alleviating the Burden of Labeling: Sentence Generation by Attention Branch Encoder-Decoder Network

Tadashi Ogura, Aly Magassouba, Komei Sugiura, Tsubasa Hirakawa, Takayoshi Yamashita, Hironobu Fujiyoshi, Hisashi Kawai

Research output: Contribution to journalArticlepeer-review

4 Citations (Scopus)


Domestic service robots (DSRs) are a promising solution to the shortage of home care workers. However, one of the main limitations of DSRs is their inability to interact naturally through language. Recently, data-driven approaches have been shown to be effective for tackling this limitation; however, they often require large-scale datasets, which is costly. Based on this background, we aim to perform automatic sentence generation of fetching instructions: for example, 'Bring me a green tea bottle on the table.' This is particularly challenging because appropriate expressions depend on the target object, as well as its surroundings. In this letter, we propose the attention branch encoder-decoder network (ABEN), to generate sentences from visual inputs. Unlike other approaches, the ABEN has multimodal attention branches that use subword-level attention and generate sentences based on subword embeddings. In experiments, we compared the ABEN with a baseline method using four standard metrics in image captioning. Results show that the ABEN outperformed the baseline in terms of these metrics.

Original languageEnglish
Article number9145673
Pages (from-to)5945-5952
Number of pages8
JournalIEEE Robotics and Automation Letters
Issue number4
Publication statusPublished - 2020 Oct


  • Novel deep learning methods
  • deep learning for visual perception

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Biomedical Engineering
  • Human-Computer Interaction
  • Mechanical Engineering
  • Computer Vision and Pattern Recognition
  • Computer Science Applications
  • Control and Optimization
  • Artificial Intelligence


Dive into the research topics of 'Alleviating the Burden of Labeling: Sentence Generation by Attention Branch Encoder-Decoder Network'. Together they form a unique fingerprint.

Cite this