Self-Supervised Audio-Visual Feature Learning for Single-Modal Incremental Terrain Type Clustering

Reina Ishikawa, Ryo Hachiuma, Hideo Saito

Research output: Contribution to journalArticlepeer-review

2 Citations (Scopus)


The key to an accurate understanding of terrain is to extract the informative features from the multi-modal data obtained from different devices. Sensors, such as RGB cameras, depth sensors, vibration sensors, and microphones, are used as the multi-modal data. Many studies have explored ways to use them, especially in the robotics field. Some papers have successfully introduced single-modal or multi-modal methods. However, in practice, robots can be faced with extreme conditions; microphones do not work well in crowded scenes, and an RGB camera cannot capture terrains well in the dark. In this paper, we present a novel framework using the multi-modal variational autoencoder and the Gaussian mixture model clustering algorithm on image data and audio data for terrain type clustering by forcing the features to be closer together in the feature space. Our method enables the terrain type clustering even if one of the modalities (either image or audio) is missing at the test-time. We evaluated the clustering accuracy with a conventional multi-modal terrain type clustering method and we conducted ablation studies to show the effectiveness of our approach.

Original languageEnglish
Article number9416486
Pages (from-to)64346-64357
Number of pages12
JournalIEEE Access
Publication statusPublished - 2021


  • Self-supervised
  • multi-modal learning
  • terrain type clustering

ASJC Scopus subject areas

  • Computer Science(all)
  • Materials Science(all)
  • Engineering(all)


Dive into the research topics of 'Self-Supervised Audio-Visual Feature Learning for Single-Modal Incremental Terrain Type Clustering'. Together they form a unique fingerprint.

Cite this