Rgb-d image inpainting using generative adversarial network with a late fusion approach

Ryo Fujii, Ryo Hachiuma, Hideo Saito

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Diminished reality is a technology that aims to remove objects from video images and fills in the missing region with plausible pixels. Most conventional methods utilize the different cameras that capture the same scene from different viewpoints to allow regions to be removed and restored. In this paper, we propose an RGB-D image inpainting method using generative adversarial network, which does not require multiple cameras. Recently, an RGB image inpainting method has achieved outstanding results by employing a generative adversarial network. However, RGB inpainting methods aim to restore only the texture of the missing region and, therefore, does not recover geometric information (i.e, 3D structure of the scene). We expand conventional image inpainting method to RGB-D image inpainting to jointly restore the texture and geometry of missing regions from a pair of RGB and depth images. Inspired by other tasks that use RGB and depth images (e.g., semantic segmentation and object detection), we propose late fusion approach that exploits the advantage of RGB and depth information each other. The experimental results verify the effectiveness of our proposed method.

Original languageEnglish
Title of host publicationAugmented Reality, Virtual Reality, and Computer Graphics - 7th International Conference, AVR 2020, Proceedings
EditorsLucio Tommaso De Paolis, Patrick Bourdot
PublisherSpringer Science and Business Media Deutschland GmbH
Pages440-451
Number of pages12
ISBN (Print)9783030584641
DOIs
Publication statusPublished - 2020
Event7th International Conference on Augmented Reality, Virtual Reality, and Computer Graphics, AVR 2020 - Lecce, Italy
Duration: 2020 Sep 72020 Sep 10

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12242 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference7th International Conference on Augmented Reality, Virtual Reality, and Computer Graphics, AVR 2020
CountryItaly
CityLecce
Period20/9/720/9/10

Keywords

  • Generative adversarial network
  • Image inpainting
  • Mixed reality

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint Dive into the research topics of 'Rgb-d image inpainting using generative adversarial network with a late fusion approach'. Together they form a unique fingerprint.

Cite this