3D PixMix: Image Inpainting in 3D Environments

Shohei Mori, Jan Herling, Wolfgang Broll, Norihiko Kawai, Hideo Saito, Dieter Schmalstieg, Denis Kalkofen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

State of the art methods for diminished reality (DR) propagate pixel information from keyframes to later frames to achieve real-time image inpainting in 3D spaces. However, this approach assumes a planar scene and produces artifacts, if the scene geometry is not sufficiently planar. In this paper, we present 3D PixMix, a new real-time inpainting method that addresses non-planar scenes by considering both color and depth information in the inpainting process. We define cost functions for both the color and the geometric appearance in the inpainting scheme and use an RGB-D sensor for depth fusion using a SLAM. Comparison results against the conventional PixMix show that 3D PixMix obtains the equivalent or even better quality in 3D scenes with additional depth information.

Original languageEnglish
Title of host publicationAdjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1-2
Number of pages2
ISBN (Electronic)9781538675922
DOIs
Publication statusPublished - 2019 Apr 25
Event17th IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018 - Munich, Germany
Duration: 2018 Oct 162018 Oct 20

Publication series

NameAdjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018

Conference

Conference17th IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018
CountryGermany
CityMunich
Period18/10/1618/10/20

Fingerprint

Color
Cost functions
Fusion reactions
Pixels
Geometry
Sensors

Keywords

  • Artificial
  • Augmented
  • H.5.1 [Information Interfaces and Presentation]
  • Multimedia Information Systems
  • Virtual Realities

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Artificial Intelligence

Cite this

Mori, S., Herling, J., Broll, W., Kawai, N., Saito, H., Schmalstieg, D., & Kalkofen, D. (2019). 3D PixMix: Image Inpainting in 3D Environments. In Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018 (pp. 1-2). [8699212] (Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ISMAR-Adjunct.2018.00020

3D PixMix : Image Inpainting in 3D Environments. / Mori, Shohei; Herling, Jan; Broll, Wolfgang; Kawai, Norihiko; Saito, Hideo; Schmalstieg, Dieter; Kalkofen, Denis.

Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018. Institute of Electrical and Electronics Engineers Inc., 2019. p. 1-2 8699212 (Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Mori, S, Herling, J, Broll, W, Kawai, N, Saito, H, Schmalstieg, D & Kalkofen, D 2019, 3D PixMix: Image Inpainting in 3D Environments. in Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018., 8699212, Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018, Institute of Electrical and Electronics Engineers Inc., pp. 1-2, 17th IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018, Munich, Germany, 18/10/16. https://doi.org/10.1109/ISMAR-Adjunct.2018.00020
Mori S, Herling J, Broll W, Kawai N, Saito H, Schmalstieg D et al. 3D PixMix: Image Inpainting in 3D Environments. In Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018. Institute of Electrical and Electronics Engineers Inc. 2019. p. 1-2. 8699212. (Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018). https://doi.org/10.1109/ISMAR-Adjunct.2018.00020
Mori, Shohei ; Herling, Jan ; Broll, Wolfgang ; Kawai, Norihiko ; Saito, Hideo ; Schmalstieg, Dieter ; Kalkofen, Denis. / 3D PixMix : Image Inpainting in 3D Environments. Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 1-2 (Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018).
@inproceedings{adb5d2e7aaa240339dae94581ad29c0b,
title = "3D PixMix: Image Inpainting in 3D Environments",
abstract = "State of the art methods for diminished reality (DR) propagate pixel information from keyframes to later frames to achieve real-time image inpainting in 3D spaces. However, this approach assumes a planar scene and produces artifacts, if the scene geometry is not sufficiently planar. In this paper, we present 3D PixMix, a new real-time inpainting method that addresses non-planar scenes by considering both color and depth information in the inpainting process. We define cost functions for both the color and the geometric appearance in the inpainting scheme and use an RGB-D sensor for depth fusion using a SLAM. Comparison results against the conventional PixMix show that 3D PixMix obtains the equivalent or even better quality in 3D scenes with additional depth information.",
keywords = "Artificial, Augmented, H.5.1 [Information Interfaces and Presentation], Multimedia Information Systems, Virtual Realities",
author = "Shohei Mori and Jan Herling and Wolfgang Broll and Norihiko Kawai and Hideo Saito and Dieter Schmalstieg and Denis Kalkofen",
year = "2019",
month = "4",
day = "25",
doi = "10.1109/ISMAR-Adjunct.2018.00020",
language = "English",
series = "Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "1--2",
booktitle = "Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018",

}

TY - GEN

T1 - 3D PixMix

T2 - Image Inpainting in 3D Environments

AU - Mori, Shohei

AU - Herling, Jan

AU - Broll, Wolfgang

AU - Kawai, Norihiko

AU - Saito, Hideo

AU - Schmalstieg, Dieter

AU - Kalkofen, Denis

PY - 2019/4/25

Y1 - 2019/4/25

N2 - State of the art methods for diminished reality (DR) propagate pixel information from keyframes to later frames to achieve real-time image inpainting in 3D spaces. However, this approach assumes a planar scene and produces artifacts, if the scene geometry is not sufficiently planar. In this paper, we present 3D PixMix, a new real-time inpainting method that addresses non-planar scenes by considering both color and depth information in the inpainting process. We define cost functions for both the color and the geometric appearance in the inpainting scheme and use an RGB-D sensor for depth fusion using a SLAM. Comparison results against the conventional PixMix show that 3D PixMix obtains the equivalent or even better quality in 3D scenes with additional depth information.

AB - State of the art methods for diminished reality (DR) propagate pixel information from keyframes to later frames to achieve real-time image inpainting in 3D spaces. However, this approach assumes a planar scene and produces artifacts, if the scene geometry is not sufficiently planar. In this paper, we present 3D PixMix, a new real-time inpainting method that addresses non-planar scenes by considering both color and depth information in the inpainting process. We define cost functions for both the color and the geometric appearance in the inpainting scheme and use an RGB-D sensor for depth fusion using a SLAM. Comparison results against the conventional PixMix show that 3D PixMix obtains the equivalent or even better quality in 3D scenes with additional depth information.

KW - Artificial

KW - Augmented

KW - H.5.1 [Information Interfaces and Presentation]

KW - Multimedia Information Systems

KW - Virtual Realities

UR - http://www.scopus.com/inward/record.url?scp=85065548979&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85065548979&partnerID=8YFLogxK

U2 - 10.1109/ISMAR-Adjunct.2018.00020

DO - 10.1109/ISMAR-Adjunct.2018.00020

M3 - Conference contribution

AN - SCOPUS:85065548979

T3 - Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018

SP - 1

EP - 2

BT - Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018

PB - Institute of Electrical and Electronics Engineers Inc.

ER -