Semantic Segmentation of 3D Point Cloud to Virtually Manipulate Real Living Space

Yuki Ishikawa, Ryo Hachiuma, Naoto Ienaga, Wakaba Kuno, Yuta Sugiura, Hideo Saito

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This paper presents a method for the virtual manipulation of real living space using semantic segmentation of a 3D point cloud captured in the real world. We applied PointNet to segment each piece of furniture from the point cloud of a real indoor environment captured by moving a RGB-D camera. For semantic segmentation, we focused on local geometric information not used in PointNet, and we proposed a method to refine the class probability of labels attached to each point in PointNet's output. The effectiveness of our method was experimentally confirmed. We then created 3D models of real-world furniture using a point cloud with corrected labels, and we virtually manipulated real living space using Dollhouse VR, a layout system.

Original languageEnglish
Title of host publicationProceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019
EditorsDongdong Weng, Liwei Chan, Youngho Lee, Xiaohui Liang, Nobuchika Sakata
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781728115719
DOIs
Publication statusPublished - 2019 May 7
Event12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019 - Ikoma, Nara, Japan
Duration: 2019 Mar 282019 Mar 29

Publication series

NameProceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019

Conference

Conference12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019
CountryJapan
CityIkoma, Nara
Period19/3/2819/3/29

Fingerprint

Semantics
Interior Design and Furnishings
Labels
semantics
Cameras
layout
manipulation
segmentation
Segmentation

Keywords

  • furniture arrangement
  • semantic segmentation
  • virtual reality

ASJC Scopus subject areas

  • Education
  • Sociology and Political Science
  • Communication
  • Marketing
  • Strategy and Management
  • Computer Networks and Communications
  • Information Systems and Management
  • Social Psychology

Cite this

Ishikawa, Y., Hachiuma, R., Ienaga, N., Kuno, W., Sugiura, Y., & Saito, H. (2019). Semantic Segmentation of 3D Point Cloud to Virtually Manipulate Real Living Space. In D. Weng, L. Chan, Y. Lee, X. Liang, & N. Sakata (Eds.), Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019 [8709156] (Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/APMAR.2019.8709156

Semantic Segmentation of 3D Point Cloud to Virtually Manipulate Real Living Space. / Ishikawa, Yuki; Hachiuma, Ryo; Ienaga, Naoto; Kuno, Wakaba; Sugiura, Yuta; Saito, Hideo.

Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019. ed. / Dongdong Weng; Liwei Chan; Youngho Lee; Xiaohui Liang; Nobuchika Sakata. Institute of Electrical and Electronics Engineers Inc., 2019. 8709156 (Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Ishikawa, Y, Hachiuma, R, Ienaga, N, Kuno, W, Sugiura, Y & Saito, H 2019, Semantic Segmentation of 3D Point Cloud to Virtually Manipulate Real Living Space. in D Weng, L Chan, Y Lee, X Liang & N Sakata (eds), Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019., 8709156, Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019, Institute of Electrical and Electronics Engineers Inc., 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019, Ikoma, Nara, Japan, 19/3/28. https://doi.org/10.1109/APMAR.2019.8709156
Ishikawa Y, Hachiuma R, Ienaga N, Kuno W, Sugiura Y, Saito H. Semantic Segmentation of 3D Point Cloud to Virtually Manipulate Real Living Space. In Weng D, Chan L, Lee Y, Liang X, Sakata N, editors, Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019. Institute of Electrical and Electronics Engineers Inc. 2019. 8709156. (Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019). https://doi.org/10.1109/APMAR.2019.8709156
Ishikawa, Yuki ; Hachiuma, Ryo ; Ienaga, Naoto ; Kuno, Wakaba ; Sugiura, Yuta ; Saito, Hideo. / Semantic Segmentation of 3D Point Cloud to Virtually Manipulate Real Living Space. Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019. editor / Dongdong Weng ; Liwei Chan ; Youngho Lee ; Xiaohui Liang ; Nobuchika Sakata. Institute of Electrical and Electronics Engineers Inc., 2019. (Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019).
@inproceedings{ce2498a6ec074a7d92b89479784eccb3,
title = "Semantic Segmentation of 3D Point Cloud to Virtually Manipulate Real Living Space",
abstract = "This paper presents a method for the virtual manipulation of real living space using semantic segmentation of a 3D point cloud captured in the real world. We applied PointNet to segment each piece of furniture from the point cloud of a real indoor environment captured by moving a RGB-D camera. For semantic segmentation, we focused on local geometric information not used in PointNet, and we proposed a method to refine the class probability of labels attached to each point in PointNet's output. The effectiveness of our method was experimentally confirmed. We then created 3D models of real-world furniture using a point cloud with corrected labels, and we virtually manipulated real living space using Dollhouse VR, a layout system.",
keywords = "furniture arrangement, semantic segmentation, virtual reality",
author = "Yuki Ishikawa and Ryo Hachiuma and Naoto Ienaga and Wakaba Kuno and Yuta Sugiura and Hideo Saito",
year = "2019",
month = "5",
day = "7",
doi = "10.1109/APMAR.2019.8709156",
language = "English",
series = "Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
editor = "Dongdong Weng and Liwei Chan and Youngho Lee and Xiaohui Liang and Nobuchika Sakata",
booktitle = "Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019",

}

TY - GEN

T1 - Semantic Segmentation of 3D Point Cloud to Virtually Manipulate Real Living Space

AU - Ishikawa, Yuki

AU - Hachiuma, Ryo

AU - Ienaga, Naoto

AU - Kuno, Wakaba

AU - Sugiura, Yuta

AU - Saito, Hideo

PY - 2019/5/7

Y1 - 2019/5/7

N2 - This paper presents a method for the virtual manipulation of real living space using semantic segmentation of a 3D point cloud captured in the real world. We applied PointNet to segment each piece of furniture from the point cloud of a real indoor environment captured by moving a RGB-D camera. For semantic segmentation, we focused on local geometric information not used in PointNet, and we proposed a method to refine the class probability of labels attached to each point in PointNet's output. The effectiveness of our method was experimentally confirmed. We then created 3D models of real-world furniture using a point cloud with corrected labels, and we virtually manipulated real living space using Dollhouse VR, a layout system.

AB - This paper presents a method for the virtual manipulation of real living space using semantic segmentation of a 3D point cloud captured in the real world. We applied PointNet to segment each piece of furniture from the point cloud of a real indoor environment captured by moving a RGB-D camera. For semantic segmentation, we focused on local geometric information not used in PointNet, and we proposed a method to refine the class probability of labels attached to each point in PointNet's output. The effectiveness of our method was experimentally confirmed. We then created 3D models of real-world furniture using a point cloud with corrected labels, and we virtually manipulated real living space using Dollhouse VR, a layout system.

KW - furniture arrangement

KW - semantic segmentation

KW - virtual reality

UR - http://www.scopus.com/inward/record.url?scp=85066317359&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85066317359&partnerID=8YFLogxK

U2 - 10.1109/APMAR.2019.8709156

DO - 10.1109/APMAR.2019.8709156

M3 - Conference contribution

T3 - Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019

BT - Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019

A2 - Weng, Dongdong

A2 - Chan, Liwei

A2 - Lee, Youngho

A2 - Liang, Xiaohui

A2 - Sakata, Nobuchika

PB - Institute of Electrical and Electronics Engineers Inc.

ER -