Semantic Segmentation of 3D Point Cloud to Virtually Manipulate Real Living Space

Yuki Ishikawa; Ryo Hachiuma; Naoto Ienaga; Wakaba Kuno; Yuta Sugiura; Hideo Saito

doi:10.1109/APMAR.2019.8709156

Semantic Segmentation of 3D Point Cloud to Virtually Manipulate Real Living Space

Yuki Ishikawa, Ryo Hachiuma, Naoto Ienaga, Wakaba Kuno, Yuta Sugiura, Hideo Saito

Department of Information and Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

9 Citations (Scopus)

Abstract

This paper presents a method for the virtual manipulation of real living space using semantic segmentation of a 3D point cloud captured in the real world. We applied PointNet to segment each piece of furniture from the point cloud of a real indoor environment captured by moving a RGB-D camera. For semantic segmentation, we focused on local geometric information not used in PointNet, and we proposed a method to refine the class probability of labels attached to each point in PointNet's output. The effectiveness of our method was experimentally confirmed. We then created 3D models of real-world furniture using a point cloud with corrected labels, and we virtually manipulated real living space using Dollhouse VR, a layout system.

Original language	English
Title of host publication	Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019
Editors	Dongdong Weng, Liwei Chan, Youngho Lee, Xiaohui Liang, Nobuchika Sakata
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781728115719
DOIs	https://doi.org/10.1109/APMAR.2019.8709156
Publication status	Published - 2019 May 7
Event	12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019 - Ikoma, Nara, Japan Duration: 2019 Mar 28 → 2019 Mar 29

Publication series

Name	Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019

Conference

Conference	12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019
Country/Territory	Japan
City	Ikoma, Nara
Period	19/3/28 → 19/3/29

Keywords

furniture arrangement
semantic segmentation
virtual reality

ASJC Scopus subject areas

Education
Sociology and Political Science
Communication
Marketing
Strategy and Management
Computer Networks and Communications
Information Systems and Management
Social Psychology

Access to Document

10.1109/APMAR.2019.8709156

Cite this

Ishikawa, Y., Hachiuma, R., Ienaga, N., Kuno, W., Sugiura, Y., & Saito, H. (2019). Semantic Segmentation of 3D Point Cloud to Virtually Manipulate Real Living Space. In D. Weng, L. Chan, Y. Lee, X. Liang, & N. Sakata (Eds.), Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019 [8709156] (Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/APMAR.2019.8709156

Semantic Segmentation of 3D Point Cloud to Virtually Manipulate Real Living Space. / Ishikawa, Yuki; Hachiuma, Ryo; Ienaga, Naoto et al.

Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019. ed. / Dongdong Weng; Liwei Chan; Youngho Lee; Xiaohui Liang; Nobuchika Sakata. Institute of Electrical and Electronics Engineers Inc., 2019. 8709156 (Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Ishikawa, Y, Hachiuma, R, Ienaga, N, Kuno, W, Sugiura, Y & Saito, H 2019, Semantic Segmentation of 3D Point Cloud to Virtually Manipulate Real Living Space. in D Weng, L Chan, Y Lee, X Liang & N Sakata (eds), Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019., 8709156, Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019, Institute of Electrical and Electronics Engineers Inc., 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019, Ikoma, Nara, Japan, 19/3/28. https://doi.org/10.1109/APMAR.2019.8709156

Ishikawa Y, Hachiuma R, Ienaga N, Kuno W, Sugiura Y , Saito H. Semantic Segmentation of 3D Point Cloud to Virtually Manipulate Real Living Space. In Weng D, Chan L, Lee Y, Liang X, Sakata N, editors, Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019. Institute of Electrical and Electronics Engineers Inc. 2019. 8709156. (Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019). doi: 10.1109/APMAR.2019.8709156

Ishikawa, Yuki ; Hachiuma, Ryo ; Ienaga, Naoto et al. / Semantic Segmentation of 3D Point Cloud to Virtually Manipulate Real Living Space. Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019. editor / Dongdong Weng ; Liwei Chan ; Youngho Lee ; Xiaohui Liang ; Nobuchika Sakata. Institute of Electrical and Electronics Engineers Inc., 2019. (Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019).

@inproceedings{ce2498a6ec074a7d92b89479784eccb3,

title = "Semantic Segmentation of 3D Point Cloud to Virtually Manipulate Real Living Space",

abstract = "This paper presents a method for the virtual manipulation of real living space using semantic segmentation of a 3D point cloud captured in the real world. We applied PointNet to segment each piece of furniture from the point cloud of a real indoor environment captured by moving a RGB-D camera. For semantic segmentation, we focused on local geometric information not used in PointNet, and we proposed a method to refine the class probability of labels attached to each point in PointNet's output. The effectiveness of our method was experimentally confirmed. We then created 3D models of real-world furniture using a point cloud with corrected labels, and we virtually manipulated real living space using Dollhouse VR, a layout system.",

keywords = "furniture arrangement, semantic segmentation, virtual reality",

author = "Yuki Ishikawa and Ryo Hachiuma and Naoto Ienaga and Wakaba Kuno and Yuta Sugiura and Hideo Saito",

note = "Funding Information: ACKNOWLEDGMENT This work was supported by JST AIP-PRISM GrantNumber JPMJCR18Y2. Publisher Copyright: {\textcopyright} 2019 IEEE.; 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019 ; Conference date: 28-03-2019 Through 29-03-2019",

year = "2019",

month = may,

day = "7",

doi = "10.1109/APMAR.2019.8709156",

language = "English",

series = "Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

editor = "Dongdong Weng and Liwei Chan and Youngho Lee and Xiaohui Liang and Nobuchika Sakata",

booktitle = "Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019",

}

TY - GEN

T1 - Semantic Segmentation of 3D Point Cloud to Virtually Manipulate Real Living Space

AU - Ishikawa, Yuki

AU - Hachiuma, Ryo

AU - Ienaga, Naoto

AU - Kuno, Wakaba

AU - Sugiura, Yuta

AU - Saito, Hideo

PY - 2019/5/7

Y1 - 2019/5/7

N2 - This paper presents a method for the virtual manipulation of real living space using semantic segmentation of a 3D point cloud captured in the real world. We applied PointNet to segment each piece of furniture from the point cloud of a real indoor environment captured by moving a RGB-D camera. For semantic segmentation, we focused on local geometric information not used in PointNet, and we proposed a method to refine the class probability of labels attached to each point in PointNet's output. The effectiveness of our method was experimentally confirmed. We then created 3D models of real-world furniture using a point cloud with corrected labels, and we virtually manipulated real living space using Dollhouse VR, a layout system.

AB - This paper presents a method for the virtual manipulation of real living space using semantic segmentation of a 3D point cloud captured in the real world. We applied PointNet to segment each piece of furniture from the point cloud of a real indoor environment captured by moving a RGB-D camera. For semantic segmentation, we focused on local geometric information not used in PointNet, and we proposed a method to refine the class probability of labels attached to each point in PointNet's output. The effectiveness of our method was experimentally confirmed. We then created 3D models of real-world furniture using a point cloud with corrected labels, and we virtually manipulated real living space using Dollhouse VR, a layout system.

KW - furniture arrangement

KW - semantic segmentation

KW - virtual reality

UR - http://www.scopus.com/inward/record.url?scp=85066317359&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85066317359&partnerID=8YFLogxK

U2 - 10.1109/APMAR.2019.8709156

DO - 10.1109/APMAR.2019.8709156

M3 - Conference contribution

AN - SCOPUS:85066317359

T3 - Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019

BT - Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019

A2 - Weng, Dongdong

A2 - Chan, Liwei

A2 - Lee, Youngho

A2 - Liang, Xiaohui

A2 - Sakata, Nobuchika

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 12th Asia Pacific Workshop on Mixed and Augmented Reality, APMAR 2019

Y2 - 28 March 2019 through 29 March 2019

ER -

Semantic Segmentation of 3D Point Cloud to Virtually Manipulate Real Living Space

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this