Mobile Photometric Stereo with Keypoint-Based SLAM for Dense 3D Reconstruction

Remy Maxence, Hideaki Uchiyama, Hiroshi Kawasaki, Diego Thomas, Vincent Nozick, Hideo Saito

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The standard photometric stereo is a technique to densely reconstruct objects' surfaces using light variation under the assumption of a static camera with a moving light source. In this work, we use photometric stereo to reconstruct dense 3D scenes while moving the camera and the light altogether. In such non-static case, camera poses as well as correspondences between pixels of each frame to apply photometric stereo are required. ORB-SLAM is a technique that can be used to estimate camera poses. To retrieve correspondences, our idea is to start from a sparse 3D mesh obtained with ORB SLAM and then densify the mesh by a plane sweep method using a multi-view photometric consistency. By combining ORB-SLAM and photometric stereo, it is possible to reconstruct dense 3D scenes with a off-the-shelf smartphone and its embedded torchlight. Note that SLAM systems usually struggle with textureless object, which is effectively compensated by the photometric stereo in our method. Experiments are conducted to show that our proposed method gives better results than SLAM alone or COLMAP, especially for partially textureless surfaces.

Original languageEnglish
Title of host publicationProceedings - 2019 International Conference on 3D Vision, 3DV 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages574-582
Number of pages9
ISBN (Electronic)9781728131313
DOIs
Publication statusPublished - 2019 Sep
Event7th International Conference on 3D Vision, 3DV 2019 - Quebec, Canada
Duration: 2019 Sep 152019 Sep 18

Publication series

NameProceedings - 2019 International Conference on 3D Vision, 3DV 2019

Conference

Conference7th International Conference on 3D Vision, 3DV 2019
CountryCanada
CityQuebec
Period19/9/1519/9/18

Fingerprint

Photometric Stereo
Simultaneous Localization and Mapping
3D Reconstruction
Cameras
Camera
Correspondence
Plane Sweep
Mesh
Smartphones
Light sources
Pixels
Pixel
Experiments
Estimate
Experiment

Keywords

  • 3D reconstruction
  • dynamic
  • ORB SLAM
  • photometric
  • photometric stereo
  • plane sweep
  • SLAM
  • smartphone

ASJC Scopus subject areas

  • Computer Vision and Pattern Recognition
  • Media Technology
  • Modelling and Simulation

Cite this

Maxence, R., Uchiyama, H., Kawasaki, H., Thomas, D., Nozick, V., & Saito, H. (2019). Mobile Photometric Stereo with Keypoint-Based SLAM for Dense 3D Reconstruction. In Proceedings - 2019 International Conference on 3D Vision, 3DV 2019 (pp. 574-582). [8885679] (Proceedings - 2019 International Conference on 3D Vision, 3DV 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/3DV.2019.00069

Mobile Photometric Stereo with Keypoint-Based SLAM for Dense 3D Reconstruction. / Maxence, Remy; Uchiyama, Hideaki; Kawasaki, Hiroshi; Thomas, Diego; Nozick, Vincent; Saito, Hideo.

Proceedings - 2019 International Conference on 3D Vision, 3DV 2019. Institute of Electrical and Electronics Engineers Inc., 2019. p. 574-582 8885679 (Proceedings - 2019 International Conference on 3D Vision, 3DV 2019).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Maxence, R, Uchiyama, H, Kawasaki, H, Thomas, D, Nozick, V & Saito, H 2019, Mobile Photometric Stereo with Keypoint-Based SLAM for Dense 3D Reconstruction. in Proceedings - 2019 International Conference on 3D Vision, 3DV 2019., 8885679, Proceedings - 2019 International Conference on 3D Vision, 3DV 2019, Institute of Electrical and Electronics Engineers Inc., pp. 574-582, 7th International Conference on 3D Vision, 3DV 2019, Quebec, Canada, 19/9/15. https://doi.org/10.1109/3DV.2019.00069
Maxence R, Uchiyama H, Kawasaki H, Thomas D, Nozick V, Saito H. Mobile Photometric Stereo with Keypoint-Based SLAM for Dense 3D Reconstruction. In Proceedings - 2019 International Conference on 3D Vision, 3DV 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 574-582. 8885679. (Proceedings - 2019 International Conference on 3D Vision, 3DV 2019). https://doi.org/10.1109/3DV.2019.00069
Maxence, Remy ; Uchiyama, Hideaki ; Kawasaki, Hiroshi ; Thomas, Diego ; Nozick, Vincent ; Saito, Hideo. / Mobile Photometric Stereo with Keypoint-Based SLAM for Dense 3D Reconstruction. Proceedings - 2019 International Conference on 3D Vision, 3DV 2019. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 574-582 (Proceedings - 2019 International Conference on 3D Vision, 3DV 2019).
@inproceedings{2bab87137ce04b8dbb92a8ab734c0a93,
title = "Mobile Photometric Stereo with Keypoint-Based SLAM for Dense 3D Reconstruction",
abstract = "The standard photometric stereo is a technique to densely reconstruct objects' surfaces using light variation under the assumption of a static camera with a moving light source. In this work, we use photometric stereo to reconstruct dense 3D scenes while moving the camera and the light altogether. In such non-static case, camera poses as well as correspondences between pixels of each frame to apply photometric stereo are required. ORB-SLAM is a technique that can be used to estimate camera poses. To retrieve correspondences, our idea is to start from a sparse 3D mesh obtained with ORB SLAM and then densify the mesh by a plane sweep method using a multi-view photometric consistency. By combining ORB-SLAM and photometric stereo, it is possible to reconstruct dense 3D scenes with a off-the-shelf smartphone and its embedded torchlight. Note that SLAM systems usually struggle with textureless object, which is effectively compensated by the photometric stereo in our method. Experiments are conducted to show that our proposed method gives better results than SLAM alone or COLMAP, especially for partially textureless surfaces.",
keywords = "3D reconstruction, dynamic, ORB SLAM, photometric, photometric stereo, plane sweep, SLAM, smartphone",
author = "Remy Maxence and Hideaki Uchiyama and Hiroshi Kawasaki and Diego Thomas and Vincent Nozick and Hideo Saito",
year = "2019",
month = "9",
doi = "10.1109/3DV.2019.00069",
language = "English",
series = "Proceedings - 2019 International Conference on 3D Vision, 3DV 2019",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "574--582",
booktitle = "Proceedings - 2019 International Conference on 3D Vision, 3DV 2019",

}

TY - GEN

T1 - Mobile Photometric Stereo with Keypoint-Based SLAM for Dense 3D Reconstruction

AU - Maxence, Remy

AU - Uchiyama, Hideaki

AU - Kawasaki, Hiroshi

AU - Thomas, Diego

AU - Nozick, Vincent

AU - Saito, Hideo

PY - 2019/9

Y1 - 2019/9

N2 - The standard photometric stereo is a technique to densely reconstruct objects' surfaces using light variation under the assumption of a static camera with a moving light source. In this work, we use photometric stereo to reconstruct dense 3D scenes while moving the camera and the light altogether. In such non-static case, camera poses as well as correspondences between pixels of each frame to apply photometric stereo are required. ORB-SLAM is a technique that can be used to estimate camera poses. To retrieve correspondences, our idea is to start from a sparse 3D mesh obtained with ORB SLAM and then densify the mesh by a plane sweep method using a multi-view photometric consistency. By combining ORB-SLAM and photometric stereo, it is possible to reconstruct dense 3D scenes with a off-the-shelf smartphone and its embedded torchlight. Note that SLAM systems usually struggle with textureless object, which is effectively compensated by the photometric stereo in our method. Experiments are conducted to show that our proposed method gives better results than SLAM alone or COLMAP, especially for partially textureless surfaces.

AB - The standard photometric stereo is a technique to densely reconstruct objects' surfaces using light variation under the assumption of a static camera with a moving light source. In this work, we use photometric stereo to reconstruct dense 3D scenes while moving the camera and the light altogether. In such non-static case, camera poses as well as correspondences between pixels of each frame to apply photometric stereo are required. ORB-SLAM is a technique that can be used to estimate camera poses. To retrieve correspondences, our idea is to start from a sparse 3D mesh obtained with ORB SLAM and then densify the mesh by a plane sweep method using a multi-view photometric consistency. By combining ORB-SLAM and photometric stereo, it is possible to reconstruct dense 3D scenes with a off-the-shelf smartphone and its embedded torchlight. Note that SLAM systems usually struggle with textureless object, which is effectively compensated by the photometric stereo in our method. Experiments are conducted to show that our proposed method gives better results than SLAM alone or COLMAP, especially for partially textureless surfaces.

KW - 3D reconstruction

KW - dynamic

KW - ORB SLAM

KW - photometric

KW - photometric stereo

KW - plane sweep

KW - SLAM

KW - smartphone

UR - http://www.scopus.com/inward/record.url?scp=85074981410&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85074981410&partnerID=8YFLogxK

U2 - 10.1109/3DV.2019.00069

DO - 10.1109/3DV.2019.00069

M3 - Conference contribution

AN - SCOPUS:85074981410

T3 - Proceedings - 2019 International Conference on 3D Vision, 3DV 2019

SP - 574

EP - 582

BT - Proceedings - 2019 International Conference on 3D Vision, 3DV 2019

PB - Institute of Electrical and Electronics Engineers Inc.

ER -