Appearance-based virtual view generation from multicamera videos captured in the 3-D room

Hideo Saito, Shigeyuki Baba, Takeo Kanade

Research output: Contribution to journalArticle

35 Citations (Scopus)

Abstract

We present an appearance-based virtual view generation method that allows viewers to fly through a real dynamic scene. The scene is captured by multiple synchronized cameras. Arbitrary views are generated by interpolating two original camera-views near the given viewpoint. The quality of the generated synthetic view is determined by the precision, consistency and density of correspondences between the two images. All or most of previous work that uses interpolation extracts the correspondences from these two images. However, not only is it difficult to do so reliably (the task requires a good stereo algorithm), but also the two images alone sometimes do not have enough information, due to problems such as occlusion. Instead, we take advantage of the fact that we have many views, from which we can extract much more reliable and comprehensive three-dimensional (3-D) geometry of the scene as a 3-D model. Dense and precise correspondences between the two images, to be used for interpolation, are obtained using this constructed 3-D model. Pseudo correspondences are even obtained for regions occluded in one of the cameras and then we used to correctly interpolate between the two images. Our method of 3-D modeling from multiple images uses the Multiple Baseline Stereo method and the Shape from Silhoutte method. The virtual view sequences are presented for demonstrating the performance of the virtual view generation in the 3-D Room.

Original languageEnglish
Pages (from-to)303-316
Number of pages14
JournalIEEE Transactions on Multimedia
Volume5
Issue number3
DOIs
Publication statusPublished - 2003 Sep

Fingerprint

Cameras
Interpolation
Geometry

Keywords

  • 3-D model
  • Image based rendering
  • Model based rendering
  • Multibaseline stereo
  • Multiple-view images
  • Shape from silhouette

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Computer Graphics and Computer-Aided Design
  • Software

Cite this

Appearance-based virtual view generation from multicamera videos captured in the 3-D room. / Saito, Hideo; Baba, Shigeyuki; Kanade, Takeo.

In: IEEE Transactions on Multimedia, Vol. 5, No. 3, 09.2003, p. 303-316.

Research output: Contribution to journalArticle

@article{9ee61d5f490e4203bd38d65d1cd67c9f,
title = "Appearance-based virtual view generation from multicamera videos captured in the 3-D room",
abstract = "We present an appearance-based virtual view generation method that allows viewers to fly through a real dynamic scene. The scene is captured by multiple synchronized cameras. Arbitrary views are generated by interpolating two original camera-views near the given viewpoint. The quality of the generated synthetic view is determined by the precision, consistency and density of correspondences between the two images. All or most of previous work that uses interpolation extracts the correspondences from these two images. However, not only is it difficult to do so reliably (the task requires a good stereo algorithm), but also the two images alone sometimes do not have enough information, due to problems such as occlusion. Instead, we take advantage of the fact that we have many views, from which we can extract much more reliable and comprehensive three-dimensional (3-D) geometry of the scene as a 3-D model. Dense and precise correspondences between the two images, to be used for interpolation, are obtained using this constructed 3-D model. Pseudo correspondences are even obtained for regions occluded in one of the cameras and then we used to correctly interpolate between the two images. Our method of 3-D modeling from multiple images uses the Multiple Baseline Stereo method and the Shape from Silhoutte method. The virtual view sequences are presented for demonstrating the performance of the virtual view generation in the 3-D Room.",
keywords = "3-D model, Image based rendering, Model based rendering, Multibaseline stereo, Multiple-view images, Shape from silhouette",
author = "Hideo Saito and Shigeyuki Baba and Takeo Kanade",
year = "2003",
month = "9",
doi = "10.1109/TMM.2003.813283",
language = "English",
volume = "5",
pages = "303--316",
journal = "IEEE Transactions on Multimedia",
issn = "1520-9210",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "3",

}

TY - JOUR

T1 - Appearance-based virtual view generation from multicamera videos captured in the 3-D room

AU - Saito, Hideo

AU - Baba, Shigeyuki

AU - Kanade, Takeo

PY - 2003/9

Y1 - 2003/9

N2 - We present an appearance-based virtual view generation method that allows viewers to fly through a real dynamic scene. The scene is captured by multiple synchronized cameras. Arbitrary views are generated by interpolating two original camera-views near the given viewpoint. The quality of the generated synthetic view is determined by the precision, consistency and density of correspondences between the two images. All or most of previous work that uses interpolation extracts the correspondences from these two images. However, not only is it difficult to do so reliably (the task requires a good stereo algorithm), but also the two images alone sometimes do not have enough information, due to problems such as occlusion. Instead, we take advantage of the fact that we have many views, from which we can extract much more reliable and comprehensive three-dimensional (3-D) geometry of the scene as a 3-D model. Dense and precise correspondences between the two images, to be used for interpolation, are obtained using this constructed 3-D model. Pseudo correspondences are even obtained for regions occluded in one of the cameras and then we used to correctly interpolate between the two images. Our method of 3-D modeling from multiple images uses the Multiple Baseline Stereo method and the Shape from Silhoutte method. The virtual view sequences are presented for demonstrating the performance of the virtual view generation in the 3-D Room.

AB - We present an appearance-based virtual view generation method that allows viewers to fly through a real dynamic scene. The scene is captured by multiple synchronized cameras. Arbitrary views are generated by interpolating two original camera-views near the given viewpoint. The quality of the generated synthetic view is determined by the precision, consistency and density of correspondences between the two images. All or most of previous work that uses interpolation extracts the correspondences from these two images. However, not only is it difficult to do so reliably (the task requires a good stereo algorithm), but also the two images alone sometimes do not have enough information, due to problems such as occlusion. Instead, we take advantage of the fact that we have many views, from which we can extract much more reliable and comprehensive three-dimensional (3-D) geometry of the scene as a 3-D model. Dense and precise correspondences between the two images, to be used for interpolation, are obtained using this constructed 3-D model. Pseudo correspondences are even obtained for regions occluded in one of the cameras and then we used to correctly interpolate between the two images. Our method of 3-D modeling from multiple images uses the Multiple Baseline Stereo method and the Shape from Silhoutte method. The virtual view sequences are presented for demonstrating the performance of the virtual view generation in the 3-D Room.

KW - 3-D model

KW - Image based rendering

KW - Model based rendering

KW - Multibaseline stereo

KW - Multiple-view images

KW - Shape from silhouette

UR - http://www.scopus.com/inward/record.url?scp=0042913178&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0042913178&partnerID=8YFLogxK

U2 - 10.1109/TMM.2003.813283

DO - 10.1109/TMM.2003.813283

M3 - Article

AN - SCOPUS:0042913178

VL - 5

SP - 303

EP - 316

JO - IEEE Transactions on Multimedia

JF - IEEE Transactions on Multimedia

SN - 1520-9210

IS - 3

ER -