Arbitrary viewpoint image synthesis based on light field construction from uncalibrated image sequence

Kenkichi Kobayashi, Hideo Saito

Research output: Contribution to journalArticle

Abstract

Recently, image-based rendering in which the actual image is efficiently utilized to synthesize a new image has been attracting attention. There have been many studies of arbitrary viewpoint image synthesis based on the light field, which is an image-based rendering procedure. It is now possible to synthesize images with a high degree of realism. However, in the method based on the theory of the light field, the camera parameters used in the imaging must be known (with calibration). In cases such as imaging by a free hand-held camera, the theory of the light field cannot be applied to the sequence of uncalibrated images. Consequently, this paper proposes an image generation method using the theory of the light field, which can be applied to a sequence of uncalibrated images. In the proposed method, the ray in each frame can be handled appropriately, using only the correspondence between frames, which is derived by tracking the feature points in the uncalibrated image sequence. This is possible because a special space, the projective grid space, which is determined by the two cameras, rather than Euclidean space, that is, ordinary three-dimensional space, is used for light field construction in the proposed method. In the projective grid space, the general geometrical relations among straight lines and planes in Euclidean space do not apply, but the position of the ray in space can be calculated by using the fundamental matrix. The proposed method is applied to uncalibrated image sequences taken by free hand-held video cameras. In experiments, the image from an arbitrary viewpoint is generated in a satisfactory way, indicating the usefulness of the proposed method.

Original languageEnglish
Pages (from-to)11-21
Number of pages11
JournalElectronics and Communications in Japan, Part III: Fundamental Electronic Science (English translation of Denshi Tsushin Gakkai Ronbunshi)
Volume87
Issue number9
DOIs
Publication statusPublished - 2004 Sep

Fingerprint

Cameras
Imaging techniques
Video cameras
Calibration
Experiments

Keywords

  • Arbitrary viewpoint image synthesis
  • Epipolar geometry
  • Image-based rendering
  • Light field
  • Self-calibration

ASJC Scopus subject areas

  • Electrical and Electronic Engineering

Cite this

@article{26ed27a1c65a49c9a14be8273b409c31,
title = "Arbitrary viewpoint image synthesis based on light field construction from uncalibrated image sequence",
abstract = "Recently, image-based rendering in which the actual image is efficiently utilized to synthesize a new image has been attracting attention. There have been many studies of arbitrary viewpoint image synthesis based on the light field, which is an image-based rendering procedure. It is now possible to synthesize images with a high degree of realism. However, in the method based on the theory of the light field, the camera parameters used in the imaging must be known (with calibration). In cases such as imaging by a free hand-held camera, the theory of the light field cannot be applied to the sequence of uncalibrated images. Consequently, this paper proposes an image generation method using the theory of the light field, which can be applied to a sequence of uncalibrated images. In the proposed method, the ray in each frame can be handled appropriately, using only the correspondence between frames, which is derived by tracking the feature points in the uncalibrated image sequence. This is possible because a special space, the projective grid space, which is determined by the two cameras, rather than Euclidean space, that is, ordinary three-dimensional space, is used for light field construction in the proposed method. In the projective grid space, the general geometrical relations among straight lines and planes in Euclidean space do not apply, but the position of the ray in space can be calculated by using the fundamental matrix. The proposed method is applied to uncalibrated image sequences taken by free hand-held video cameras. In experiments, the image from an arbitrary viewpoint is generated in a satisfactory way, indicating the usefulness of the proposed method.",
keywords = "Arbitrary viewpoint image synthesis, Epipolar geometry, Image-based rendering, Light field, Self-calibration",
author = "Kenkichi Kobayashi and Hideo Saito",
year = "2004",
month = "9",
doi = "10.1002/ecjc.20102",
language = "English",
volume = "87",
pages = "11--21",
journal = "Electronics and Communications in Japan, Part III: Fundamental Electronic Science (English translation of Denshi Tsushin Gakkai Ronbunshi)",
issn = "1042-0967",
publisher = "John Wiley and Sons Inc.",
number = "9",

}

TY - JOUR

T1 - Arbitrary viewpoint image synthesis based on light field construction from uncalibrated image sequence

AU - Kobayashi, Kenkichi

AU - Saito, Hideo

PY - 2004/9

Y1 - 2004/9

N2 - Recently, image-based rendering in which the actual image is efficiently utilized to synthesize a new image has been attracting attention. There have been many studies of arbitrary viewpoint image synthesis based on the light field, which is an image-based rendering procedure. It is now possible to synthesize images with a high degree of realism. However, in the method based on the theory of the light field, the camera parameters used in the imaging must be known (with calibration). In cases such as imaging by a free hand-held camera, the theory of the light field cannot be applied to the sequence of uncalibrated images. Consequently, this paper proposes an image generation method using the theory of the light field, which can be applied to a sequence of uncalibrated images. In the proposed method, the ray in each frame can be handled appropriately, using only the correspondence between frames, which is derived by tracking the feature points in the uncalibrated image sequence. This is possible because a special space, the projective grid space, which is determined by the two cameras, rather than Euclidean space, that is, ordinary three-dimensional space, is used for light field construction in the proposed method. In the projective grid space, the general geometrical relations among straight lines and planes in Euclidean space do not apply, but the position of the ray in space can be calculated by using the fundamental matrix. The proposed method is applied to uncalibrated image sequences taken by free hand-held video cameras. In experiments, the image from an arbitrary viewpoint is generated in a satisfactory way, indicating the usefulness of the proposed method.

AB - Recently, image-based rendering in which the actual image is efficiently utilized to synthesize a new image has been attracting attention. There have been many studies of arbitrary viewpoint image synthesis based on the light field, which is an image-based rendering procedure. It is now possible to synthesize images with a high degree of realism. However, in the method based on the theory of the light field, the camera parameters used in the imaging must be known (with calibration). In cases such as imaging by a free hand-held camera, the theory of the light field cannot be applied to the sequence of uncalibrated images. Consequently, this paper proposes an image generation method using the theory of the light field, which can be applied to a sequence of uncalibrated images. In the proposed method, the ray in each frame can be handled appropriately, using only the correspondence between frames, which is derived by tracking the feature points in the uncalibrated image sequence. This is possible because a special space, the projective grid space, which is determined by the two cameras, rather than Euclidean space, that is, ordinary three-dimensional space, is used for light field construction in the proposed method. In the projective grid space, the general geometrical relations among straight lines and planes in Euclidean space do not apply, but the position of the ray in space can be calculated by using the fundamental matrix. The proposed method is applied to uncalibrated image sequences taken by free hand-held video cameras. In experiments, the image from an arbitrary viewpoint is generated in a satisfactory way, indicating the usefulness of the proposed method.

KW - Arbitrary viewpoint image synthesis

KW - Epipolar geometry

KW - Image-based rendering

KW - Light field

KW - Self-calibration

UR - http://www.scopus.com/inward/record.url?scp=2542438904&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=2542438904&partnerID=8YFLogxK

U2 - 10.1002/ecjc.20102

DO - 10.1002/ecjc.20102

M3 - Article

AN - SCOPUS:2542438904

VL - 87

SP - 11

EP - 21

JO - Electronics and Communications in Japan, Part III: Fundamental Electronic Science (English translation of Denshi Tsushin Gakkai Ronbunshi)

JF - Electronics and Communications in Japan, Part III: Fundamental Electronic Science (English translation of Denshi Tsushin Gakkai Ronbunshi)

SN - 1042-0967

IS - 9

ER -