Improving quality of free-viewpoint image by mesh based 3D shape deformation

In this paper, we present a method to synthesize high-quality virtual viewpoint image targeting the detailed texture objects. About 30 images are taken from multiple uncalibrated cameras around the object, and the Visual Hull model is reconstructed with Shape from Silhouette method. To deform 3D surface model that is converted from Visual Hull Model using the information such as image texture and object silhouette, the difference between the real object and the reconstructed model is evaluated as a cost function of optimization problem. Our deforming model algorithm is based on single vertex iterative shifting. The vertex of surface triangle mesh is moved to the selected candidate point that maximizes the cost function. The cost function is consisted by four constraint criteria, texture correlation, smoothness, object silhouette, and mesh shape regularity. In addition to the cost function, such as judging mesh direction and combining / dividing meshes are applied for refined 3D models to avoid mesh folding and mesh size unevenness. The refined model provides a quite accurate dense corresponding relationship between the input images, so that high quality image can be synthesized at virtual viewpoint. We also demonstrate the proposed method by showing virtual viewpoint images to applying the real image that are taken from multiple uncalibrated cameras.