HySAR: Hybrid material rendering by an optical see-through head-mounted display with spatial augmented reality projection

Spatial augmented reality (SAR) pursues realism in rendering materials and objects. To advance this goal, we propose a hybrid SAR (HySAR) that combines a projector with optical see-through head-mounted displays (OST-HMD). In an ordinary SAR scenario with co-located viewers, the viewers perceive the same virtual material on physical surfaces. In general, the material consists of two components: a view-independent (VI) component such as diffuse reflection, and a view-dependent (VD) component such as specular reflection. The VI component is static over viewpoints, whereas the VD should change for each viewpoint even if a projector can simulate only one viewpoint at one time. In HySAR, a projector only renders the static VI components. In addition, the OST-HMD renders the dynamic VD components according to the viewer's current viewpoint. Unlike conventional SAR, the HySAR concept theoretically allows an unlimited number of co-located viewers to see the correct material over different viewpoints. Furthermore, the combination enhances the total dynamic range, the maximum intensity, and the resolution of perceived materials. With proof-of-concept systems, we demonstrate HySAR both qualitatively and quantitatively with real objects. First, we demonstrate HySAR by rendering synthetic material properties on a real object from different viewpoints. Our quantitative evaluation shows that our system increases the dynamic range by 2.24 times and the maximum intensity by 2.12 times compared to an ordinary SAR system. Second, we replicate the material properties of a real object by SAR and HySAR, and show that HySAR outperforms SAR in rendering VD specular components.