In this study, we propose a method of noncontact elastography, which allows us to investigate stiffness of soft structures by combining optical and acoustic modalities. We use optical coherence tomography (OCT) as a means of detecting internal deformation of a sample appearing in response to a mechanical force applied by acoustic radiation pressure. Unlike most of other stiffness sensing, this method can be performed without any contacts between the sample and actuator that generates pressure. To demonstrate the method, we measure the vibration velocity of a uniform phantom made of polyurethane, and characterize the mechanical parameters. We then confirm that the measured and calculated attenuation of the vibration over the depth agree well, which is inaccessible with a conventional laser Doppler vibrometer. This result paves a way to characterize more complex internal structures of soft materials.