This paper describes a new accurate interpolation method of a low-resolution depth image using a high-resolution color image for the structural analysis of 3d data. Our method adopts the strategy to detect and omit regions where it is difficult to estimate depth value accurately. In order to achieve this, Color geodesic neighborhood is used as the local smooth surface, and depth data is interpolated with checking smoothness of connections between them. Pixels where they can not be smoothly connected are omitted automatically as indeterminable pixels in this process. Experiments with images of the Middlebury stereo datasets demonstrate that in the quantitative sense our method can interpolate depth images more accurately than other interpolation methods.