3D semantic segmentation for high-resolution aerial survey derived point clouds using deep learning (demonstration)

Three-dimensional (3D) Semantic segmentation of aerial derived point cloud aims at assigning each point to a semantic class such as building, tree, road, and so on. Accurate 3D-segmentation results can be used as an essential information for constructing 3D city models, for assessing the urban expansion and economical condition. However, the fine-grained semantic segmentation is a challenge in high-resolution point cloud due to irregularly distributed points unlike regular pixels of image. In this demonstration, we present a case study to apply PointNet, a novel deep learning network, to outdoor aerial survey derived point clouds by considering intensity (depth) as well as spectral information (RGB). PointNet was basically designed for indoor point cloud data based on the permutation invariance of 3D points. We firstly fuse two surveying datasets of Light Detection and ranging (LiDAR) and aerial images for generating multi-sourced aerial point clouds (RGB-DI). Then, each point of fused data is classified into a semantic class of ordinary building, public facility, apartment, factory, transportation network, park, and water by reworking PointNet. The result of our approach by using deep learning shows about 0.88 accuracy and 0.64 F-measure of semantic segmentation with the RGB-DI data we have fused. It outperforms a Support Vector Machine(SVM) approach based on geometric features of linearity, planarity, scattering, and verticality of a set of 3D points.