Many Structural Health Monitoring (SHM) methods have been proposed for the purposes of reducing maintenance costs and/or assuring performance of civil structures. The objective of this research is to propose a damage detection system that can obtain the detailed damage information by use of the minimum number of sensors. The proposed system minimizes the possibility of incorrect judgments. Modal frequencies of a structure are used for pattern recognition in the proposed method. Changes in multiple natural frequencies can be correlated to the spatial information of the location of damaged stories. Typically only two vibration sensors, one on the roof and the other on the ground, detecting a single input and a single output for the structure are needed to determine modal frequencies. Out of many pattern recognition tools, we propose to use the Support Vector Machine (SVM). This technique has been found effective. Our previous studies demonstrated that the proposed damage detection method worked well for simple models such as shear structures and bending structures. However, real buildings have various boundary conditions at their supports. In this study, the SVM technique was applied to damage detection of structures with various boundary conditions. The feature vectors for SVMs are generated based on the model of a structure. Then locations of structural damage are detected by inputting the measured structural vibration data into the SVMs. From simulation, it was found that the influence of the change in boundary conditions on the lower modes is larger. We performed experimental studies on damage detection of power distribution poles that had overhead wires. We proposed a method for determining the boundary conditions of the poles and verified this method based on measured vibration data. We demonstrated the effectiveness of the proposed method in detecting damage in the poles.