Thermography is designed to record the surface temperature information of the photographed objects. It is difficult to efficiently extract the useful information related the defects presented in the building outside walls because it is hard to analyze many thermographs at a time. Principal component analysis (PCA) has been widely used to analyze the hyperspectral satellite images by generating the new image that is composed of major characteristics extracted from hundreds of bands contained in the hyperspectral images. In this study, a scheme is proposed by applying PCA on the collected thermographs such that large data can be limited into the few enhanced thermographs, and then object-based image segmentation is introduced to analyze those enhanced thermographs such that the boundaries of the segmented regions can be described and lied on those enhanced thermographs. The image segmentation presented in the paper can efficiently group those pixels with collecting similar surface temperatures into the same regions such that each thermography can be composed by few groups. In each segmented group, the average surface temperature of each segmented region can be used to replace the surface temperature recorded in each pixel. Furthermore, the environmental effects acting on the given thermal image can be estimated by the proposed model. In doing so, those regions with the highest surface temperature information can be considered as defects located on the exterior building layer. Different nondestructive testing methods are, then, applied to those identified locations to verify the processed results. From the experimental results, the proposed approach does offer a reliable way to locate the defects presented on the building exterior layers because the results obtained by applying impact echo method are almost the same with the processed results of the proposed approach.