As the urban population continues to expand and is expected to comprise 80% of the total population in 2050, it is crucial to ensure the sustainability and energy efficiency of cities. Among all the homes globally, Hard-to-Decarbonize (HtD) buildings are estimated to be a quarter of them. Identifying the HtD houses and proposing strategies for these houses is important to reach the global net zero target. However, the study of HtD houses has historically been neglected. Previous studies mainly focus on simulating, predicting and understanding attributes that are directly related to energy usage and efficiency. In this research, a methodology for identifying HtD buildings with publicly available data is proposed and tested in Cambridge, UK. A dataset of HtD houses in Cambridge is organized, with criteria derived from the Energy Performance Certificate (EPC), which results from detailed inspections of houses. Street view images (SVI), ariel view images (AVI), land surface temperature (LST), and building stock data are used together for the prediction with deep learning. The classification accuracy for HtD buildings is able to achieve 82\%. This study also explores the minimal data needed for the high-accuracy prediction of HtD houses. Results show that SVI contributes the most to the prediction.