For heritage buildings, energy-efficient retrofitting cannot be applied with the same range of possibilities as with existing buildings. Applying such improvements to heritage buildings can be challenging due to their historic and/or cultural significance and non-standard construction methods. This paper reviews the technical challenges and potential of applying energy efficient retrofit elements in heritage buildings. The retrofitting measures reviewed are draught-proofing, windows, insulation, ventilation, heating, solar photovoltaics and phase change materials. It is possible to significantly reduce energy use in heritage buildings with such retrofits. However, there is no universal way to apply energy-efficient retrofitting in heritage buildings, which is apparent in the literature, where case studies are prevalent.