To improve the accuracy of infrared radiation characteristics measurement in the aviation field, an infrared Fourier transform imaging spectrometer based on a double-swing solid angle reflector was designed. This imaging spectrometer operates in the 3–5 μm wavelength range and has a field of view of 1.7° × 1.7°. This article presents a comprehensive analysis of the system’s stray light and also studies the impact of external stray light on the imaging quality, along with the influence of internal stray light on the interference effects and the spectral resolution. It also present the design of a hood that suppresses the point source transmittance of the external stray light down to the order of 10−4. Based on this, we propose a method that incorporates the introduction of wedge and inclination angles. Additionally, a numerical range is provided for the addition of these angles on the beam splitter mirror and compensation plate. This ensures the effective suppression of any internal stray light. This study fills the gap in the knowledge about Fourier transform imaging spectrometers operating in the mid-infrared band for aviation applications, and proposes a suppression method suitable for interference systems, which is also suitable for Fourier transform imaging spectrometers based on other types of interferometers. This study broadens the application field of Fourier transform imaging spectrometers in stray light, and has great significance to promote the development of Fourier transform imaging spectrometer.