Pancake lens has been widely used in virtual reality (VR) and mixed reality (MR) due to its compact form factor. However, using a half mirror (HM) to fold the optical path results in a tremendous optical loss. To enhance the optical efficiency while keeping a compact form factor, we present a new folded optical system incorporating a nonreciprocal polarization rotator. In our proof‐of‐concept experiment using a commercial Faraday rotator (FR), the theoretically predicted 100% efficiency is validated. Besides, the angular response of terbium gallium garnet (TGG) material is simulated for the first time and the results indicate that such an FR is relatively insensitive to the incident angle, which can dramatically enhance the contrast ratio of our pancake design. Finally, the potential application of such pancake optics for ultracompact near‐infrared (NIR) night vision goggles is also discussed.