The third-order nonlinear susceptibility χ³ occurs universally in materials and can provide label-free fingerprint of materials’ electronic, vibrational, and structural information. One quantitative spectroscopic method to access low-energy resonances of χ³ is the terahertz electric field induced second harmonic generation (TEFISH), particularly suitable for centrosymmetric materials without second-order processes. However, quantifying TEFISH requires light sources with high spectral intensity, which is challenging for the “new terahertz gap” frequencies between 5-15 THz that, not by coincidence, is the fingerprint phonon bands for many solid-state materials. Here we report the realization of phase-sensitive heterodyne TEFISH microscopy offering simultaneous temporal, spectral, and spatial resolution in the frequency range of 4 to 18 THz for the first time. Using chirped pulse difference frequency generation, ee generated intense and frequency-tunable narrowband terahertz fields, and then demonstrated time-resolved hyperspectral TEFISH microscopy in polymer thin films (SU-8), 2D crystalline semiconductors (MoS₂), and a sub-wavelength terahertz resonator. By interfering the nonlinear emission with a local oscillator field, we retrieved both the amplitude and sign of TEFISH, quantitatively resolved the nonlinear spectra, and observed the resonance features of the samples with high sensitivity. The spatial resolution is much better than the diffraction limit of the terahertz excitation due to the short wavelength of the visible TEFISH signal. Our compact implementation of TEFISH allows ultrafast time-domain imaging of vibrational and polaritonic resonances in micro- and nanoscale quantum materials.