Recently, the surface‐enhanced Raman spectroscopy (SERS) technique emerges as a potential promising analytical tool for rapid, sensitive, and selective detection which can provide unique information about the substances in the presence of plasmonic nanostructures via finger‐printing SERS spectra. The progress in nanostructure SERS substrates and portable Raman spectrometers will promote this novel detection technique to play an important role in the future rapid on‐site assay. In this review, we summarized the latest research on SERS detection technique combined based on nanostructure SERS‐active substrates with portable Raman spectrometers and their potential on‐site detection applications in environment pollutants, agricultural pollutants, biomedical analysis, and other newly emerging fields (focusing particularly on the last 5 years of research). The future perspective of SERS for on‐site analysis is also discussed.