Exploring the quantum effects in plasmonic responses has attracted growing interest during the past decade. Extreme downsizing of nanoparticles and interparticle gaps is promising for altering the nature of plasmonic resonances through the spillover and tunneling of electronic wave functions at the metallic nanocomposites. While such quantum effects have been studied in the frequency domain in the past, their manifestation in the ultrafast time domain remains to be elucidated. In this work, we demonstrate that the quantum effect significantly alters the ultrafast plasmonic response by studying a novel atomically thin multilayer structure of Cs/graphene/Cs/Ir(111). The atomic scale confinement leads to a reduction in the damping rate of the surface plasmon−polariton that gives a strong transient optical response within several tens of femtoseconds. In addition, the localized excitation nonthermally launches coherent surface phonons of the composite, exhibiting the plasmon band modulation in the THz frequency range. These novel findings will pave the way for tailoring plasmon-induced processes by the atomic-scale fabrication.