Ultrafast infrared spectroscopy was used to unveil the rates at which different vibrational modes of atomic-scale defects in diamond interact with their environment. We adopted the N3VH0 defect as a model system, and studied perturbed free induction decay (PFID) and population decay features. The faster dephasing times of the fundamental stretch mode in comparison to the observed bend mode (first overtone) was a result of its more direct perturbation of the crystal lattice. We further demonstrate the versatility of ultrafast spectroscopy by investigating the energy level structure and dynamics of an unknown defect that is infrared active at 3237 cm−1.