We discuss the symmetry factors of Feynman diagrams of scalar field theories with polynomial potential. After giving a concise general formula for them, we present an elementary and direct proof that when computing scattering amplitudes using the homological perturbation lemma, each contributing Feynman diagram is indeed included with the correct symmetry factor.