Simulation of a large number of deformable bodies is often difficult because complex high-level modeling is required to address both multi-body contact and viscoelastic deformation. This necessitates the combined use of a discrete element method (DEM) and a finite element method (FEM). In this study, a quadruple discrete element method (QDEM) was developed for dynamic analysis of viscoelastic materials using a simpler algorithm compared to the standard FEM. QDEM easily incorporates the contact algorithm used in DEM. As the first step toward multi-body simulation, the fundamental performance of QDEM was investigated for viscoelastic analysis. The amplitude and frequency of cantilever elastic vibration were nearly equal to those obtained by the standard FEM. A comparison of creep recovery tests with an analytical solution showed good agreement between them. In addition, good correlation between the attenuation degree and the real physical viscosity was confirmed for viscoelastic vibration analysis. Therefore, the high accuracy of QDEM in the fundamental analysis of infinitesimal viscoelastic deformations was verified. Finally, the impact response of a ballast and sleeper under cyclic loading on a railway track was analyzed using QDEM as an application of deformable multi-body dynamics. The results showed that the vibration of the ballasted track was qualitatively in good agreement with the actual measurements. Moreover, the ballast layer with high friction reduced the ballasted track deterioration. This study suggests that QDEM, as an alternative to DEM and FEM, can provide deeper insights into the contact dynamics of a large number of deformable bodies.