Nonlinear energy sinks (NES) are efficient vibration control devices, which have been studied and applied in mechanical, automobile, and aerospace engineering.However, there are few applications in civil engineering. A new type of NES, which is termed as track NES, is proposed in this paper. The optimal mass ratio and track shape expression of NES were determined based on a preliminary optimization design process. To verify its vibration control effects on building structures, a series of shake table tests were conducted on a five-story steel frame. Tracks of the NES were installed at the roof of the frame with rigid connections and the mass of the NES was constrained to slide along the track by using wheels. Five earthquake waves with different frequency spectrums were selected to excite the frame coupled with NES under minor, moderate, and major levels. Accelerations and displacements on each story of the frame were measured, recorded, and evaluated. The experimental results demonstrate that with small mass ratio (2%) of main structure, NES has good performance in reducing the dynamic responses of the frame under seismic excitations. The reduction ratio for peak response is up to 50%, while for root mean square response is up to 80%. NES also exhibits wide-band frequency vibration controlling attributes, and the responses of the frame are reduced in multiple vibration modes. In addition, the vibration reduction capability of the NES with steel wheels and that with rubber wheels are compared, and it is verified that different damping of NES makes a difference to the vibration control effects. The displacement reduction performance is not sensitive to the damping factor of the NES, but acceleration response is highly affected by the damping feature of the NES.