We study a system consisting of a particle adsorbed on a carbon nanotube resonator. The particle is allowed to diffuse along the resonator, in order to enable study of, e.g., room-temperature mass sensing devices. The system is initialized in a state where only the fundamental vibration mode is excited, and the ring-down of the system is studied by numerically and analytically solving the stochastic equations of motion. We find two mechanisms of dissipation, induced by the diffusing adsorbate. First, short-time correlations between particle and resonator motions means that the net effect of the former on the latter does not average out, but instead causes nonexponential dissipation of vibrational energy. For vibrational amplitudes that are much larger than the thermal energy this dissipation is linear; for small amplitudes the decay takes the same form as that of a nonlinearly damped oscillator. Second, the particle diffusion mediates a coupling between vibration modes that opens a new dissipation channel by enabling energy transfer from the fundamental mode to the excited modes, which rapidly reach thermal equilibrium.