We consider the quasinormal modes, quasibound states and superradiant instability of a rotating hairy black hole, which possesses a Horndeski hair as deviation from Kerr black hole, under the perturbation of massive scalar field. With the use of the matrix method, we mainly calculate the eigenfrequencies related to those modes of the perturbation. Under the perturbation of the massless scalar field, the Horndeski hair and spin parameter have significant influences on the quasinormal frequency, but its imaginary part is always finite negative and no unstable mode is found. Under the perturbation of the massive scalar field, we focus on the eigenfrequencies of quasibound states and find the modes of which the imaginary part of eigenfrequencies is positive, indicating that the black hole undergoes superradiant instability. Then we scan the parameters and figure out a diagram in the space of Horndeski hair and spin parameters to distinguish the rotating hairy black hole with superradiant instability from the stable one.