We study N -component interacting particles (hardcore bosons and fermions) loaded in topological lattice models with SU(N )-invariant interactions based on exact diagonalization and density matrix renormalization group method. By tuning the interplay of interspecies and intraspecies interactions, we demonstrate that a class of SU(N ) fractional quantum Hall states can emerge at fractional filling factors ν = N/(N + 1) for bosons (ν = N/(2N + 1) for fermions) in the lowest Chern band, characterized by the nontrivial fractional Hall responses and the fractional charge pumping. Moreover, we establish a topological characterization based on the K matrix, and discuss the close relationship to the fractional quantum Hall physics in topological flat bands with Chern number N .