In this work we use Floquet theory to theoretically study the influence of monochromatic circularly and linearly polarized light on the Hofstadter butterfly-induced by a uniform perpendicular magnetic field-for both the kagome and triangular lattices. In the absence of the laser light, the butterfly has fractal structure with inversion symmetry about magnetic flux φ = 1/4, and reflection symmetry about φ = 1/2. As the system is exposed to an external laser, we find circularly polarized light deforms the butterfly by breaking the mirror symmetry at flux φ = 1/2. By contrast, linearly polarized light deforms the original butterfly while preserving the mirror symmetry at flux φ = 1/2. We find the inversion symmetry is always preserved for both linear and circular polarized light. For linearly polarized light, the Hofstadter butterfly depends on the polarization direction. Further, we study the effect of the laser on the Chern number of lowest band in the off-resonance regime (laser frequency is larger than the bandwidth). For circularly polarized light, we find that low laser intensity will not change the Chern number, but beyond a critical intensity the Chern number will change. For linearly polarized light, the Chern number depends on the polarization direction. Our work highlights the generic features expected for the periodically driven Hofstadter problem on different lattices.