We calculate elliptic and triangular flow parameters of thermal photons using an event-by-event hydrodynamic model with fluctuating initial conditions at 200A GeV Au+Au collisions at RHIC and at 2.76A TeV Pb+Pb collisions at the LHC for three different centrality bins. The photon elliptic flow shows strong centrality dependence where v2(pT ) increases towards peripheral collisions both at RHIC and at the LHC energies. However, the triangular flow parameter does not show significant dependence to the collision centrality. The elliptic as well as the triangular flow parameters found to underestimate the PHENIX data at RHIC by a large margin for all three centrality bins. We calculate pT spectrum and anisotropic flow of thermal photons from 200A GeV Cu+Cu collisions at RHIC for 0-20% centrality bin and compare with the results with those from Au+Au collisions. The production of thermal photon is found to decrease significantly for Cu+Cu collisions compared to Au+Au collisions. However, the effect of initial state fluctuation is found to be more pronounced for anisotropic flow resulting in larger v2 and v3 for Cu+Cu collisions. We study the correlation between the anisotropic flow parameters and the corresponding initial spatial anisotropies from their event by event distributions at RHIC and at the LHC energies. The linear correlation between v2 and ǫ2 is found be stronger compared to the correlation between v3 and ǫ3. In addition, the correlation coefficient is found to be larger at LHC than at RHIC.