We investigate the critical behaviors of correlation length and critical exponents for strongly interacting bosons in a two-dimensional optical lattice via quantum Monte Carlo simulations. By comparing the full numerical results to those given by the effective theory, we quantitatively determine the critical regime where the universal scaling behaviors applies at a finite temperature, for both classical Berezinskii-Kosterlitz-Thouless transition and quantum phase transition from superfluid to Mott insulator. Our results represent the critical regime that should be observed in present experimental conditions.