Characterizing the nonlinear optical properties of numerous materials plays a prerequisite role in nonlinear imaging and quantum sensing. Here, we present the evaluation of the nonlinear optical properties of Rb vapor by the Gaussian-Bessel beam assisted z-scan method. Owed to the concentrated energy in the central waist spot and the constant intensity of the beam distribution, the Gaussian-Bessel beam enables enhanced sensitivity for nonlinear refractive index measurement. The nonlinear self-focusing and self-defocusing effects of the Rb vapor are illustrated in the case of blue and red frequency detunings from 5S1/2 − 5P3/2 transition, respectively. The complete images of the evolution of nonlinear optical properties with laser power and frequency detuning are acquired. Furthermore, the nonlinear refractive index n2 with a large scale of 10−6 cm2/W is determined from the measured transmittance peak-to-valley difference of z-scan curves, which is enhanced by a factor of ∼ 1.73 compared to the result of a equivalent Gaussian beam. Our research provides an effective method for measuring nonlinear refractive index, which will considerably enrich the application range of nonlinear material.