Phase change materials like vanadium dioxide (VO 2 ) have recently gained particular attention to realize effective optical modulators. In this paper, we propose a high-performance optical modulator based on VO 2 . The structure is constructed by a stack of two graphene sheets spaced by a VO 2 layer that are all grown over a silicon waveguide. The input light propagating along the modulating region of the waveguide can interact with the VO 2 at the top and side of the structure. When an external voltage is applied across the graphene sheets, the VO 2 changes to the metallic phase and the modulator is set at the off state. Most of the previously reported VO 2 -based modulators are polarization sensitive and suffer from high polarization diversity. When such modulators are integrated into a fiber-based optical system containing light with random polarization, the modulation efficiency is profoundly degraded. In our proposed modulator, the structure is optimized to achieve a polarization-insensitive response with an off-state propagation loss of ∼14 dB/μm. The TE (TM) insertion loss is also calculated to be as low as 2.25 dB/μm (3.22 dB/μm), so that an extinction ratio of as high as ∼11.5 dB/μm is obtained. In addition, the required low modulating voltage of 1.12 V and reasonable modulation speed of 390 MHz together with the low energy consumption of 2.1 fJ/bit could make our proposed modulator a good candidate for photonic integrated systems.