Axion dark matter differentiates the phase velocities of the circular-polarized photons. In this Letter, a scheme to measure the phase difference by using a linear optical cavity is proposed. If the scheme is applied to the Fabry-Pérot arm of Advanced LIGO-like (Cosmic-Explorer-like) gravitational wave detector, the potential sensitivity to the axion-photon coupling constant, gaγ, reaches gaγ 8 × 10 −13 GeV −1 (4 × 10 −14 GeV −1 ) at the axion mass m 3 × 10 −13 eV (2 × 10 −15 eV) and remains at around this sensitivity for 3 orders of magnitude in mass. Furthermore, its sensitivity has a sharp peak reaching gaγ 10 −14 GeV −1 (8×10 −17 GeV −1 ) at m = 1.563×10 −10 eV (1.563 × 10 −11 eV). This sensitivity can be achieved without loosing any sensitivity to gravitational waves.