We propose the ZQ Berry phase as a topological invariant for higher-order symmetry-protected topological (HOSPT) phases. It is topologically stable for electron-electron interactions assuming the gap remains open. As a concrete example, we show that the Berry phase is quantized in Z4 and characterizes the HOSPT phase of the extended Benalcazar-Bernevig-Hughes (BBH) model, which contains the next-nearest neighbor hopping and the intersite Coulomb interactions. Furthermore, we introduce the Z4 Berry phase for the spin-model-analog of the BBH model, whose topological invariant has not been found so far. We also confirm the bulk-corner correspondence between the Z4 Berry phase and the corner states in the HOSPT phases.