Magnetic disturbance field from ferromagnetic structural parts is a dominant factor that influences the accuracy of a geomagnetic vector measuring instrument. In this paper, a new vector compensation method for a three-axis magnetometer is proposed. In the first step, combined with posture information from inertial sensors, the dataset of the three-axis magnetometer outputs in different postures is utilized to construct linear equations of the error parameters; then the soft-iron parameters are determined with singular value decomposition. In the second step, the hard-iron parameters are estimated by changing the fixing direction of the three-axis magnetometer. Simulations and experiments are performed to assess the performance of the proposed method. The results show that the error parameters can be accurately estimated, and the measurement errors of geomagnetic field vectors and magnitude are suppressed greatly. After compensation, the standard deviations of the errors of the magnetic vector components decrease from hundreds of nT to tens of nT. The main advantages of this proposed method are as follows: (i) it can compensate not only the scalar error but also the vector error, and (ii) it compensates the errors of vectors and magnitude with higher accuracy.