Underground, indoor, and harsh environments, which are complex, mutable, and hard to model, prevent high accuracy positioning and navigation. In this paper, a novel positioning system suitable to such environments is proposed. This system is based on a magnetic feature vector integrated with a miniature inertial measurement unit (MIMU). Compared with traditional magnetic-based positioning systems (MBPS), the proposed system exhibits better robustness, and applicability, which is proved by experimental results. Its non-accumulated error characteristics and outstanding penetrability enable the system to provide long-time accurate position and attitude service in underground, indoor, and harsh environments. The experiments indicate that the expected precision of the system can reach 0.055 and 0.062 m and an angle error of 2.1° and 2.8° in line of sight (LOS) and non-line of sight (NLOS) environments, respectively, in the static experiment and 0.051 m in the moving target tracking experiment.