The advancement of smartphones has been a prerequisite for multi-sensor fusion-based indoor mapping due to their high penetration and low cost. This paper constructs a sparse single-track semantic map by fusing semantic landmarks, Wi-Fi landmarks, and waypoints, which are scene-representable and navigational in unknown indoor environments. The pedestrian dead reckoning (PDR)-aided visual-inertial simultaneous localization and mapping (PDR-aided VI-SLAM) method uses the PDR velocity as an external observation to constrain the preintegration of inertial measurement unit (IMU) measurements. The semantic objects detected by the YOLO V4 are prefiltered by the proposed semantic object filtering algorithm before semantics matching. The centroid of feature points clustered by the density-based spatial clustering of applications with noise (DBSCAN) algorithm characterizes the location of a semantic landmark relative to the waypoints estimated by the PDR-aided VI-SLAM. To enhance the stability of Wi-Fi landmarks, Wi-Fi signals are processed by the proposed sliding window-based Wi-Fi fusion algorithm. By using waypoints as an intermediate quantity, the association relationships between waypoints, semantic landmarks, and Wi-Fi landmarks are established. After that, a lightweight single-track semantic map is constructed. A landmark matching-based localization method is proposed to evaluate the similarity between the local map and the constructed scene map to infer the location of a pedestrian in the scene map. Experiments are conducted in the office building and mall scenes under various illumination conditions and semantic density. Results demonstrate the high quality of the single-track semantic map and high-precision localization of a local map in the constructed scene map.