Mode‐division multiplexing (MDM) techniques based on various spatial mode bases are of great significance in satisfying the demand for efficient capacity scaling. With the development of diverse MDM systems, the seamless connection and integration between them are particularly important. In this scenario, a laudable goal would be to implement flexible spatial mode‐based mapping. To break the barrier between different MDM systems, a compact 3D photonic device based on diffractive neural network (DNN) is presented for mode‐based mapping, transforming orbital angular momentum (OAM) modes into linearly polarized (LP) modes. Through simulations and experiments, a four‐layer optical neural network mode mapper (ONNMM) is successfully demonstrated, exhibiting its capability of mapping five orthogonal spatial modes between OAM and LP mode bases simultaneously. The ONNMM shows a vision for grooming MDM optical communications and interconnects, as well as other emerging applications enabled by intelligent 3D integrated compact devices.