Earlier researches have proved that the magnetic induction (MI) communication has better performance compared to the electromagnetic (EM) wave propagation in underground (UG) medium as it provides low signal attenuation using small MI coils. In general, non-planar coils are employed as transceivers in the MI UG communication and their performance is dependent on coil parameters.The disadvantages of non-planar coils are their bulkiness and limited transmission distance. In this article, a novel idea of using compact filamentary planar spiral coils for MI UG communication is proposed to achieve higher received power at greater transmission distance. An enhanced MI channel model is also proposed to study the medium's influence on MI performance accurately by considering different soil characteristics. Performances of the circular and square coils in the MI system are evaluated based on mutual inductance, path loss, received power, and signal-to-noise ratio. The performance comparison suggests that the filamentary planar spiral square coil MI system achieves 9.22% higher received power than the filamentary planar spiral circular coil MI system. Moreover, the lateral and angular misalignment study shows that square coils are more tolerant to misalignments. For the proposed filamentary planar square spiral coil (FPSSC) MI system, the influence of coil parameters, channel parameters, and coil misalignment on the received power was carried out and its least sensitive and most sensitive parameters were identified for further optimization. The performance comparison between the proposed FPSSC MI system and the traditional non-planar MI coil system suggests that the received power of the proposed system is improved by 47-49 dBm thus extending the achievable transmission distance.