The rapid development of information and wireless communication technologies together with the large increase in the number of end-users have made the radio spectrum more crowded than ever. Besides, providing a stable and reliable service is challenging, as electromagnetic environments are evolving and becoming more sophisticated. Accordingly, there is an urgent need for more reliable and intelligent communication systems that can improve the spectrum efficiency and the quality of service to provide agile management of network resources, so as to better meet the needs of future wireless users. Specifically, Automatic Modulation Recognition (AMR) plays an essential role in most intelligent communication systems especially with the emergence of Software Defined Radio (SDR). AMR is an indispensable task while performing spectrum sensing in Cognitive Radio (CR). Thanks to the significant advancements in Deep Learning (DL) applications, new and powerful tools have been provided which can tackle problems in this space. Thus, today, integrating DL models into AMR has gained the attention of many researchers. This work aims to provide a comprehensive state-of-the-art review of the most recent Machine Learning (ML) based AMR methods for Single-Input Single-Output (SISO) and Multiple-Input Multiple-Output (MIMO) systems. Furthermore, the architecture of each model will be identified along with a detailed comparison in terms of specifications and performance. Finally, an outline of the open problems, challenges, and potential research directions is provided along with discussion and conclusion.

Automatic modulation recognition (AMR) has become increasingly important in the field of signal processing, especially with the advancements of intelligent communication systems. Deep Learning (DL) technologies have been incorporated into the AMR field and they have shown outstanding performances against conventional AMR methods. The robustness of DL-based AMR methods under varying noise regimes is one of major concerns for the widespread utilization of this technology. Furthermore, most existing works have neglected the contributions of hand-crafted features (HCFs) in boosting the classification performances of DL-based AMR methods. In order to address the aforementioned technical challenges, a novel and robust DL-AMR method is proposed by leveraging the benefits of both contextual features (CFs) and HCFs for a specific range of signal-to-noise ratio (SNR). A novel feature selection algorithm is also proposed to search for the optimal sets of HCFs in order to reduce the dimensions of feature vectors without losing any important and relevant features. Simulation studies are performed to investigate the feasibility of proposed method in classifying 11 types of modulation schemes. Extensive performance analyses revealed the superiority of proposed method over baseline method in terms of the classification performance as well as the excellent capability of proposed feature selection algorithm in determining an optimal subset of HCFs.INDEX TERMS Automatic modulation recognition (AMR), convolutional neural network (CNN), deep learning (DL), wireless signal classification.