Deep learning (DL) based autoencoder is a promising architecture to implement end-to-end communication systems. In this paper, we focus on the fundamental problems of DL-based communication systems, including high rate transmission and performance analysis. To address the limited data rate issue, we first consider the error rate constraint and design a transmission algorithm to adaptively select the transmission vectors to maximize the data rate for various channel scenarios. Furthermore, a novel generalized data representation (GDR) scheme is proposed to improve the data rate of DLbased communication systems. Then, we analyze the effect of signal-to-noise ratio (SNR) and mean squared error performance of the proposed DL-based communication systems. Finally, numerical results show that the proposed adaptive transmission and GDR schemes achieve higher data rate and have lower training complexity than the conventional one-hot vector scheme. Both the new schemes and the conventional scheme have comparable block error rate (BLER) performance. According to both theoretical analysis and simulated results, it is suggested that low or wide-range training SNR is beneficial to attain good BLER performance for practical transmission with various channel scenarios. ). 2 Autoencoder, communication systems, data rate, deep learning, transmission strategy.
I. INTRODUCTIONTo ensure high demand for various communication applications and services, the next-generation network must be able to deliver enhanced mobile broadband, ultra-reliable and low-latency communications (URLLC), and massive Internet of Things (IoT) ecosystems [1]- [4]. The primary concern is to satisfy the exponential rise in the number of user equipments and the traffic capacity of future communication systems. Hence, several promising technologies have been proposed, and they include massive multi-input and multi-output (MIMO) transmissions, millimeter wave (mmWave) communications, ultra-dense networks (UDNs) [5]-[9]. However, there exist a number of limitations for conventional communication systems, such as unavailable channel state information in complex transmission scenario, high complexity to process big data, and sub-optimal performance caused by conventional block structure. For this reason, with the significant development of deep learning (DL) [10]-[12], researchers are attempting to apply the machine learning (ML), especially DL technologies, to communication system design for new benefits [13]-[16] that cannot be obtained using the conventional approaches.As a promising technique, deep learning applies a useful and insightful way to implement communication systems using deep neural networks (NNs). Different from the conventional communication system that consists of multiple independent blocks (e.g., source/channel coding, modulation, channel estimation, equalization), the DL-based communication system can jointly optimize transmitter and receiver for end-to-end performance without block structure [17], [18]. DL-based system design is promising for f...