Training data augmentation for deep learning radio frequency systems

Abstract

Applications of machine learning are subject to three major components that contribute to the final performance metrics. Within the category of neural networks, and deep learning specifically, the first two are the architecture for the model being trained and the training approach used. This work focuses on the third component, the data used during training. The primary questions that arise are “what is in the data” and “what within the data matters?” looking into the radio frequency machine learning (RFML) field of automatic modulation classification (AMC) as an example of a tool used for situational awareness, the use of synthetic, captured, and augmented data are examined and compared to provide insights about the quantity and quality of the available data necessary to achieve desired performance levels. Three questions are discussed within this work: (1) how useful a synthetically trained system is expected to be when deployed without considering the environment within the synthesis, (2) how can augmentation be leveraged within the RFML domain, and, lastly, (3) what impact knowledge of degradations to the signal caused by the transmission channel contributes to the performance of a system. In general, the examined data types each make useful contributions to a final application, but captured data germane to the intended use case will always provide more significant information and enable the greatest performance. Despite the benefit of captured data, the difficulties and costs that arise from live collection often make the quantity of data needed to achieve peak performance impractical. This paper helps quantify the balance between real and synthetic data, offering concrete examples where training data is parametrically varied in size and source.

An RFML Ecosystem: Considerations for the Application of Deep Learning to Spectrum Situational Awareness

“…2 depicts the relationship between the three dataset types discussed herein, and acknowledges that no dataset will ever consist of all relevant data. A more descriptive comparison of quality and quantity for these three dataset types is discussed in [57].…”

“…As a result, synthetically generated RFML datasets can be good analogs for captured RFML datasets, if carefully crafted and known models exist for the simplistic environment. However, a recent AMC analysis [57] showed that, without considering channel effects, models trained on simulated datasets are insufficient when applied to real captured data (i.e. during realworld deployment).…”

“…Though augmented datasets minimize the limitations of synthetic datasets (i.e. real-world model accuracy) and reducing the amount of captured data needed, augmented datasets do not yield the highest performance per observation, when compared to captured data [57]. Further, there are a multitude of open research questions related to the development and use of augmented datasets.…”

“…Perhaps the most important of these open questions is how to balance the amount of real, synthetic, and augmented data used in training datasets to avoid network bias. Work in [30] and [57] both examined data augmentation in the context of AMC, with results in [57] showing that for a fixed quantity of captured data, augmentations which consider the distribution of receiver degradations (SNR, Frequency Offset, Sample Rate Mismatch) can improve the performance of a model over captured data alone. While both works showed that augmentation of the original data result in increased performance, particularly when the degradation distribution is considered, no conclusions were drawn as to how performance changes in response to varying levels of augmentation.…”

“…and categorizing whether each potential degradation is fundamental to the application or a limiting nuisance parameter. For example, a waveform's modulation is fundamental to AMC, while the transmitter imperfections are fundamental to SEI, yet both applications are significantly affected by the channel over which they are observed and is therefore regarded a nuisance parameter [57], [64].…”