User-based AutoenCoder for QoS prediction

Reviewing Autoencoders for Missing Data Imputation: Technical Trends, Applications and Outcomes

“…The activation functions used by the nodes of the hidden and output layers are reported in 21 of the 26 works. For the standard AEs the Sigmoid function is always used (Hau et al, 2016;Sakurai et al, 2017;Jia et al, 2017), but in the work of Sakurai et al (2017) a custom linear function is used on the output layer. For multi-layer networks the Sigmoid is also used in some works (Duan et al, 2014(Duan et al, , 2016Xie et al, 2019;Sánchez-Morales et al, 2020), but ReLU is the one more often applied (Gondara and Wang, 2017;Ryu et al, 2020;McCoy et al, 2018;Boquet et al, 2019Boquet et al, , 2020Xie et al, 2019;Saeed et al, 2018;Fortuin et al, 2020), sometimes through the Leaky ReLU variant (Ryu et al, 2020;Saeed et al, 2018).…”

“…The training phase of an AE depends on the same aspects of any other ANN: an optimization algorithm, a loss function and the maximum number of epochs. From the 17 works that describe the used optimization algorithm, 5 use the well-known Stochastic Gradient Descent (Sánchez-Morales et al, 2017;Hau et al, 2016;Sánchez-Morales et al, 2019;Xie et al, 2019;Lai et al, 2019) and 6 use one of its variants called Adam (El Esawey et al, 2015;Ryu et al, 2020;Boquet et al, 2019Boquet et al, , 2020Saeed et al, 2018;Fortuin et al, 2020). Other algorithms are used less often, namely the Nesterov's Accelerated Gradient (Gondara and Wang, 2018), the Scaled Conjugate Gradient Algorithm (Sakurai et al, 2017) and the RMSProp (McCoy et al, 2018).…”

“…Regarding the loss functions, the majority of the works use the standard Mean Squared Error (MSE) Wang, 2017, 2018;El Esawey et al, 2015;Sakurai et al, 2017;Jia et al, 2017;Boquet et al, 2019;Saeed et al, 2018) or the Squared Error (Hau et al, 2016;Chen et al, 2015;Sánchez-Morales et al, 2019;Ryu et al, 2020). The exceptions are a work that uses a custom loss function roughly based on the Squared Error (Zhao et al, 2016), 3 works that use the Cross-Entropy (Jaques et al, 2017;Lee and Lee, 2017;Sánchez-Morales et al, 2020), another one that uses the Binary Cross-Entropy with a modification to handle the missing values (Beaulieu-Jones and Moore, 2017), and finally 4 VAE-based works which use the log-likelihood (McCoy et al, 2018;Xie et al, 2019;Nazabal et al, 2020;Fortuin et al, 2020).…”

“…To solve this issue, pre-imputation of the missing values is often performed. Gondara and Wang (2018) Some works also define a maximum number for the training epochs: less than 50 (Saeed et al, 2018;Fortuin et al, 2020), 100 (Beaulieu-Jones and Moore, 2017), 300 (Hau et al, 2016), 500 (Gondara and Wang, 2018), 1000 (Gondara and Wang, 2017;Sakurai et al, 2017), 2000 (Nazabal et al, 2020) and 10000 (McCoy et al, 2018;Lai et al, 2019). Although early stopping rules are common on ANNs training, they were only applied by Gondara and Wang (2018), stopping the training when the MSE achieves 1 * 10 −6 or no improvement exists in an average of 5 epochs.…”

“…The L2 term consists of the sum of the squared values of the weights that is multiplied by an attenuation coefficient, which results on a higher error for larger weights, causing the training algorithm to favor smaller weights. Considering all research works, 7 use this approach (Gondara and Wang, 2018;Zhao et al, 2016;Jaques et al, 2017;Hau et al, 2016;Boquet et al, 2019Boquet et al, , 2020Saeed et al, 2018) with the coefficient values varying between 0.01 and 0.001. Gondara and Wang (2017) used batch normalization, where the output of each node of the hidden layers is normalized using the mean and variance of the batch, and 20% dropout in each hidden layer, meaning that 20% of the nodes are randomly set to 0 (they are dropped, becoming inactive) in each layer.…”