On Transformative Adaptive Activation Functions in Neural Networks for Gene Expression Inference

Abstract

Motivation: Gene expression profiling was made cheaper by the NIH LINCS program that profiles only ∼1, 000 selected landmark genes and uses them to reconstruct the whole profile. The D-GEX method employs neural networks to infer the whole profile. However, the original D-GEX can be further significantly improved. Results: We have analyzed the D-GEX method and determined that the inference can be improved using a logistic sigmoid activation function instead of the hyperbolic tangent. Moreover, we propose a novel transformative adaptive activation function that improves the gene expression inference even further and which generalizes several existing adaptive activation functions. Our improved neural network achieves average mean absolute error of 0.1340 which is a significant improvement over our reimplementation of the original D-GEX which achieves average mean absolute error 0.1637

Adaptive activation functions accelerate convergence in deep and physics-informed neural networks

“…Shen et al [22], used a similar idea of tunable activation function but with multiple outputs. Recently, Kunc and Kléma proposed a transformative adaptive activation functions for gene expression inference, see [14]. One such adaptive activation function is proposed by Jagtap and Karniadakis [6] by introducing scalable hyper-parameter in the activation function, which can be optimized.…”

On tower and checkerboard neural network architectures for gene expression inference

“…The extension of the D-GEX [8] significantly improves the gene expression inference firstly by replacing the hyperbolic tangent activation functions with the logistical sigmoids and secondly by including novel transformative adaptive activation functions (TAAFs) which add four additional parameters per neuron that control the scale and translation of the inner activation function (i.e., the sigmoid function). These parameters increase the total number of parameters of the network only slightly as most of a NN's parameters are the weights of the connections; furthermore, the D-GEX with TAAFs outperform the original D-GEX even when the number of neurons in each layer is set such that the total number of parameters is same for both the original and the modified D-GEX variants.…”

“…One of the approaches lowering the costs and allowing larger-scale experiments is represented by the LINCS 1 program which developed the L1000 platform based on Luminex bead technology. The L1000 platform measures further improved the quality of the inference by introducing a novel family of adaptive activation functions called transformative adaptive activation functions (TAAFs) that allowed significantly lower error [8].…”

“…However, the model with TAAFs [8] still uses the same architecture as the original D-GEX with the exception of the adaptive activation functions. We present a novel model based on a NN with a novel architecture that further improves gene inference by replacing the dense layers with a more complex structure while not increasing the number of parameters and thus retaining the same memory constraints that are necessary for training the model on GPUs.…”

“…They are also commonly used in biology [9][10][11]. Our work builds on the D-GEX [7] and its extension with TAAFs [8].…”