TensorFlow.jl: An Idiomatic Julia Front End for TensorFlow

Malmaud, Jonathan; White, Lyndon

doi:10.21105/joss.01002

Cited by 11 publications

(6 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In subsection 4.1, we train a DNN to map y ∈ R 2 to a scalar c ∈ R. The feature extractor is a ResNet with a width of w = 8 and a depth of d = 8 corresponding to a Appendix C. Autoencoder architecture. We adapt the MNIST autoencoder from [32]. The autoencoder consists of two convolutional neural networks with a user-defined width w and intrinsic dimension d. The width controls the number of convolutional filters used and the intrinsic dimension is the size of the low-dimensional embedding.…”

Section: Appendix B Residual Neural Network (Resnets)mentioning

confidence: 99%

slimTrain -- A Stochastic Approximation Method for Training Separable Deep Neural Networks

Newman¹,

Chung²,

Chung³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep neural networks (DNNs) have shown their success as high-dimensional function approximators in many applications; however, training DNNs can be challenging in general. DNN training is commonly phrased as a stochastic optimization problem whose challenges include nonconvexity, non-smoothness, insufficient regularization, and complicated data distributions. Hence, the performance of DNNs on a given task depends crucially on tuning hyperparameters, especially learning rates and regularization parameters. In the absence of theoretical guidelines or prior experience on similar tasks, this requires solving a series of repeated training problems which can be time-consuming and demanding on computational resources. This can limit the applicability of DNNs to problems with non-standard, complex, and scarce datasets, e.g., those arising in many scientific applications. To remedy the challenges of DNN training, we propose slimTrain, a stochastic optimization method for training DNNs with reduced sensitivity to the choice hyperparameters and fast initial convergence. The central idea of slimTrain is to exploit the separability inherent in many DNN architectures; that is, we separate the DNN into a nonlinear feature extractor followed by a linear model. This separability allows us to leverage recent advances made for solving large-scale, linear, ill-posed inverse problems. Crucially, for the linear weights, slimTrain does not require a learning rate and automatically adapts the regularization parameter. In our numerical experiments using function approximation tasks arising in surrogate modeling and dimensionality reduction, slimTrain outperforms existing DNN training methods with the recommended hyperparameter settings and reduces the sensitivity of DNN training to the remaining hyperparameters. Since our method operates on mini-batches, its computational overhead per iteration is modest and savings can be realized by reducing the number of iterations (due to quicker initial convergence) or the number of training problems that need to be solved to identify effective hyperparameters.

show abstract

Section: Appendix B Residual Neural Network (Resnets)mentioning

confidence: 99%

slimTrain -- A Stochastic Approximation Method for Training Separable Deep Neural Networks

Newman¹,

Chung²,

Chung³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The neural bias model is written using the julia (Bezanson et al 2017) interface to TensorFlow (Abadi et al 2016;Malmaud & White 2018). This is embedded into the hmclet which is one sub-block of the borg algorithm.…”

Section: Neural Bias Modelmentioning

confidence: 99%

Neural physical engines for inferring the halo mass distribution function

Charnock

Lavaux

Wandelt

et al. 2020

Monthly Notices of the Royal Astronomical Society

View full text Add to dashboard Cite

An ambitious goal in cosmology is to forward-model the observed distribution of galaxies in the nearby Universe today from the initial conditions of large-scale structures. For practical reasons, the spatial resolution at which this can be done is necessarily limited. Consequently, one needs a mapping between the density of dark matter averaged over ∼ Mpc scales, and the distribution of dark matter halos (used as a proxy for galaxies) in the same region. Here we demonstrate a method for determining the halo mass distribution function by learning the tracer bias between density fields and halo catalogues using a neural bias model. The method is based on the Bayesian analysis of simple, physically motivated, neural network-like architectures, which we denote as neural physical engines, and neural density estimation. As a result, we are able to sample the initial phases of the dark matter density field whilst inferring the parameters describing the halo mass distribution function, providing a fully Bayesian interpretation of both the initial dark matter density distribution and the neural bias model. We successfully run an upgraded borg inference using our new likelihood and neural bias model with halo catalogues derived from full N-body simulations. We notice orders of magnitude improvement in modelling compared to classical biasing techniques.

show abstract

“…float16, float32 and float64). Though high precision computations are enabled in Julia using multiple-digit "BigFloat", deep learning libraries built on top of Julia such as Flux (Innes et al, 2018;Innes, 2018), MXNet.jl (Chen et al, 2015), and TensorFlow.jl (Malmaud & White, 2018) do not support training with BigFloat. Furthermore, BigFloat in Julia can only exist on CPUs, but not on GPUs, which greatly limits its usages.…”

Section: Introductionmentioning

confidence: 99%

MCTensor: A High-Precision Deep Learning Library with Multi-Component Floating-Point

Chen¹,

Guo²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we introduce MCTensor, a library based on PyTorch for providing general-purpose and high-precision arithmetic for DL training. MCTensor is used in the same way as PyTorch Tensor: we implement multiple basic, matrixlevel computation operators and NN modules for MCTensor with identical PyTorch interface. Our algorithms achieve high precision computation and also benefits from heavily-optimized PyTorch floating-point arithmetic. We evaluate MCTensor arithmetic against PyTorch native arithmetic for a series of tasks, where models using MCTensor in float16 would match or outperform the PyTorch model with float32 or float64 precision.

show abstract

TensorFlow.jl: An Idiomatic Julia Front End for TensorFlow

Cited by 11 publications

References 2 publications

slimTrain -- A Stochastic Approximation Method for Training Separable Deep Neural Networks

slimTrain -- A Stochastic Approximation Method for Training Separable Deep Neural Networks

Neural physical engines for inferring the halo mass distribution function

MCTensor: A High-Precision Deep Learning Library with Multi-Component Floating-Point

Contact Info

Product

Resources

About