Conditional information gain networks as sparse mixture of experts

Biçici, Ufuk Can; Akarun, Lale

doi:10.1016/j.patcog.2021.108151

Cited by 7 publications

(5 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Bayesian Optimization has been used in previous work to set optimal probability thresholds for routing in a tree structure rather than a trellis [16]. Bayesian Optimization relies on sequentially updating a surrogate model (often a Gaussian Process) with new data points based on an acquisition function.…”

Section: Discussion Of Resultsmentioning

confidence: 99%

“…Bayesian Optimization relies on sequentially updating a surrogate model (often a Gaussian Process) with new data points based on an acquisition function. [16] used the Expected Improvement for this purpose. Users cannot explicitly define the data-generating process for the samples.…”

Section: Discussion Of Resultsmentioning

confidence: 99%

“…The first algorithm is based on a Cross-Entropy Search, which finds the optimal probability thresholds using a validation using a loss function that balances extra accuracy and computation. Whenever a router unit in the CIGT structure generates probabilities larger than these thresholds, a sample is routed into multiple routes, resulting in a Sparse Mixture of Experts mode of inference [16]. The second algorithm is based on a supervised Q-learning model, which is made possible by the finite horizontal structure of the CIGT models facilitated by its practical use cases.…”

Section: Discussionmentioning

confidence: 99%

“…These distributions, determined by the network (θ) and router (ϕ) parameters, are optimized using information gain losses. This approach is in line with CIGN models [15], [16]. At each layer of the CIGT, the following joint probability distribution is defined:…”

Section: B Local Information Gain Lossesmentioning

confidence: 99%

“…Building on the principles of conditional computing, ''Conditional Information Gain Networks'' (CIGNs) have been developed to further refine sample routing through network models [15], [16]. The tree structure allows parts of the model to be bypassed.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Multi-Path Routing for Conditional Information Gain Trellis Using Cross-Entropy Search and Reinforcement Learning

Bicici,

Akarun

2024

IEEE Access

Self Cite

View full text Add to dashboard Cite

Convolutional neural networks have made significant strides in solving computer-vision tasks at the expense of high computational demands. This complexity hinders efficient processing, particularly on devices with limited computational resources such as edge devices. One way to overcome this limitation is conditional computing, which optimizes inference by selectively utilizing parts of the network depending on the characteristics of the input. A recent conditional execution method is Conditional Information Gain Trellis (CIGT), which routes samples based on an information gain-based router mechanism. The original CIGT model was designed to route a single sample along a single path in a trellis structure. In this study, advanced inference strategies that allow inputs to traverse multiple paths are proposed to improve the performance of the vanilla CIGT model. These strategies aim to find a middle ground between improved model performance and increased computational demands. For this purpose, two techniques were proposed: A Cross-Entropy Search-based threshold optimization algorithm and a Reinforcement Learningbased routing strategy. The first method treats multi-path routing in CIGT as a black-box optimization problem and the second interprets it as a Markov Decision Process with a Q-Learning-based supervised regression algorithm designed as the solution. It has been shown that both of these methods provide significant performance improvements compared to the original CIGT model, with an adjustable increase in the computation. Experiments were conducted on two image datasets, with additional statistical tests and analyses to inspect the behaviors of the proposed algorithms. The novel methods designed in this study for multi-path routing show potential for both the original CIGT model and similar conditional computation approaches that use specific routing mechanisms for selecting network parts based on samples.

show abstract

Section: Discussion Of Resultsmentioning

confidence: 99%

Section: Discussion Of Resultsmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: B Local Information Gain Lossesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations