Hierarchical neural networks for the classification of undersea events

Shazeer, Dov J.

doi:10.1117/12.140065

Cited by 13 publications

(36 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By specifically modeling the foreground and background of cross-modal images, the fusion network is motivated to learn foreground and background features more directly. Mixture-of-Expert (MoE) [27,29] can dynamically adjust its own structure/parameters while processing different samples to select the most suitable one for the current sample. Inspired by this, we introduce MoE into the modeling of cross-modal foreground and background features with the purpose of expect-ing a better fusion effect through specialized learning of cross-modal foreground and background information.…”

Section: Infrared Imagementioning

confidence: 99%

“…where the pixel loss L pixel constrains the fused image to preserve more significant pixel intensities originating from the target images, while the gradient loss L grad forces the fused image to contain more texture details from different modalities. L load represents load loss, which encourages experts to receive roughly equal numbers of training examples [29]. More details about pixel loss, gradient loss and load loss are provided in the appendix.…”

Section: Loss Functionmentioning

confidence: 99%

“…Following [29], the L F G load , L BG load , and L Decoder load are defined as the square of the coefficient of variation of the load vector. We also follow [29] to initialize the weight matrix of gate network in each MoE to zero, which keeps the expert load of MoE in an approximately equal state during the initial phase.…”

Section: C3 Load Lossmentioning

confidence: 99%

“…can dynamically adjust its structure according to different inputs. Shazeer et al[29] constructed a sparsely-gated MoE layer that uses a gate network to select multiple experts and assign weights to each selected expert. The final result of the MoE is a weighted sum of the outputs of the different experts.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Data envelopment analysis on evaluating the efficiency of public hospitals in Tianjin, China

Bin

Zhang

Yang

et al. 2016

Trans. Tianjin Univ.

View full text Add to dashboard Cite

Infrared and visible image fusion can compensate for the incompleteness of single-modality imaging and provide a more comprehensive scene description based on cross-modal complementarity. Most works focus on learning the overall cross-modal features by high-and low-frequency constraints at the image level alone, ignoring the fact that cross-modal instance-level features often contain more valuable information. To fill this gap, we model cross-modal instance-level features by embedding instance information into a set of Mixture-of-Experts (MoEs) for the first time, prompting image fusion networks to specifically learn instance-level information. We propose a novel framework with instance embedded Mixture-of-Experts for infrared and visible image fusion, termed MoE-Fusion, which contains an instance embedded MoE group (IE-MoE), an MoE-Decoder, two encoders, and two auxiliary detection networks. By embedding the instance-level information learned in the auxiliary network, IE-MoE achieves specialized learning of crossmodal foreground and background features. MoE-Decoder can adaptively select suitable experts for cross-modal feature decoding and obtain fusion results dynamically. Extensive experiments show that our MoE-Fusion outperforms state-of-the-art methods in preserving contrast and texture details by learning instance-level information in cross-modal images. Our code will be available: https://github.com/SunYM2020/MoE-Fusion.

show abstract

Section: Infrared Imagementioning

confidence: 99%

Section: Loss Functionmentioning

confidence: 99%

Section: C3 Load Lossmentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Data envelopment analysis on evaluating the efficiency of public hospitals in Tianjin, China

Bin

Zhang

Yang

et al. 2016

Trans. Tianjin Univ.

View full text Add to dashboard Cite

show abstract

“…On the one hand, the dynamic pattern of MoE results in severe computation load-imbalance problems that a small number of experts may receive, process, and send the majority of data. Several approaches were proposed to make full use of the available experts, such as adding an auxiliary loss [26], controlling expert capacity [11; 6], and optimizing the assignment scheme for a balanced load [12; 28; 22]. On the other hand, global communication is another main obstacle to efficient MoE training.…”

Section: Introductionmentioning

confidence: 99%

TA-MoE: Topology-Aware Large Scale Mixture-of-Expert Training

Chen¹,

Li²,

Wu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Sparsely gated Mixture-of-Expert (MoE) has demonstrated its effectiveness in scaling up deep neural networks to an extreme scale. Despite that numerous efforts have been made to improve the performance of MoE from the model design or system optimization perspective, existing MoE dispatch patterns are still not able to fully exploit the underlying heterogeneous network environments. In this paper, we propose TA-MoE, a topology-aware routing strategy for large-scale MoE trainging, from a model-system co-design perspective, which can dynamically adjust the MoE dispatch pattern according to the network topology. Based on communication modeling, we abstract the dispatch problem into an optimization objective and obtain the approximate dispatch pattern under different topologies. On top of that, we design a topology-aware auxiliary loss, which can adaptively route the data to fit in the underlying topology without sacrificing the model accuracy. Experiments show that TA-MoE can substantially outperform its counterparts on various hardware and model configurations, with roughly 1.01x-1.61x, 1.01x-4.77x, 1.25x-1.54x improvements over the popular DeepSpeed-MoE, FastMoE and FasterMoE systems.

show abstract

Research on flow stress in ferrite deformation of a Ti-IF steel

Zhao

et al. 2006

Journal of University of Science and Technology Beijing, Minera

View full text Add to dashboard Cite

Hierarchical neural networks for the classification of undersea events

Cited by 13 publications

References 0 publications

Data envelopment analysis on evaluating the efficiency of public hospitals in Tianjin, China

Data envelopment analysis on evaluating the efficiency of public hospitals in Tianjin, China

TA-MoE: Topology-Aware Large Scale Mixture-of-Expert Training

Research on flow stress in ferrite deformation of a Ti-IF steel

Contact Info

Product

Resources

About