This article proposes several second-order optimization methods for time-delay ARX model. Since the time-delay in the information vector makes the traditional identification algorithms be inefficient, a redundant rule based method is utilized to transformed the model into a redundant model. Then, the nature gradient descent (NGD) algorithm is developed for such a model. To reduce the computational efforts of the NGD algorithm and to adaptively update each element in the parameter vector, two modified NGD algorithms are also presented.The simulation examples verify the effectiveness of the proposed algorithms.
Dynamic computation has emerged as a promising strategy to improve the inference efficiency of deep networks. It allows selective activation of various computing units, such as layers or convolution channels, or adaptive allocation of computation to highly informative spatial regions in image features, thus significantly reducing unnecessary computations conditioned on each input sample. However, the practical efficiency of dynamic models does not always correspond to theoretical outcomes. This discrepancy stems from three key challenges: 1) The absence of a unified formulation for various dynamic inference paradigms, owing to the fragmented research landscape; 2) The undue emphasis on algorithm design while neglecting scheduling strategies, which are critical for optimizing computational performance and resource utilization in CUDA-enabled GPU settings; and 3) The cumbersome process of evaluating practical latency, as most existing libraries are tailored for static operators. To address these issues, we introduce Latency-Aware Unified Dynamic Networks (LAUDNet), a comprehensive framework that amalgamates three cornerstone dynamic paradigms-spatially-adaptive computation, dynamic layer skipping, and dynamic channel skipping-under a unified formulation. To reconcile theoretical and practical efficiency, LAUDNet integrates algorithmic design with scheduling optimization, assisted by a latency predictor that accurately and efficiently gauges the inference latency of dynamic operators. This latency predictor harmonizes considerations of algorithms, scheduling strategies, and hardware attributes. We empirically validate various dynamic paradigms within the LAUDNet framework across a range of vision tasks, including image classification, object detection, and instance segmentation. Our experiments confirm that LAUDNet effectively narrows the gap between theoretical and real-world efficiency. For example, LAUDNet can reduce the practical latency of its static counterpart, ResNet-101, by over 50% on hardware platforms such as V100, RTX3090, and TX2 GPUs. Furthermore, LAUDNet surpasses competing methods in the trade-off between accuracy and efficiency. Code is available at: https://www.github.com/LeapLabTHU/LAUDNet.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.