Deep learning approach to link weight prediction

Hou, Yuchen; Holder, Lawrence B.

doi:10.1109/ijcnn.2017.7966076

Cited by 7 publications

(3 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, we further explore the performance of our method on the well-known model (SVHN) used in LAQ (Hou and Kwok 2018). Table 5 shows the test accuracy on SVHN.…”

Section: Experimental Results On Svhnmentioning

confidence: 99%

“…To overcome the above limitation, the research (Hou et al 2017) proposes a loss-aware quantization approach, directly optimizing binarized weights to minimize the final loss. The following work (Hou and Kwok 2018) extends this binarization scheme to m-bit quantization, but Peng et al (Peng et al 2021) have discovered that the above approaches under certain conditions may fail to converge due to the quantization error. Zhou et al (Zhou et al 2018) have proposed an explicit loss-aware weight quantization method by integrating the information of loss function with respect to full-precision weights into the quantization.…”

Section: Related Workmentioning

confidence: 99%

“…This problem becomes even more serious if the extremely low bitwidth fixed point (e.g., 1-bit) representation is used in the quantization (Bai, Wang, and Liberty 2018). Therefore, many studies on loss-aware quantization (LAQ) have proposed to solve the above issue (Hou and Kwok 2018;Hou et al 2017;Zhou et al 2018;Qu et al 2020;Saito and Mitsui 1996), and the main idea of LAQ is to optimize the quantized weights rather than the full-precision ones during the minimization of the loss function.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

One-Step Forward and Backtrack: Overcoming Zig-Zagging in Loss-Aware Quantization Training

Ma,

Zhou,

et al. 2024

AAAI

View full text Add to dashboard Cite

Weight quantization is an effective technique to compress deep neural networks for their deployment on edge devices with limited resources. Traditional loss-aware quantization methods commonly use the quantized gradient to replace the full-precision gradient. However, we discover that the gradient error will lead to an unexpected zig-zagging-like issue in the gradient descent learning procedures, where the gradient directions rapidly oscillate or zig-zag, and such issue seriously slows down the model convergence. Accordingly, this paper proposes a one-step forward and backtrack way for loss-aware quantization to get more accurate and stable gradient direction to defy this issue. During the gradient descent learning, a one-step forward search is designed to find the trial gradient of the next-step, which is adopted to adjust the gradient of current step towards the direction of fast convergence. After that, we backtrack the current step to update the full-precision and quantized weights through the current-step gradient and the trial gradient. A series of theoretical analysis and experiments on benchmark deep models have demonstrated the effectiveness and competitiveness of the proposed method, and our method especially outperforms others on the convergence performance.

show abstract

“…In this section, we further explore the performance of our method on the well-known model (SVHN) used in LAQ (Hou and Kwok 2018). Table 5 shows the test accuracy on SVHN.…”

Section: Experimental Results On Svhnmentioning

confidence: 99%