Compressing Local Descriptor Models for Mobile Applications

Abstract

Feature-based image matching has been significantly improved through the use of deep learning and new large datasets. However, there has been little work addressing the computational cost, model size, and matching accuracy tradeoffs for the state of the art models. In this paper, we consider these practical aspects and improve the state-of-theart HardNet model through the use of depthwise separable layers and an efficient tensor decomposition. We propose the Convolution-Depthwise-Pointwise (CDP) layer, which partitions the weights into a low and full rank decomposition to exploit the naturally emergent structure in the convolutional weights. We can achieve an 8× reduction in the number of parameters on the HardNet model, 13× reduction in the computational complexity, while sacrificing less than 1% on the overall accuracy across the HPatches benchmarks. To further demonstrate the generalisation of this approach, we apply it to other state-of-the-art descriptor models, where we are able to a significant performance improvement.

Network compression and faster inference using spatial basis filters

“…This has led to the development of optimized GPU kernels that bridge the gap between the theoretical FLOP improvements and the practical on-device latency. Both CPdecomposition [17] and Tucker decomposition [45] have also been used to construct or compress pre-trained models [21,23,35]. Another line of work has explored the use of tensor networks as a mathematical framework for generalising tensor decomposition in the context of deep learning [14,46].…”

Using a Single Input to Forecast Human Action Keystates in Everyday Pick and Place Actions

“…Moreover, we show that knowledge of the object of interest allow ours and existing methods to be built with far fewer parameters without any degradation in performance. The result is a method that can be deployed in resource-limited systems, a line of research that is actively pursued by the scientific community [5,6,7]. To the best of our knowledge, our method is the first that uses an input from a single timestep to directly forecast the key pose the instant the pick or place action is performed, and the time taken to get to the key pose (Figure 1).…”