Canonical Polyadic Tensor Decomposition With Low-Rank Factor Matrices

Phan, Anh-Huy; Tichavský, Petr; Sobolev, Konstantin; Sozykin, Konstantin; Ermilov, Dmitry; Cichocki, Andrzej

doi:10.1109/icassp39728.2021.9414606

Cited by 8 publications

(4 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, we introduce subspace-constrained SBTDs as the formal model of decompositions resulting from Bro and Andersson's [7] compress-decompose-expand approach. Subspace-constrained CPDs were also considered in the recent paper [29]. Suppose that we want to compute an SBTD of the tensor A P R n1ˆ¨¨¨ˆn D associated with the M r -structured Tucker manifolds M n1,...,n D r .…”

Section: Proof the Columns Of Thementioning

confidence: 99%

The condition number of many tensor decompositions is invariant under Tucker compression

Dewaele¹,

Breiding²,

Vannieuwenhoven³

2021

Preprint

View full text Add to dashboard Cite

We characterise the sensitivity of several additive tensor decompositions with respect to perturbations of the original tensor. These decompositions include canonical polyadic decompositions, block term decompositions, and sums of tree tensor networks. Our main result shows that the condition number of all these decompositions is invariant under Tucker compression. This result can dramatically speed up the computation of the condition number in practical applications. We give the example of an 265 ˆ371 ˆ7 tensor of rank 3 from a food science application whose condition number was computed in 6.9 milliseconds by exploiting our new theorem, representing a speedup of four orders of magnitude over the previous state of the art.

show abstract

Section: Proof the Columns Of Thementioning

confidence: 99%

The condition number of many tensor decompositions is invariant under Tucker compression

Dewaele¹,

Breiding²,

Vannieuwenhoven³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…This approach, when applied over the CIFAR-10 dataset with VGG-16 architecture, achieved 3.6× times smaller compression ratio to the SVD. Other commonly used methods for low-rank factorization are tucker decomposition (TD) [155,156,157] and canonical polyadic decomposition (CPD) [158,159]. The main idea behind the early exit approach is to find the best tradeoff between the deep DNN structure of a DL model and the latency requirements for inference.…”

Section: C) Knowledge Distillationmentioning

confidence: 99%

Enabling All In-Edge Deep Learning: A Literature Review

et al. 2023

View full text Add to dashboard Cite

In recent years, deep learning (DL) models have demonstrated remarkable achievements on non-trivial tasks such as speech recognition, image processing, and natural language understanding. One of the significant contributors to its success is the proliferation of end devices that acted as a catalyst to provide data for data-hungry DL models. However, computing DL training and inference is the main challenge. Usually, central cloud servers are used for the computation, but it opens up other significant challenges, such as high latency, increased communication costs, and privacy concerns. To mitigate these drawbacks, considerable efforts have been made to push the processing of DL models to edge servers (a mesh of computing devices near end devices). Moreover, the confluence point of DL and edge has given rise to edge intelligence (EI). International Electrotechnical Commission (IEC) defines EI as the concept where the data is acquired, stored, and processed utilizing edge computing with DL and advanced networking capabilities. Broadly, EI has six levels of categories based on where the training and inference of DL take place, e.g., cloud server, edge server and end devices. This survey paper focuses primarily on the fifth level of EI, called all in-edge level, where DL training and inference (deployment) are performed solely by edge servers. All in-edge is suitable when the end devices have low computing resources, e.g., Internet-of-Things, and other requirements such as latency and communication cost are important such as in mission-critical applications (e.g., health care). Besides, 5G/6G networks are envisioned to use all in-edge. Firstly, this paper presents all in-edge computing architectures, including centralized, decentralized, and distributed. Secondly, this paper presents enabling technologies, such as model parallelism, data parallelism, and split learning, which facilitates DL training and deployment at edge servers. Thirdly, model adaptation techniques based on model compression and conditional computation are described because the standard cloud-based DL deployment cannot be directly applied to all in-edge due to its limited computational resources. Fourthly, this paper discusses eleven key performance metrics to evaluate the performance of DL at all in-edge efficiently. Finally, several open research challenges in the area of all in-edge are presented. INDEX TERMS Artificial intelligence, all in-edge, deep learning, distributed systems, decentralized systems, edge intelligence I. INTRODUCTION T HE global community is increasingly becoming a datadriven environment in which end devices are generating vast quantities of data outside of the traditional data centers. International Telecommunication Union anticipates that global internet traffic per month will reach 607 Exabytes (EB) in 2025 and 5016 EB in 2030 [1]. This enormous amount of data has a positive impact on artificial intelligence (AI) applications. In particular, deep learning (DL) rely on the availability of large quantities of data for its d...

show abstract

“…CP decomposition [2,12] Given a tensor X ∈ R I1×•••×I N , CP decomposition factorizes the tensor into a summation of several rank-one components as…”

Section: Preliminariesmentioning

confidence: 99%

Tensorized LSSVMs for Multitask Regression

Liu¹,

Tao²,

Zhu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Multitask learning (MTL) can utilize the relatedness between multiple tasks for performance improvement. The advent of multimodal data allows tasks to be referenced by multiple indices. Highorder tensors are capable of providing efficient representations for such tasks, while preserving structural task-relations. In this paper, a new MTL method is proposed by leveraging low-rank tensor analysis and constructing tensorized Least Squares Support Vector Machines, namely the tLSSVM-MTL, where multilinear modelling and its nonlinear extensions can be flexibly exerted. We employ a high-order tensor for all the weights with each mode relating to an index and factorize it with CP decomposition, assigning a shared factor for all tasks and retaining task-specific latent factors along each index. Then an alternating algorithm is derived for the nonconvex optimization, where each resulting subproblem is solved by a linear system. Experimental results demonstrate promising performances of our tLSSVM-MTL.

show abstract

Canonical Polyadic Tensor Decomposition With Low-Rank Factor Matrices

Cited by 8 publications

References 24 publications

The condition number of many tensor decompositions is invariant under Tucker compression

The condition number of many tensor decompositions is invariant under Tucker compression

Enabling All In-Edge Deep Learning: A Literature Review

Tensorized LSSVMs for Multitask Regression

Contact Info

Product

Resources

About