Continual 3D Convolutional Neural Networks for Real-time Processing of Videos

Hedegaard, Lukas; Iosifidis, Alexandros

doi:10.48550/arxiv.2106.00050

Cited by 3 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, a modification to the spatio-temporal 3D convolution was proposed [10], which enables existing 3D CNNs to operate efficiently during continual inference by weights transfer to a Continual 3D CNN. Importantly, Co3D CNNs produce identical output to that of regular 3D CNNs during regular step-wise inference, and the learned weights are directly transferable between regular 3D CNNs and Co3D CNNs.…”

Section: Definition (Continual Inference Network) a Continualmentioning

confidence: 99%

“…Unlike these efforts, our approach produces the exact same computational outputs for temporal sequences as the original Multi-Head Attention module and retains full weightcompatibility. With this work, we also extend the family of Continual Inference Networks [10] to include Transformers with our proposed Continual Retroactive Attention and Continual Single-output Attention. Notably, our attention formulations reduce the per-step cost of the Scaled Dot-Product Attention from time complexity O(n 2 d) to O(nd) and memory complexity O(n 2 ) to O(nd) while producing identical results to those in the original formulation.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Continual Transformers: Redundancy-Free Attention for Online Inference

Hedegaard¹,

Bakhtiarnia²,

Iosifidis³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Transformers are attention-based sequence transduction models, which have found widespread success in Natural Language Processing and Computer Vision applications. Yet, Transformers in their current form are inherently limited to operate on whole token sequences rather than on one token at a time. Consequently, their use during online inference entails considerable redundancy due to the overlap in successive token sequences. In this work, we propose novel formulations of the Scaled Dot-Product Attention, which enable Transformers to perform efficient online token-by-token inference in a continual input stream. Importantly, our modification is purely to the order of computations, while the produced outputs and learned weights are identical to those of the original Multi-Head Attention. To validate our approach, we conduct experiments on visual, audio, and audio-visual classification and detection tasks, i.e. Online Action Detection on THUMOS14 and TVSeries and Online Audio Classification on GTZAN, with remarkable results. Our continual one-block transformers reduce the floating point operations by respectively 63.5× and 51.5× in the Online Action Detection and Audio Classification experiments at similar predictive performance.

show abstract

Section: Definition (Continual Inference Network) a Continualmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Continual Transformers: Redundancy-Free Attention for Online Inference

Hedegaard¹,

Bakhtiarnia²,

Iosifidis³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…First introduced in [34] and subsequently formalized in [35], Continual Inference Networks are Deep Neural Networks that can operate efficiently on both fixed-size (spatio-)temporal batches of data, where the whole temporal sequence is known up front, as well as on continual data, where new input steps are collected continually and inference needs to be performed efficiently in an online manner for each received frame.…”

Section: Continual Inference Networkmentioning

confidence: 99%

“…Recently, Continual 3D CNNs were made possible through the proposal of Continual 3D Convolutions [34]. Likewise, shallow Continual Transformers based on Continual Dotproduct Attentions were introduced in [35].…”

Section: Definition (Continual Inference Network) a Continual Inferen...mentioning

confidence: 99%

“…The temporal convolutions of regular Spatio-Temporal Graph Convolution blocks usually employ zero-padding to ensure equal temporal shape for input and output feature maps. This zero-padding is discarded for Continual Convolutions to avoid continual redundancies [34]. To retain weight compatibility between the regular and continual networks, a delay to the residual connection is necessary.…”

Section: B Delayed Residualmentioning

confidence: 99%

See 1 more Smart Citation

Online Skeleton-based Action Recognition with Continual Spatio-Temporal Graph Convolutional Networks

Hedegaard¹,

Heidari²,

Iosifidis³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Graph-based reasoning over skeleton data has emerged as a promising approach for human action recognition. However, the application of prior graph-based methods, which predominantly employ whole temporal sequences as their input, to the setting of online inference entails considerable computational redundancy. In this paper, we tackle this issue by reformulating the Spatio-Temporal Graph Convolutional Neural Network as a Continual Inference Network, which can perform step-by-step predictions in time without repeat frame processing. To evaluate our method, we create a continual version of ST-GCN, CoST-GCN, alongside two derived methods with different self-attention mechanisms, CoAGCN and CoS-TR. We investigate weight transfer strategies and architectural modifications for inference acceleration, and perform experiments on the NTU RGB+D 60, NTU RGB+D 120, and Kinetics Skeleton 400 datasets. Retaining similar predictive accuracy, we observe up to 109× reduction in time complexity, on-hardware accelerations of 26×, and reductions in maximum allocated memory of 52% during online inference.

show abstract

Continual Inference: A Library for Efficient Online Inference with Deep Neural Networks in PyTorch

Hedegaard¹,

Iosifidis²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

We present Continual Inference, a Python library for implementing Continual Inference Networks (CINs) in PyTorch, a class of Neural Networks designed specifically for efficient inference in both online and batch processing scenarios. We offer a comprehensive introduction and guide to CINs and their implementation in practice, and provide best-practices and code examples for composing complex modules for modern Deep Learning. Continual Inference is readily downloadable via the Python Package Index and at www.github.com/lukashedegaard/continual-inference.

show abstract

Continual 3D Convolutional Neural Networks for Real-time Processing of Videos

Cited by 3 publications

References 0 publications

Continual Transformers: Redundancy-Free Attention for Online Inference

Continual Transformers: Redundancy-Free Attention for Online Inference

Online Skeleton-based Action Recognition with Continual Spatio-Temporal Graph Convolutional Networks

Continual Inference: A Library for Efficient Online Inference with Deep Neural Networks in PyTorch

Contact Info

Product

Resources

About