Continual Inference: A Library for Efficient Online Inference with Deep Neural Networks in PyTorch

Abstract

We present Continual Inference, a Python library for implementing Continual Inference Networks (CINs) in PyTorch, a class of Neural Networks designed specifically for efficient inference in both online and batch processing scenarios. We offer a comprehensive introduction and guide to CINs and their implementation in practice, and provide best-practices and code examples for composing complex modules for modern Deep Learning. Continual Inference is readily downloadable via the Python Package Index and at www.github.com/lukashedegaard/continual-inference.

Efficient High-Resolution Deep Learning: A Survey

“…, f k+1 ) are processed to get the result for time step k + 1, and so on. Continual Inference Networks [127] can exploit redundancies in the computation of overlapping inputs in order to increase the overall efficiency. Continual 3-Dimensional Convolutional Neural Networks (Co3D CNNs) reuse pre-existing 3D CNN weights to reduce the FLOPs for each prediction while retaining similar memory requirements and accuracy [128], and Continual Transformers reuse the redundancy in the selfattention operation of overlapping windows to greatly reduce the time and memory complexity per prediction [129].…”