Deep Network-Based Frame Extrapolation With Reference Frame Alignment

Affine Transformation-Based Deep Frame Prediction

“…Additionally, the coding performance of our method is compared with one other recent work focusing on extrapo- lation. Specifically, Huo et al [17] recently proposed small extrapolation networks using the pre-aligned reference frames as inputs. Although their networks are integrated with HM-12.0, we believe the comparison with our method integrated into HM-16.6 is valid, because both version of HM use the same parameters associated with the reference lists in the LD and LDP configuration.…”

“…In order to deploy these DNN-based prediction approaches in both P and B frames for general compression scenarios, our earlier study [15] proposed a modified architecture of the network [21], whose input comprises not only two decoded frames, but also their temporal indices, so that the network is explicitly instructed whether to perform extrapolation or interpolation. Huo et al [17] focused on frame extrapolation by applying the frame alignment pre-processing between reference frames before using them as inputs to a lightweight prediction network. Recently, Xia et al [16] introduced multi-scale ASFs to the FRUC network [21].…”

“…In a typical video coding pipeline, there are several components [5]- [17] and optimization processes [18]- [20] that can be aided or replaced completely by DNNs. In particular, many studies regarding DNN-aided inter prediction have been published in the last few years [8]- [17].…”

“…In a typical video coding pipeline, there are several components [5]- [17] and optimization processes [18]- [20] that can be aided or replaced completely by DNNs. In particular, many studies regarding DNN-aided inter prediction have been published in the last few years [8]- [17]. In order to minimize residuals after inter prediction, [8], [9] proposed networks that learn interpolation filters, while [10]- [12] developed refinement networks using motion compensated patches as input.…”

“…In order to minimize residuals after inter prediction, [8], [9] proposed networks that learn interpolation filters, while [10]- [12] developed refinement networks using motion compensated patches as input. In [17], a lighter model for uni-directional prediction (extrapolation) was proposed. Models presented in [14]- [16] share a common network architecture originally developed in [21] for frame rate up-conversion (FRUC).…”