Custom precision accelerators for energy-efficient image-to-image transformations in motion picture workflows

Abstract

Image to Image (I2I) transformations have been an integral part of video processing workflows with applications in Image Synthesis for Virtual Productions, Segmentation, and Matting, among others. Over the years, deep learning-based approaches have been enabling new methods and tools for automating parts of the processing pipeline, reducing the human effort i nvolved i n p ost-production w orkflows. Th ese co mpute-intensive models are often accelerated through on-premise or in-cloud GPU instances to improve the responsiveness and latency while expending large amounts of energy in performing these complex transformations. In this work, we present an approach for optimising the energy efficiency of I2 I de ep-learning mo dels us ing qu antised ne ural networks accelerated on a server-style FPGA. We use deep learning-based alpha background matting as the I2I application which is implemented using a U-Net conditional Generative Adversarial Network deep learning model. The model is trained and quantised using Vitis-AI flow f rom A MD/Xilinx a nd d eployed o n a d ata c entre c lass A lveo U50 FPGA device. Our results show that the quantised model on the FPGA achieves a 1.14× higher throughput for inference acceleration while consuming 11× lower energy consumption per inference when compared to a GPU-accelerated version of the model on a 3080-Ti, while generating nearly identical results with an average IoU > 0.95 across multiple user images at 1080p and 4K resolutions. Additionally, offloads to the FPGA device can be seamlessly integrated into widely used motion picture tools like NUKE with minimal effort. W ith most cloud providers integrating heterogenous platforms (including FPGAs) into systems, we envision that this work paves the way for more efficient ut ilisation of cu stom pr ecision de ep-learning mo dels an d FP GA ac celeration in deep learning-based motion picture workflows.