IMPART: Big media data processing and analysis for film production

Blat, Josep; Evans, Alun; Agenjo, Javi; Kim, Han Sung; Imre, Evren; Hilton, Adrian; Tefas, Anastasios; Nikolaidis, Nikos; Pitas, Ioannis; Polok, Lukas; Smrž, Pavel; Zemčík, Pavel

doi:10.1109/icmew.2015.7169822

Cited by 2 publications

(3 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Complex processing tasks within the pipeline such as 3D rendering, modelling, reconstruction and compositing also take advantage of the massive parallel processing offered by modern GPUs. 16,17 Specialised tasks such as texture synthesis 18 and feature point tracking, 19 among others, have also shown that GPU acceleration can improve real-time performance in the workflow.…”

Section: Background and Related Workmentioning

confidence: 99%

“…The amount of data generated by commercial video production has vastly increased in the last decade, with several Terabytes generated per day by many commercial films. 16 Most modern production methods use multiple high-resolution cameras, sensors and CGI models to generate different scenes and sequences of the story. Post-production pipelines use sequences of read, processing and write operations multiple times to generate the final composited video.…”

Section: Background and Related Workmentioning

confidence: 99%

See 1 more Smart Citation

A case for FPGA-based accelerators for energy-efficient motion picture video processing

Boyle,

Shanker

2023

Applications of Digital Image Processing XLVI

View full text Add to dashboard Cite

In recent years, computing-based video processing workflows have become an integral part of the motion picture industry. These workloads are highly data- and compute-intensive, requiring capable hardware to achieve the required performance. Currently, general-purpose CPUs and GPUs are used to accelerate video processing functions in motion picture workflows. While such devices are highly suited to software-driven video processing workflows, they consume large amounts of energy in these tasks. In this work, we present a case for deploying an FPGA-based accelerator as an energy-efficient alternative to general-purpose hardware in high-resolution motion picture video processing, using an ingest module as a case study. We show that an FPGA-based accelerator for decoding 8K OpenEXR B44 video frames, designed using a commercial high-level synthesis workflow and executing on a PCIe-connected Alveo U50 device, outperforms a highly parallel, CPU-optimised inbuilt B44 decoder implementation in terms of energy consumption per frame decoded. In our experiments, the FPGA-accelerated B44 was able to decode 8K frames with 47.9 ms latency while consuming 0.98 J of energy per frame, compared to the 58.3 ms achieved by a high-end Intel 11700-K CPU while consuming 4.5 J per frame, when averaged over 1000 runs. We further show that this offload can be seamlessly integrated into state-of-the-art motion picture tools such as NUKE with minimal effort. With FPGAs becoming mainstream in cloud servers, we envision that this work paves the way for more efficient integration and utilisation of custom hardware and FPGAs in compute-intensive motion picture workflows.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Section: Background and Related Workmentioning

confidence: 99%

A case for FPGA-based accelerators for energy-efficient motion picture video processing

Boyle,

Shanker

2023

Applications of Digital Image Processing XLVI

View full text Add to dashboard Cite

show abstract

“…Deep-learning tools, such as C opyCat 1 built into Foundry's motion picture software suite Nuke, 2 aid in reducing the amount of human effort in data-intensive tasks in motion picture workflows s uch a s i nteractive s egmentation. 3 W ith m any c ommercial v ideo p roduction r eportedly generating several terabytes of data each day, 4 such reduction can have a cumulative positive impact on the energy consumed over the production phase. Compute-intensive deep-learning models are often accelerated by Graphics Processing Units in video processing workflows, o ften d eployed i n s erver e nvironments s pecialised f or r endering tasks.…”

Section: Introductionmentioning

confidence: 99%

Custom precision accelerators for energy-efficient image-to-image transformations in motion picture workflows

Murphy,

Khandelwal,

Shanker

2023

Applications of Digital Image Processing XLVI

View full text Add to dashboard Cite

Image to Image (I2I) transformations have been an integral part of video processing workflows with applications in Image Synthesis for Virtual Productions, Segmentation, and Matting, among others. Over the years, deep learning-based approaches have been enabling new methods and tools for automating parts of the processing pipeline, reducing the human effort i nvolved i n p ost-production w orkflows. Th ese co mpute-intensive models are often accelerated through on-premise or in-cloud GPU instances to improve the responsiveness and latency while expending large amounts of energy in performing these complex transformations. In this work, we present an approach for optimising the energy efficiency of I2 I de ep-learning mo dels us ing qu antised ne ural networks accelerated on a server-style FPGA. We use deep learning-based alpha background matting as the I2I application which is implemented using a U-Net conditional Generative Adversarial Network deep learning model. The model is trained and quantised using Vitis-AI flow f rom A MD/Xilinx a nd d eployed o n a d ata c entre c lass A lveo U50 FPGA device. Our results show that the quantised model on the FPGA achieves a 1.14× higher throughput for inference acceleration while consuming 11× lower energy consumption per inference when compared to a GPU-accelerated version of the model on a 3080-Ti, while generating nearly identical results with an average IoU > 0.95 across multiple user images at 1080p and 4K resolutions. Additionally, offloads to the FPGA device can be seamlessly integrated into widely used motion picture tools like NUKE with minimal effort. W ith most cloud providers integrating heterogenous platforms (including FPGAs) into systems, we envision that this work paves the way for more efficient ut ilisation of cu stom pr ecision de ep-learning mo dels an d FP GA ac celeration in deep learning-based motion picture workflows.

show abstract

IMPART: Big media data processing and analysis for film production

Cited by 2 publications

References 12 publications

A case for FPGA-based accelerators for energy-efficient motion picture video processing

A case for FPGA-based accelerators for energy-efficient motion picture video processing

Custom precision accelerators for energy-efficient image-to-image transformations in motion picture workflows

Contact Info

Product

Resources

About