Human-Machine Collaborative Video Coding Through Cuboidal Partitioning

Abstract

Video coding algorithms encode and decode an entire video frame while feature coding techniques only preserve and communicate the most critical information needed for a given application. This is because video coding targets human perception, while feature coding aims for machine vision tasks. Recently, attempts are being made to bridge the gap between these two domains. In this work, we propose a video coding framework by leveraging on to the commonality that exists between human vision and machine vision applications using cuboids. This is because cuboids, estimated rectangular regions over a video frame, are computationally efficient, has a compact representation and object centric. Such properties are already shown to add value to traditional video coding systems. Herein cuboidal feature descriptors are extracted from the current frame and then employed for accomplishing a machine vision task in the form of object detection. Experimental results show that a trained classifier yields superior average precision when equipped with cuboidal features oriented representation of the current test frame. Additionally, this representation costs 7% less in bit rate if the captured frames are need be communicated to a receiver.

Versatile Video Coding-Based Coding Tree Unit Level Image Compression With Dual Quantization Parameters for Hybrid Vision

“…Ahmmed et al [28] proposed a cuboids-based human-machine collaborative video coding method. The method generates a cuboid map by extracting cuboidal features from a video frame and takes the mean pixel intensity value of each cuboid area.…”

Task Oriented Video Coding: A Survey

“…Recently, attempts are being made to bridge the gap between these two domains. In work [69], authors propose a video coding framework by leveraging on to the commonality that exists between human vision and computer vision applications using cuboids. This is because cuboids, estimated rectangular regions over a video frame, are computationally efficient, has a compact representation and object centric.…”

Scalable Video Coding for Humans and Machines

“…additional information in the enhancement layer, these methods also support high-quality input reconstruction for human vision. Meanwhile, [14], [15] tried to recover the input image directly from features, without an additional bitstream: [14] from intermediate-layer activations of YOLOv2 [16], and [15] from cuboidal features targeted at YOLOv2. If X denotes the input image, Y the latent space features, X the reconstructed image, and T the machine task output, then a typical machine vision pipeline can be described by a Markov chain X → Y → X → T .…”