Distributed video transcoding based on MapReduce

Song, Chenwei; Shen, Wenfeng; Sun, Lianqiang; Zhou, Lihua; Xu, Weimin

doi:10.1109/icis.2014.6912152

Cited by 13 publications

(10 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Kim et al [15] propose a distributed transcoding system based on Apache Hadoop to improve transcoding throughput for H.264 and HEVC encoders, achieving higher throughput for larger batches of videos. Song et al [31] obtain similar results that in [15] for their Hadoop-based approach using H.264 encoders. As an evolution of these previous proposals, Sameti et al [27] propose a transcoder agnostic design based on Apache Spark.…”

Section: Related Worksupporting

confidence: 74%

Event-Driven Serverless Pipelines for Video Coding and Quality Metrics

et al. 2023

View full text Add to dashboard Cite

Nowadays, the majority of Internet traffic is multimedia content. Video streaming services are in high demand by end users and use HTTP Adaptive Streaming (HAS) as transmission protocol. HAS splits the video into non-overlapping chunks and each video chunk can be encoded independently using different representations. Therefore, these encode tasks can be parallelized and Cloud computing can be used for this. However, in the most extended solutions, the infrastructure must be configured and provisioned in advance. Recently, serverless platforms have made posible to deploy functions that can scale from zero to a configurable maximum. This work presents and analyses the behavior of event-driven serverless functions to encode video chunks and to compute, optionally, the quality of the encoded videos. These functions have been implemented using an adapted version of embedded Tomcat to deal with CloudEvents. We have deployed these event-driven serverless pipelines for video coding and quality metrics on an on-premises serverless platform based on Knative on one master node and eight worker nodes. We have tested the scalability and resource consumption of the proposed solution using two video codecs: x264 and AV1, varying the maximum number of replicas and the resources allocated to them (fat and slim function replicas). We have encoded different 4K videos to generate multiple representations per function call and we show how it is possible to create pipelines of serverless media functions. The results of the different tests carried out show the good performance of the serverless functions proposed. The system scales the replicas and distributes the jobs evenly across all the replicas. The overall encoding time is reduced by 18% using slim replicas but fat replicas are more adequate in live video streaming as the encoding time per chunk is reduced. Finally, the results of the pipeline test show an appropriate distribution and chaining among the available replicas of each function type.

show abstract

Section: Related Worksupporting

confidence: 74%

Event-Driven Serverless Pipelines for Video Coding and Quality Metrics

et al. 2023

View full text Add to dashboard Cite

show abstract

“…On the other hand, this work attempts to introduce a third-party file system in the system, and wants to further improve the transcoding efficiency by reducing the disk I / O and network time overhead during the MapReduce process of the cluster. Song et al [25] also implemented a distributed video transcoding system based on MapReduce and FFmpeg, and experimentally analyzed that the system has different transcoding speed promotion rates for videos of different sizes. It can achieve a speed increase of 1.38 times, 1.51 times, and 1.64 times for 500MB, 1GB, and 2GB video processing respectively.…”

Section: Related Workmentioning

confidence: 99%

“…In general, the above researches [17,21,22,24,25,[28][29][30] have proposed a variety of ideas for the application of processing large-scale video data transcoding, and revealed the advantages of distributed transcoding for traditional transcoding methods when processing large-scale data. On the other hand, they did not further explore the factors affecting the efficiency of distributed transcoding under the proposed mechanism.…”

Section: Related Workmentioning

confidence: 99%

CDVT: A Cluster-Based Distributed Video Transcoding Scheme for Mobile Stream Services

Ren

et al. 2021

Wireless Algorithms, Systems, and Applications

View full text Add to dashboard Cite

Distributed video transcoding has been used to huge video data storage overhead and reduce transcoding delay caused by the rapid development of mobile video services. Distributed transcoding can leverage the computing power of clusters for various user requests and diverse video processing demands. However, it imposes a remaining challenge on how to efficiently utilize the computing power of the cluster as well as achieve optimized performance through reasonable system parameters and video processing configurations. In this paper, we design a Cluster-based Distributed Video Transcoding System called CDVT using Hadoop, FFmpeg, and Mkvmerge to achieve on-demand video splitting, on-demand transcoding, and distributed processing, which can be applied to large scale video sharing over mobile devices. In order to further optimize system performance, we conducted extensive experiments on various data sets to find relevant factors that affect transcoding efficiency. We dynamically reconfigure the cluster and evaluate the impacts of different intermediate tasks, splitting strategies, and memory configuration strategies on system performance. Experimental results obtained under various workloads demonstrate that the proposed system can ensure the quality of transcoding tasks while reducing the time cost by up to 50 percent.

show abstract

“…If we pay attention to video transcoding on cloud infrastructures, there are few works in the literature that are directly related to the work proposed in this paper. Previous proposed architectures for distributed video encoding/transcoding in the cloud has been proposed for H.264 [17][18][19][20][21]. These approaches rely on MapReduce or on cluster of machines to perform the encoding, therefore the main difference with the proposed work is that the number of encoders cannot be modified dynamically once the encoding is started.…”

Section: Related Workmentioning

confidence: 99%

A Cloud-Based Distributed Architecture to Accelerate Video Encoders

et al. 2020

View full text Add to dashboard Cite

Nowadays, video coding and transcoding have a great interest and important impact in areas such as high-definition video and entertainment, healthcare and elderly care, high-resolution video surveillance, self-driving cars, or e-learning. This growing demand for high-resolution video boosts the proposal of new codecs and the development of their encoders that require high computational requirements. Therefore, new strategies are needed to accelerate them. Cloud infrastructures offer interesting features for video coding, such as on-demand resource allocation, multitenancy, elasticity, and resiliency. This paper proposes a cloud-based distributed architecture, where the network and the storage layers have been tuned, to accelerate video encoders over an elastic number of worker encoder nodes. Moreover, an application is developed and executed in the proposed architecture to allow the creation of encoding jobs, their dynamic assignment, their execution in the worker encoder nodes, and the reprogramming of the failed ones. To validate the proposed architecture, the parallel execution of existing video encoders, x265 for H.265/HEVC and libvpx-vp9 for VP9, has been evaluated in terms of scalability, workload, and job distribution, varying the number of encoder nodes. The quality of the encoded videos has been analyzed for different bit rates and number of frames per job using the Peak Signal-to-Noise Ratio (PSNR). Results show that our proposal maintains video quality compared with the sequential encoding while improving encoding time, which can decrease near 90%, depending on the codec and the number of encoder nodes.

show abstract

Distributed video transcoding based on MapReduce

Cited by 13 publications

References 6 publications

Event-Driven Serverless Pipelines for Video Coding and Quality Metrics

Event-Driven Serverless Pipelines for Video Coding and Quality Metrics

CDVT: A Cluster-Based Distributed Video Transcoding Scheme for Mobile Stream Services

A Cloud-Based Distributed Architecture to Accelerate Video Encoders

Contact Info

Product

Resources

About