Best practices of mixed-codec adaptive bitrate streaming

Abstract

One of the biggest challenges in Meta's video delivery system is device fragmentation. Due to the large user base of Meta's Family of Apps, our supported devices range from single-core Galaxy Y to the latest Galaxy S22, from the first generation of iPad mini to iPhone 14 pro. Moreover, for devices that do not support hardware decoding for more advanced codecs, such as AV1, we have to rely on software decoders which require high compute power and memory bandwidth. It would be ideal if we can deliver high resolution AVC or VP9 encoded ABR lanes along with low resolution AV1 encoded ABR lanes for the same video, so the client device can choose which one to play based on its compute capacity. In addition, when user uploaded videos that are already encoded with advanced codecs, such as VP9 or HEVC, while generating ABR encoding ladders from the uploaded video using AVC, we also want to deliver the original uploaded video as passthrough to maximize the quality. In both use cases, we will need to support streaming ABR manifest with multiple encoded bitstreams from different codecs and play them smoothly on the client side. On top of that, we can also optimize the ABR encoding and delivery system to select ABR lanes encoded with different codecs for different bitrate targets. In this paper, we will describe how we implemented the end-to-end mixed codec manifest support and deployed the solution in production. The proposed approach effectively generalizes encoding selection into a filtering pipeline, evaluating device capacity via a feedback loop, and being adaptive to the bandwidth estimation, viewport size and device capacity.