<title>ECHO: a community video streaming system with interactive visual overlays</title>

Abstract

We describe a networked video application where personalized avatars, controlled by a group of "hecklers", are overlaid on top of a real-time encoded video stream of an Internet game for multicast consumption. Rather than passively observing the streamed content individually, the interactivity of the controllable avatars, along with heckling voice exchange, engenders a sense of community during group viewing. We first describe how the system splits video into independent regions with and without avatars for processing in order to minimize complexity. Observing that the region with avatars is more delay-sensitive due to their interactivity, we then show that the regions can be logically packetized into separable sub-streams, and be transported and buffered with different delay requirements, so that the interactivity of the avatars can be maximized. The utility of our system extends beyond Internet game watching to general community streaming of live or pre-encoded video with visual overlays.

Community Streaming With Interactive Visual Overlays: System and Optimization

“…This has motivated our work on optimizations for graphics-to-video encoding and streaming of virtual world observation for both single-view [10], and multi-view video [11]. In the context of community streaming with IVOs, our earlier work [12] proposed the use of the separable video in the gaming context, and provided empirical results for the case of a single stream over a known channel. In this paper, we significantly extend the scope and generality of our earlier work to include: 1) formal derivation of an optimal scheduling algorithm for the single-separable-stream case; and 2) consideration of multiplexing of multiple streams with derivation of the optimal scheduling algorithm.…”

Coding structure optimization for interactive multiview streaming in virtual world observation

“…For gamegenerated data, in our previous work [7] we have optimized the video quality of regions of interest (ROI) during encoding of single game view in H.263 using per-pixel depth information, which can be extracted from the depth buffer during frame rendering [8]. We have also developed community streaming [9], where a group of game observers can interact visually via overlaid talking heads or personalized avatars in a shared view. Most recently, we have developed fast mode selection algorithms for H.264 video encoding of 3D virtual worlds using per-pixel depth information [10].…”