Enabling neural-enhanced video streaming

Abstract

and 2) at the client side, adaptive bitrate (ABR) streaming has addressed the problem of bandwidth heterogeneity and its variations across time and space. However, there is a fundamental limitation that the quality of existing video delivery critically depends on the bandwidth resource. Consequently, user quality of experience (QoE) inevitably suffers when network conditions become unfavorable. This dissertation proposes an alternative and complementary approach to enhancing video quality, inspired by the ever-increasing computational power and recent advances in deep learning. Our thesis is that one can substantially improve user QoE in video streaming by neural enhancement based on the client and server computation. Unlike traditional streaming, which heavily depends on network resources, we apply a deep neural network (DNN)-based quality enhancement on video content to obtain high-definition (e.g., 1080p) video from lower quality transmissions. This provides a powerful mechanism for maximizing QoE on top of the existing video delivery infrastructure. However, realizing neural-enhanced video streaming poses two fundamental challenges. First, the performance of DNN predictions is unreliable for unseen/new content. Ensuring consistent quality enhancement is especially difficult, which presents a significant barrier to practical deployment. Second, a DNN used for quality enhancement requires too much computation to run on commodity servers and clients. This severely limits its applicability to Internet-scale streaming services. For downstream video streaming, where a DNN runs at clients, it is infeasible to support real-time neural enhancement on mobile devices. For upstream video ingest, where a DNN runs at media servers, it is too costly to support commercial-scale live streaming using a public cloud. To prove the thesis, this dissertation presents a collection of algorithmic and architectural solutions that address these challenges based on domain-specific insights into DNNs and video streaming. First, we develop a content-aware approach to ensure reliable quality enhancement powered by a DNN. In this approach, we train a separate DNN for each content to exploit the DNN's overfitting property, which delivers reliable and predictable training accuracy instead of relying on unpredictable test accuracy. Next, we reduce the computing cost of a DNN by devising a selective inference approach. In this approach, the DNN is applied only to a few select frames, and the results are then reused to benefit the entire video, by leveraging the vast amount of temporal redundancies within a video. Lastly, we validate our solutions using full-fledged systems to demonstrate significant improvements in user QoE for video streaming.

Internet video has experienced tremendous growth over the last few decades. Current video delivery infrastructure has been successful in handling scalability challenges with two key technologies: 1) at the server side, distributed computing technologies and content delivery networks (CDNs) have enabled content delivery at the Internet scale