In this dissertation, we mainly present our work on three challenging problems of digital video applications: video compression, summarization and adaptation. Unlike conventional techniques, we focus on the video content modeling and investigate how the characteristics of human attention will help solving these problems. We denote these approaches “content-aware” approaches and present our innovations in the main body of this dissertation.

The first problem is content-aware video adaptation. We employ saliency analysis, which generates a saliency map to indicate the relative importance of pixels within a frame for human attention modeling. We also propose a nonlinear saliency map fusing approach that considers human perceptual characteristics. To effectively map the important content from source to the target display, we propose to have both intra-frame visual considerations and inter-frame visual considerations, where intra-considerations focus on measuring the information loss within a frame, and inter-considerations emphasis the visual smoothness between frames. The mapping problem is formulated as a shortest path problem and is solved with dynamic programming.

The second problem is content-aware video summarization. We introduce an automatic video summarization approach that includes unsupervised learning of original video-audio concept primitives and hierarchical (both frame and shot levels) skimming. For video concept mining, we propose a novel model using bag-of-word (BoW) shot features. We further design a hierarchical video summarization framework which jointly considers content completeness, saliency, smoothing and scalability.

Another problem that we investigate is content-aware video compression. We study this problem in two aspects. In one aspect, we propose a content-aware framework and formulate the constraint optimization of rate-distortion as a resource allocation problem, where bit-allocation is adjusted differently at two levels: region-of-interest (ROI) and non-ROI, intra frames and inter frames. The results exhibit better visual quality as well as objective quality improvement. In the other aspect, we aim at improving the coding efficiency of existing H.264 intra prediction. We incorporate a reverse encoding order with geometric analysis for binary transition points on block boundaries to explicitly derive the prediction direction. Besides, we design and implement a video coding parameter analyzer to facilitate the development of new coding tools for state-of-the-art and next-generation video compression standards.