(A) Multidimensional sequence search method for motion-based shot retrieval supporting time-warping

Abstract

Content-based video retrieval is one of crucial facilities required to support various kinds of video applications including video on demand, surveillance systems, and so on, which can benefit from the large number of digital video materials. To provide content-based retrieval functionality, video data needs to be properly modeled and indexed based on its contents. In the efficient and effective content-based video modeling, it is natural to incorporate a video abstraction so as to select essential video content from various kinds and aspects of content features. Motion is such a kind of content feature, which is unique to video, that an effective content-based video retrieval system must allow users to search video materials by motion. While required, motion in video is very difficult to deal with. This difficulty is largely due to the projective formation of the video and the temporal vagueness in human motion recognition. In this paper, we propose a framework for motion-based video indexing that can serve as a basis for motion-based video retrieval. To deal with the former difficulty mentioned above, we adopt the affine motion model to describe the instantaneous state of a moving object, which is a popular model in the motion estimation era. Every motion in video is then modeled in the form of multidimensional sequence, whose element describes cumulative motion state with regard to some reference frame. Moreover, we adopt the notion of dynamic time-warping to deal with the latter difficulty of temporal vagueness. The similarity between motion states is devised to reflect the cost of transforming the one into the other, and dynamic time-warping distance is adopted as the overall motion similarity. By modeling as above, the problem of searching video clips by motion becomes an instance of the problem of searching similar multidimensional sequences. When combined to dynamic time-warping distance measure, the problem of searching similar sequences is difficult ...