A view-based multiple objects tracking and human action recognition for interactive virtual environments

Abstract

As environments become smart in accordance with

advances in ubiquitous computing technology, researchers are

struggling to satisfy users’ diverse and sophisticated demands.

The aim of the present work is to enable multiple persons in a

interactive virtual environment to simultaneously and

conveniently interact with virtual agents. To this end, we propose

a real-time system that robustly tracks multiple persons in virtual

environments and recognizes their actions through image

sequences acquired from a single fixed camera. The proposed

system is compromised of three components: blob extraction,

object tracking, and human action recognition. Given an image,

we extract blobs using the Mixture of Gaussians algorithm with a

hierarchical data structure and we additionally remove shadows

and highlights in order to obtain a more accurate object silhouette.

We then track multiple objects using a motion-based object model

and an inference graph for handling grouping and fragment

problems. Finally, we model an action as a Motion History Image

(MHI) based on given object tracks, normalize the MHI, reduce

the MHI using PCA, and classify an action using a multi-layer

perceptron. To evaluate the performance of the proposed system,

we employed it in an augmented reality application where

multiple persons can interact with a virtual pet. The results

confirm that reliable object tracking is achieved and multiple

persons’ actions can be recognized for applications in interactive

virtual environments.