Visual Categorization of Compositional Human Actions by a Multiple Spatio-Temporal Scales Recurrent Neural Network

Abstract

The current thesis proposes a novel neural network model for categorizing pixel level dynamic visual patterns of human actions. Proposed multiple spatio-temporal scales recurrent neural network (MSTRNN) adds recurrent connectivity to a prior model, the multiple spatio-temporal scales neural network (MSTNN). By developing improved categorical memories than the prior model, the MSTRNN can learn to extract latent spatio-temporal structures from its visual input stream more effectively than MSTNN. In the first experiment using a relatively simple human action video dataset, differences between MSTNN and MSTRNN are examined both qualitatively and quantitatively. The second experiment examines how MSTRNN can learn to categorize video image patterns that can be represented by compositions of several objects and object-directed actions by using a newly prepared human action video dataset for this purpose. The third experiment further examines the categorization capacity of the MSTRNN and its characteristics in more complex situations where human actions are composed of objects, object-directed actions, and action modifiers with newly prepared action dataset for this purpose. The analysis across these different classes of experiments demonstrates that the MSTRNN can deal with human action videos that have different levels of compositionality by developing adequate categorical memories.