Dynamic hand gesture recognition using arm movement features and discriminative training of HMMs

Abstract

In this thesis, we propose a novel method that recognizes Arabic numerals (0-9) from a hand motion trajectory by utilizing arm movement features and discriminative training with hidden Markov models (HMMs). The previous works have mainly focused on 2D gesture trajectory features and an individual HMM classifier for hand gesture recognition. The proposed method extends the previous works by extracting fea-tures from 3D whole arm movement with associated joint angles by utilizing the position data of shoulder, elbow, and hand available from skeletal tracking by KinectTM and building a framework for multiple HMMs with discriminative training. The proposed method consists of the following three parts: arm tracking, feature extraction, and clas-sification. In the arm tracking part, the KinectTM skeletal tracking is used to collect a sequential position data of shoulder, elbow and hand; In the second part, the arm angle features are extracted based on an arm mod-el for trajectory formation and are then quantized as the feature vectors to be used as an observable se-quence for HMM; In the last part, a majority voting of 5 different HMMs based on the weights obtained from discriminative training is incorporated to build a classifier with multiple HMMs. The experimental results show that the proposed method can achieve a successful and high gesture recognition rate for both the training and the testing data. Consequently, the 3D arm movement features are reliable for dynamic hand gesture recognition compared to the system using 2D trajectory features, and the multiple HMMs with discriminative training is effective compared to the system using the individual HMM classifier.