Real-time full-body virtual avatar from sparse tracking data

Abstract

Currently, animating a full-body virtual avatar requires tracking data of six human joints including head, pelvis, two hands and two feet. However, trackers on feet often fail due to infrared occlusion caused by furniture and objects inside the playing area and relatively farther distance from tracking stations than the other trackers'. Moreover, commercial devices for virtual/augmented reality only offer three trackers on head and two hands as default

consumers have to buy additional trackers to experience full-body virtual avatar. In this paper, we introduce a neural network based method for real-time prediction of feet positions with limited number of tracking devices on upper body joints. From input tracking data and predicted feet positions, our framework reconstructs the full-body pose of a virtual avatar. With motion data with lower dimension of joints, preprocessed from existing motion capture dataset, we train a Gated Recurrent Unit (GRU) based neural network to predict feet positions in current time frame from the sequence of tracked transformations of head, pelvis and two hands. The full-body pose is computed from input upper body tracking data and output feet positions by an inverse kinematics solver. In addition, we propose regularization terms designed to minimize artifacts including foot-sliding, foot-floating and discontinuity between output poses. Our system contributes to produce plausible full body animation of a virtual avatar without direct tracking on feet joints, while being fast enough to run in real-time applications.