Hand-object pose estimation via interaction-aware graph attention mechanism

Abstract

Estimating hand and object pose from images has emerged as a promising research field due to the increasing demand for practical applications in virtual and augmented reality. The primary objective of this research is to understand the interaction between a hand and an object. To capture the intricate interaction, most existing approaches estimate meshes of the hand and object to represent hand-object interaction, such as contact region. Recently, graph neural networks (GNNs) is used to leverage the graph-like structure of the hand and object meshes, enabling the incorporation of spatial information during inference. However, existing GNN-based methods have not fully exploited the potential of these graphs by not changing their connectivity (edges) within and between different classes. In this thesis, we propose a graph-based refinement method that improves initially estimated hand and object meshes from an image. Our method considers hand-object interaction via an interaction-aware graph attention mechanism, connecting highly-correlated nodes between intra-class graphs and between inter-class graphs. Experiments demonstrate that our proposed method improves accuracy in estimating hand and object pose, and aspects of hand-object interaction.