Augmentation of sign language poses by including the understanding of the sign language domain by body part

Abstract

Sign language recognition refers to the process of recognizing sign language expressions, into spoken language expressions. Various studies are being conducted to solve this with deep learning models, but these models face the issue of not performing satisfactorily because of an insufficient amount of data to train the models stably, so typically, augmenting the data is used to solve such problems. In the case of existing sign language data, a method similar to augmenting images has been used. However, these general augmentation methods do not effectively capture the characteristics of sign language, such as the difference in meaning and structure included in each body part and the varying degrees to which they contribute to recognition. This study analyzes the impact of each augmentation method on different body parts, its effect on the structure of sign language, and the relationship with learning performance in perception. Additionally, two hypotheses are proposed for the augmentation methods for each body part. For hands, which are generally more important in sign language representation and contain critical information such as handshapes, methods that can preserve the structural relationship between key-points will be very useful in the recognition process. For the body, which has less representation ability of sign language, augmentation may not have a significant impact. However, since the body contains spatial information including the position of the wrists, which convey the location component, methods that preserve spatial information could be beneficial. Through recognition experiments, we confirmed the similarity between the learning of the sign language recognition model and the hypotheses proposed in this study, and we distinguished appropriate augmentation methods and their characteristics for each body part. Our research, through these findings, seeks to understand the relationship between the characteristics of different body parts and augmentation methods, and offers insights into augmentation techniques.