WITA: writing in the air recognition system using RGB data

Abstract

With the rapid advancement in technology, the interaction between human and computer has become inevitable. This has led to an active studies on different text entry systems. In particular, air writing recognition systems gained a lot of attention due to an intrinsic writing pattern that it allows users to follow. Current air writing recognition systems, however, contain many shortcomings. For instance, use of expensive motion sensors and character level recognition hinders real life deployment where easily accessible and word to sentence level recognition system is desirable. In addition, they require users to follow unistroke writing patterns which defeats the purpose of air writing systems: offering an easy communication via natural writing patterns. In this study, we investigate a new air writing recognition system which we call “Writing In The Air (WITA)” recognition system. The WITA text entry system that we propose requires no extra device other than an RGB camera, ensuring accessibility and cost-efficiency, and allows users to follow their natural writing pattern. In order to reach our objective, we take a deep learning approach which requires sufficient amount of data. Thus, a large and comprehensive benchmark dataset, composed of five sub-datasets in two languages (Korean and English) was collected. In addition, four ResNet based end-to-end spatio-temporal networks are proposed to recognize a handwriting from finger movements in RGB data. The best performing network achieves 46% character error rate (CER) in English dataset.