AR-T: Temporal Relation Embedded Transformer for the Real World Activity Recognition

Abstract

Activity recognition is a fundamental way to support context-aware services for users in smart spaces. Data sources such as video or wearable devices are used in many recognition approaches, but there are challenges in utilizing them in the real world. Recent approaches propose deep learning-based methods on IoT sensor data streams to overcome the issues. Since they only describe single user-based spaces, they are vulnerable to complex sequences of events triggered by multiple users. When multiple users exist in a space, various overlapping events occur with longer correlations than a single user situation. Additionally, ambient sensor-based events appear far more than actuator-based events, making it difficult to extract actuator-based events as important features. We propose a transformer-based approach to derive long-term event correlations and important events as elements of activity patterns. We also develop a duration incorporated embedding method to differentiate between the same type but different duration events and add a sequential manner to the transformer approach. In the experiments section, we prove that our approach outperforms the existing approaches based on real datasets.