Performance evaluation of online learning system based on concept drift adaptation scheme for AMI data processing

Abstract

DL models working on AMI streaming data presents the problem of concept drift, in which the data distribution changes over time. This phenomenon might lead to performance degradation in the DL processing. Updating models should therefore be required to maintain a high performance. Until now, there have been several online learning systems that support automatic updates. However, the common problem of existing approaches is that they do not consider the factors affecting the training efficiency, particularly in the occurrence of the concept drift. In this thesis, we address the problem by developing an adaptive online learning system, taking into account the concept drift and batch size effects on the model training. We basically proposed a light-weight concept drift detection method using the cosine similarity and sliding windows. The model training is then performed with the consideration of the drift detection for each update. Specifically, when no concept drift occurs, the model is trained without early stopping to ensure a good convergence on the new data. Otherwise, early stopping is applied to reduce the training cost. In addition, we proposed an adaptive batch size algorithm as an improvement of the existing method, with simpler metrics and objective function, yet more efficient. The objective of this algorithm is to find an optimal batch size for each update to make a trade-off for the prediction accuracy and data incorporation latency. To verify our proposed algorithms, we implemented our online learning system on Apache Kafka cluster system, which enables processing a large amount of AMI streaming data. In the evaluation task, we conducted experiments using AMI data from 2015 to 2018, and measured the training loss, inference error, and test loss. The experimental results showed our system performance was better than Continuum (r = 4), and comparable to Accelerated Streaming Data Processing.