Spatiotemporal anomaly detection using convolutional and recurrent reconstructive network

Abstract

Nowadays, most production lines are automated and optimized only per each unit process. Thus, it makes hard to know what happened in previous or next process as there are no connections between processes. Recently, as many companies introducing smart factories into existing manufacturing processes, manufacturing fields are changing drastically. The smart factory is a plant that has sensors installed in the machines, which collects and analyzes extracted data in real time. The smart factory enables the operator to observe all process at a glance, analyze them, and control according to the purpose. Once a smart factory is implemented, the processes are analyzed using a large amount of data, which enables to identify the defective factors, and obtain the correlation between processes. Surface Mount Technology (SMT) is one of fields being applied smart factories. SMT is a process of soldering and mounting chips onto a printed circuit board (PCB). SMT process is divided into printing solders onto PCB, mounting chips onto printed PCB, and melting the solders to attach the chips to solders. Kohyoung technology has solder paste inspection (SPI) equipment for inspecting solders on the printed PCB and automatic optical inspection (AOI) equipment for inspecting chips mounted onto the PCB. Both SPI and AOI equipment can reconstruct the soldering and mounting status of the PCB to a three-dimensional image. The inspection image enables to distinguish whether he PCB is defective or not. In order to increase the productivity in the SMT line, it is important to defect PCB defects at an early stage, which is possible to detect PCB’s soldering defects through SPI. In this paper, we use PCB images measured by SPI to diagnose the defect of the printer itself connected to SPI. If the printer has a defect, it is assumed that the PCB images measured by the SPI differs from the normal PCB images. In this paper, we propose a novel neural network structure, a convolutional recurrent reconstructive network (CRRN) for diagnosing printer defects. ConvREM is a kind of recurrent auto-encoder with attention mechanism. As a core network of CRRN, we propose Convolutional Spatio-Temporal Memory to capture the spatiotemporal patterns easily. The model is trained unsupervised manner using only normal images and the anomaly is detected by comparing normal and abnormal results. In addition, we classify the defect factors of the printer through classification model. We verified the performance of proposed model using moving MNIST dataset, SPI data, and welding images. The solder paste printer defects and welding defects were classified using the reconstruction error map obtained through the reconstruction model.