Towards event-based end-task learning via image reconstruction

Abstract

This dissertation is concerned with developing end-task learning methodologies using novel camera sensors, especially event cameras, via intelligent imaging from event data in challenging conditions, such as low dynamic range and motion blur. Event cameras are bio-inspired sensors that perceive the intensity changes and output asynchronous event streams. Event cameras exhibit distinct advantages over conventional cameras, such as high dynamic range (HDR), no motion blur. Therefore, event data could be potentially used to tackle challenging vision problems. However, intelligent imaging with event data is distinguished from other modality data by its unique characteristics. First, event data is usually represented as event streams encoding the per-pixel spatial and temporal information, unlike the canonical images. Second, event data predominantly perceive the edges of scenes, rendering them with very sparse outputs. Third, as the output of event cameras are asynchronous event streams, existing vision algorithms can not be directly applied. Although events and images are essentially two distinct modalities, it might be a less optimal solution to learn end-tasks from merely event data while neglect to bridge one to the other. Fourth, compared with the image data, high-quality labeled (per-pixel) event data is scarce and difficult to obtain, thus imposing challenges for end-task learning. Considering the above characteristics, the goal of my research is to develop novel methodologies specialized in intelligent imaging with event data based upon a profound understanding of its nature to better bridge the image data for the purpose of end-task learning. Over the course of my Ph.D. studies, I have pursued research aiming to achieve (1) a proper representation of event data and reconstruct intensity images for end-task task learning, (2) restore better quality images tackling the artifacts induced by the noise of events and enlarging the spatial resolution for end-task learning and (3) bridge image reconstruction approaches with end-task learning. The underlying ideas in my research are encapsulated in the following themes, all of which are focused on developing methodologies for event-based end-task learning via image reconstruction. The research goal is accomplished by two learning strategies: sequential learning and parallel learning. The sequential learning for end-tasks is studied in two approaches, where the end-task learning is based on the generated images from events. In approach 1, a general supervised learning pipeline is proposed to reconstruct intensity images from events based on the conditional generative adversarial network (cGAN). In approach 2, considering existing event cameras are in a low-resolution (LR) and the events and active pixel sensor (APS) frames are noisy and with artifacts, this dissertation strives to jointly reconstruct, restore LR intensity images and generate high-resolution (HR) high-quality intensity images from the LR events via unsupervised adversarial learning. These reconstructed intensity images from approach 1 and approach 2 are applied to learning end-tasks, such as semantic segmentation, object recognition, and object detection. Although sequential learning is effective, they lead to considerable inference latency and less optimal optimization for end-task learning. To overcome the difficulties in sequential learning, approach 3 and approach 4 of this research propose to directly learn from events for end-tasks in parallel with image reconstruction methods developed in approach 1 and approach 2. To better bridge image reconstruction with end-task learning from event data, knowledge distillation (KD) and transfer learning are applied to the end-task learning process. In such a way, the quality of image reconstruction can also be enhanced by the end-tasks via KD losses. And image reconstruction enhances the learning of end-tasks in an end-to-end learning manner. Meanwhile, the feature-level knowledge and prediction-level knowledge are explored to facilitate the end-task learning from events. Such methods lead to no inference latency for learning end-tasks and show more promising results, in especially HDR and blurred conditions.