Multi-task vision transformer for multi-modal medical image processing

Abstract

In this paper, we proposes various Vision Transformer-based models for different medical imaging modalities. Leveraging the intrinsic properties of the Vision Transformer, we applied the proposed methods to classification, segmentation and regression task using optical coherence tomography and radiograph images, to verify the benefit of using Vision Transformer compared with the conventional convolutional neural network. Optical coherence tomography is a medical imaging modality that utilizes light to obtain high resolution images with micrometer scale without harming the living tissue thanks to its non-invasiveness. In addition, as the acquisition speed is fast, it can be used as a real-time imaging modality which can be obtained during the medical procedures. Due to these properties, the optical coherence tomography has been widely used in the field of cardiology for evaluation of underlying pathology in patients with acute coronary syndrome. The optical coherence tomography image has three-dimensional structure with the frames stacked in direction of vessels. Leveraging these volumetric three-dimensional structure, we devises an algorithm that utilizes the Transformer to process the sequential optical coherence tomography images in similar way to natural language processing. Owing to its convenience and the cost-effectiveness, the radiograph image has been widely used for the purpose of screening for a variety of pathologic conditions. As it has two-dimensional structure similar to the natural image, the algorithms devised to process the natural image can readily be applied to this imaging modality. Therefore, we introduces the algorithms based on the properties of Vision Transformer, a recently introduced attention-based architecture without convolution, to improve the generalization capacity as well as the model performance given the limited data and label. In addition, noting that the Vision Transformer is suitable for multi-task learning and distributed learning, we also introduces the multi-task distributed learning methods tailored for Vision Transformer. Finally, as Vision Transformer-based model can benefit more form the self-supervised and semi-supervised learning than convolutional neural networks, we proposes a self-evolving framework that can amalgamate the strengths of two methods under the common ground of knowledge distillation.