Text-guided few-shot multi-organ segmentation for chest X-ray images using vision-language pre-training

Abstract

Recently, many deep learning-based segmentation algorithms have been developed for Chest X-ray (CXR) images. Unfortunately, existing methods have limitations in that one can get segmentation results for the designated organs only when sufficient number of training data are available. In addition, the size of the required training dataset grows rapidly with multi-organ segmentation. To address this, here we present a novel segmentation method based on Vision Language Pre-training (VLP). Thanks to the semantic alignment of images and texts in VLP, our method can generate reliable segmentation results even with extremely scarce label data set, enabling few-shot and even a single-shot segmentation, which is not possible with existing approaches. Furthermore, our method can generate flexible segmentation results for various organs mentioned in the input sentence. Experimental results confirmed that our method significantly outperforms the existing methods when the number of training data is scarce. Furthermore, even with sufficient number of training dataset, the proposed method provides comparable segmentation results to the existing methods, using various kind of input sentences including sentences with multi-organ, overlapping organs, practical expression, and misspelled word.