(A) study on segmentation-guided masked autoencoder learning for optical flow estimation

Abstract

Masked autoencoding (MAE) can be valuable for state-of-the-art optical flow estimation models. FlowFormer++ introduced Masked Cost Volume Autoencoding (MCVA) to pretrain its transformer-based cost-volume encoder, along with a block-sharing masking strategy to prevent information leakage between highly correlated cost maps of neighboring source pixels. In this thesis, we propose a segment-sharing masking strategy to further suppress masked information leakage and promote the learning of relations between cost maps of source pixels at different semantic regions. We show that our pretraining task accelerates optical flow training and enables more accurate recovery of motion boundaries. We also show that the proposed segment-sharing MCVA is more difficult than the original block-sharing MCVA, and that it indeed facilitates the propagation of information between cost maps of source pixels in different semantic regions.