PPI-BERT: Pretraining transformers with masked sequence-structure of protein fragments for learning protein-protein interactions

Abstract

The ability to treat 3D data or point clouds has tremendously impacted various applications. Proteins are functional components in biological processes that comprise amino acid residues linked by peptide bonds. Linear polypeptides fold into a specific 3D structure and form a complex with other proteins or biomolecules for their cellular functions. Predicting whether two proteins interact, also known as protein-protein interactions (PPI), is a fundamental challenge in biomedical fields. Here, we propose PPI-BERT, a pre-trained Transformer to learn PPI using protein sequences and structures repre- sented as heterogenous point clouds. Our model uses a rotation invariant method to obtain a canonical representation of protein structures and segments them into fragments of fixed amino acid lengths while retaining information regarding atom positions and amino acid classes. This “sequence-structure” representation is used to train a tokenizer that learns discrete token IDs to optimize the sequence and structure reconstruction. Masked modeling is used to train the Transformer encoder model on tokenized fragments. Our self-supervised model was trained on protein complex structures (N=85,885) from the Protein Data Bank. Evaluation shows that our model outperforms existing methods in two critical PPI downstream tasks: binding and interface region predictions. These results are an important step toward developing computational models for PPI applications such as drug discovery.