A wafer bin map (WBM) represents the wafer testing results for individual dies on a wafer using a binary value that represents pass or fail. WBMs often have specific defect patterns, which occur because of assignable causes. Therefore, the identification of defect patterns in WBMs aids in understanding the root causes of process failure. Previous studies on the classification of WBM defect patterns have demonstrated effective performances. However, in previous studies, the effect of class imbalance on the WBM defect patterns was not considered, although in practice, it is more reasonable to assume that there is a significantly large number of WBMs that lack defect patterns because they occur when there are process faults. In this paper, we propose memory-augmented convolutional neural networks with triplet loss for classifying defect patterns in highly imbalanced WBM data. We use a triplet loss-based convolutional neural network as an embedding function to obtain a well-separated low-dimensional space according to the defect patterns. We then use a memory module to balance the number of WBMs between defect-pattern classes. We train the proposed model end-to-end to learn the embedding function and update the memory simultaneously. We then validate the proposed model using simulated WBM data. The proposed model demonstrates a high classification performance and effective embedding results for imbalanced WBM data.