Ship collisions are major types of maritime accidents which may involve the loss of life and significant damage to property and environments. To ensure navigational safety in ship encounter situations, the International Maritime Organization (IMO) formalized international regulations for preventing collision at sea (COLREGs) that define the rules for evasive procedures depending on the geometric configuration and relative motion between two ships. However, not all ships strictly follow this procedure and the rules can sometimes be interpreted differently between encountering ships, which may lead to dangerous situations. This paper addresses the intent inference-based automatic collision avoidance in the encounter situations with the COLREG-violating vessels. The reciprocal fast probabilistic velocity obstacle (R-fPVO) is proposed to calculate the best evasive action considering the trajectory uncertainty. Also, to quantify the rule violation of the other vessel, a probabilistic graphical model is designed and constructed and the probabilistic belief of the vessel's intention is inferred using the acquirable information. To verify the feasibility of the proposed algorithm, Monte-Carlo simulations were conducted, and the results have been discussed.