Calling out, a phenomenon where people publicly broadcast their critiques of someone to a larger audience using, has become increasingly common on social media. However, there has been concerns that it could develop into harassment, deteriorating the quality of public discourse by over-punishing individuals for minor transgressions. To investigate this phenomenon, we interviewed 32 Twitter users who had been called out, had called out, or had witnessed a calling out on Twitter. We found that a key determining factor that distinguishes criticism from harassment was the callee's ability to respond to or engage with the criticism, and that different stakeholders hold different perspectives toward how online harassment is defined. We also discovered that the distinction between callers and callees was not absolute, and that there was high interchangeability of roles both within and across events. Through these findings, we discuss design implications for the platform in promoting healthy discourse while preventing toxic behavior on social media.