Toward chatbot safety: Context-aware offensive language detection in chatbot-human conversation data

Abstract

Offensive language exchanged between chatbots and human users can have detrimental consequences on users, chatbot services, and society. Despite notable advancements in autonomous offensive language detection, extending these efforts to chat data remains challenging due to the inherent briefness and contextual complexity of chat messages. This paper introduces a novel offensive language dataset based on real-world chatbot-human conversations, annotated with a focus on context-awareness. Our dataset distinguishes context-dependent offensive language, which is offensive expression under specific context. To address this nuanced task, a neural network model named CALIOPER (Context-Aware modeL for Identifying Offensive language using Pre-trained Encoder and Retrieval) is proposed. CALIOPER leverages a context-aware encoder and attention mechanism to incorporate previous messages and retrieve relevant information from them for detecting offensiveness. Experimental results show that CALIOPER outperforms existing offensive language detection models on multi-turn datasets, particularly for context-dependent offensive language. This work provides a direction in offensive language detection in chat data, contributing to the mitigation of the harmful effects of chatbots and safer chatbot ecosystems.